Kernel level attack?

Question

In a Unix-like system, the concept of privileged and non-privileged users is used for security, preventing numerous attacks. When a non-privileged user executes malicious code at a normal level, the kernel can reject its execution by recognizing the user's lack of privilege.

However, it appears to me that this security mechanism is effective only if the malicious code is crafted to exclusively utilize the kernel API. It is crucial to remember that the kernel itself is essentially a sequence of instructions for the micro-controller. In theory, there could be instruction code that bypasses the privilege check of the kernel by executing without utilizing the kernel API. To make it worse, I can't imagine how the kernel could preemptively identify, before execution, whether a particular code segment ultimately involves the use of the kernel non-API.

In theory, the existence of such code seems plausible to me. Is my understanding correct? If so, I am interested in discovering examples of such code or exploring discussions related to historical instances.

Answer 1

Someone will hopefully come along with a more complete answer, but this will get you started. I would write more but this is typically a 200 level course in CS, and I’m on mobile. I also may do a poor job of explaining. Here it goes…

What you’re missing is that Unix-like operating systems (and most/all other general purpose operating systems) make use of different CPU contexts, which have different privilege levels; these are effectively security controls and they’re enforced by the CPU

Note, when I say “context”, I mean both the general register state AND the control registers which dictate privilege. Not only a simple “context switch”, if you know the term

Yes, you’re correct that the instructions run on the same CPU, but there are instructions that can only execute in the kernel context (and registers only addressable in that context, as well)

These instructions will fault when executed in userland, as userland is not privileged

It’s no coincidence that these privileged instructions (and registers) are critical to performing I/O as well as setting up userland memory mappings, which act as security bondareis (no mapping means no access)

Things you need to learn about to understand this:

• Protection rings, specifically ring-3 (userland context) and ring-0 (kernel context) in the case of Unix-like operating systems as well as Windows
• The MMU which is critical for restricting userland access to privileged memory
• Virtual Memory, closely related to the MMU
• Interrupts, which can be used as a signal that a userland process wants to perform a privileged operation. They trigger a context switch into interrupt handlers, implemented by the kernel
• System Calls, the interface for userland to request specific operations implementes in the kernel. There are hundreds of system calls (maybe well over a thousand at this point?) implemented in Linux. Examples of system calls include open(), read(), write(), etc. Invoking a system call from userland typically involves setting a specific general purpose register to a system call number and then invoking a specific interrupt number (via a break or trap instruction, depending on the CPU architecture)

Answer 2

In linux, a process runs in a specific context. This context includes memory assigned to the process by the kernel and a permission set including user ID.

When the process executes, it only has access to the memory assigned to it and to kernel system calls. There is no way (in software at least) to "bypass privilege check". What is it going to do? Access to devices requires a kernel system call. Access other processes' memory will either be blocked by the cpu page table permissions or it requires a kernel system call to request access to the memory. Privileged instructions reserved for the kernel are also blocked by the cpu.

There are some hardware ways to bypass the kernel (search for "row hammer"), but this is probably out of the scope you were thinking.