How the Kernel Isolates Processes

TL;DR: The processor has a privileged mode for the kernel and an unprivileged mode for everything else. Each process sees memory only through page tables the kernel controls, so other processes' memory is simply not addressable. A program asks for privileged work through a system call, a fixed entry point where the kernel validates every argument, especially pointers, before acting with its own authority. Isolation has deliberate doors (debuggers, shared memory, IPC), each with its own access check. All of it depends on the kernel being correct: one kernel bug can break every boundary on the machine.

Two modes, enforced by hardware

In user mode, instructions that would change page tables, disable interrupts or talk to devices directly cause a fault. Only code running in kernel mode can execute them. On x86 the modes are called ring 3 and ring 0; on ARM, exception levels EL0 and EL1. The names differ; the property is the same: a user program cannot simply decide to be privileged.

Virtual memory is the wall

Programs never use physical addresses. Every load and store goes through the memory management unit, which translates virtual addresses using the current process's page tables. Each page table entry carries permissions: present or not, writable or read-only, executable or not, user-accessible or supervisor-only.

When the kernel switches from process A to process B, it switches page tables. B's addresses now map to B's pages. Nothing B can compute reaches A's memory, because A's pages are not in B's tables. Only the kernel edits page tables, so this wall holds as long as the kernel is correct.

When the wall leaks through the CPU

Page tables are enforced by hardware, but in 2018 Meltdown showed that speculative execution on many processors could read kernel memory mapped into a process's address space before the permission check took effect, leaking it through cache timing. The kernel's correctness was not the problem; the CPU's behaviour was. Linux's fix, kernel page-table isolation (KPTI), keeps two sets of page tables per process: while user code runs, the tables map almost none of the kernel, only the entry and exit code and the interrupt table. Switching tables on every entry and exit costs on the order of a hundred cycles each time, plus about 4 KB of extra memory per process. Newer CPUs fixed the underlying behaviour, so KPTI is not needed on them.

The interview point: an isolation boundary is only as strong as every layer beneath it, including the processor.

The system call is the gate

A program that wants to open a file puts the system call number and arguments in registers and executes a special instruction (syscall on x86-64, svc on ARM64). The CPU switches to kernel mode and jumps to an entry point the kernel registered at boot. The kernel then:

1. Validates every argument. Pointers must point into the caller's own user memory, and are copied in once through helpers that handle faults (copy_from_user on Linux). Reading the same user memory twice is a bug: another thread can change it between reads (a double fetch).
2. Checks permissions against the process's credentials at the moment of use.
3. Does the work with kernel privilege and returns a result.

Hardware helps the kernel avoid mistakes: SMEP stops the kernel executing user-space pages, and SMAP stops it reading or writing user memory except inside the deliberate copy routines.

What the kernel tracks per process

The deliberate doors

Isolation would be useless if processes could never cooperate, so the kernel offers doors, each with a lock:

• Debugging (ptrace on Linux, process handles with debug rights on Windows) reads and writes another process's memory. Allowed for the same user, or with a capability such as CAP_SYS_PTRACE or Windows SeDebugPrivilege. Linux's Yama module can restrict even same-user debugging.
• Shared memory and memory-mapped files put the same pages in two address spaces, governed by permissions on the shared object.
• IPC (pipes, sockets, message queues, RPC) passes data, which the receiver must treat as untrusted input.

Two consequences. Processes running as the same user on a desktop are weakly isolated from each other, which is why browsers and mobile platforms put each app or site in its own sandboxed identity. And threads inside one process share everything, so there is no isolation between them at all.

Yama's levels for debugging

Level 1 is a common default on desktop distributions. It means a compromised browser extension host running as you cannot simply attach to your password manager.

fork, exec and what crosses them

A child created with fork gets a copy of the parent's memory, its credentials, its environment variables and all of its open file descriptors. After exec loads a new program, the memory is replaced, but credentials, environment and file descriptors carry over unless a descriptor was marked close-on-exec. Two common leaks follow: a privileged parent passes an open handle to a sensitive file into an untrusted child, and secrets placed in environment variables are inherited by every child process and visible to anything that can read the process's environment. Open sensitive files with O_CLOEXEC, and pass secrets to children deliberately rather than through the environment.

Time of check to time of use in user space

The kernel's double-fetch rule has a user-space twin. A privileged program checks that a path is safe, then opens it. Between the check and the open, an attacker who controls a directory on the path swaps a file for a symbolic link to something sensitive, and the program opens the wrong file with its privileges.

The fix pattern is to check the thing you will use, not a name for it: open first, with flags such as O_NOFOLLOW and relative to a directory handle you already trust (openat), then inspect the open file descriptor with fstat and act only on that descriptor. Newer interfaces such as openat2 with resolve restrictions make this easier.

When the wall falls: the kernel

A user process attacks the kernel through the system calls it can make. A memory-safety bug in kernel code reachable from user space lets the attacker rewrite their own credentials, read any memory or disable security modules. Every container on a host shares the host kernel, so a kernel exploit from inside a container is a container escape.

This is the reasoning behind every sandbox design: reduce the kernel surface a workload can reach (seccomp), and for hostile code put another kernel in the way (gVisor's user-space kernel, or a microVM with its own guest kernel behind a hypervisor).

Is it a security boundary?

Interviewers often ask whether some separation "counts". Microsoft publishes the list it services as security boundaries in Windows: the network, kernel, process, AppContainer sandbox, user, session, web browser, virtual machine and virtualisation-based security boundaries. Just as useful is what it says is not a boundary: administrator to kernel. An administrator who loads code into the kernel has not crossed a boundary in Microsoft's model, and User Account Control is a defence-in-depth feature, not a boundary Microsoft commits to service. The Linux equivalent is that root can usually reach the kernel unless lockdown, Secure Boot policy or similar controls are in force.

Measuring how much kernel a workload can reach

Surface reduction can be counted.

• Default container profile. Docker's default seccomp profile is an allowlist that disables around 44 of more than 300 system calls. Everything else stays reachable.
• Concentrated risk. In June 2023 Google reported that 60% of the 42 Linux kernel exploits submitted to its kCTF vulnerability reward programme used io_uring. Google disabled io_uring on ChromeOS and its production servers and blocked Android apps from it with seccomp.
• What seccomp cannot see. A seccomp filter sees the system call number and the raw register values of its arguments, not the memory those arguments point to. It cannot decide which file paths a process may open; that belongs to a Linux security module or Landlock.
• Locking root out of the kernel. The kernel lockdown feature (Linux 5.4) stops even root from modifying the running kernel through interfaces such as /dev/mem or unsigned modules, and many distributions turn it on under Secure Boot.

Worked example: a code execution service

A service runs agent-generated Python in containers on a shared kernel.

Surface reduction is measurable and worth doing. Whether a shared kernel is acceptable at all is a separate decision driven by how hostile the code is.

Worked example: a model loader

A model-serving process loads third-party model files with a loader that can execute embedded code. If the loader runs inside the server process, that code gets the server's address space, credentials and network: every secret it holds, every database it can reach. Moving the loader into a separate process, running as an unprivileged user with no network, no secrets and a seccomp allowlist, means a malicious model gets a process that can do almost nothing. The kernel boundary is the same; what changed is which process sits on each side of it.

Misconceptions interviewers listen for

• "Admin or root to kernel is a sandbox escape." On Windows, Microsoft says administrator to kernel is not a security boundary.
• "seccomp can control which files a process opens." It cannot read pointed-to memory; path policy belongs to an LSM or Landlock.
• "Threads are isolated from each other." They share one address space.
• "Page tables alone guarantee isolation." Meltdown read kernel memory across them until KPTI.
• "A container is a lightweight VM." It shares the host kernel.

How this shows up in interviews

"What separates kernel space from user space?" and "What happens on a system call?" open many systems interviews, and the follow-ups go straight to containers and sandboxes. Interviewers listen for who enforces the boundary (the CPU and the kernel's page tables), what the kernel must distrust (arguments, especially pointers), and why a shared kernel matters.

Say it in 30 seconds: "The CPU runs apps unprivileged and the kernel privileged. Each process sees memory through page tables only the kernel controls, so other processes' memory isn't addressable. Apps reach privileged operations through system calls, where the kernel validates every argument, especially pointers, and checks credentials. Debuggers, shared memory and IPC are deliberate doors with their own checks. A kernel bug breaks everything at once, which is why containers sharing a kernel are weaker than VMs, and why we use seccomp and gVisor or microVMs for untrusted code."