Protect the System Call, Protect (Most of) the World with BASTION

An attacker who has corrupted memory in a running program still has to do something with it, and almost always that means reaching a system call — the only way to affect anything outside the process; existing defences guard the ROUTES an attacker might take, checking every control transfer or every memory write, which is expensive enough to be rarely deployed and evadable by finding a route the defence does not model, such as corrupting an array index rather than a pointer; this paper (Virginia Tech and Oregon State) guards the DESTINATION instead, running three checks only when a sensitive system call is attempted, and finds that in three real server programs such calls are never legitimately made through a function pointer at all.

Why this problem is worth solving

The reason a system call is the right place to stand is worth making precise.

The existing defences guard the routes instead, and both of their weaknesses follow from that choice.

What people did before, and where it broke

• Debloat the binary. Static analysis or test coverage identifies code the program never runs, and it is removed, eliminating the system calls that code would have made. Genuinely reduces the attack surface.
• But the calls attackers want are the ones nobody can remove. Memory permission calls are used by the dynamic loader to map libraries, so they survive debloating in essentially every program.
• Filter system calls per process. Seccomp lets an administrator allow or deny each call for a process, and even pin an argument to a constant. Widely deployed and cheap.
• But a per-process decision has no notion of context. A call needed anywhere is permitted everywhere, and an argument restriction must permit the union of every legitimate use — so a program that legitimately maps memory read-only in one place and executable in another must permit both, everywhere.
• Enforce control-flow integrity across the program. Check every indirect transfer against a statically computed set of legal targets.
• But precision and cost trade against each other, and it is not enough anyway. Coarse versions leave sets large enough to exploit; exact versions need heavy instrumentation. And an attacker facing perfect control-flow enforcement can simply corrupt the arguments instead.
• Enforce data-flow integrity across the program. Instrument every load and store so a read only ever sees a value a legitimate write produced.
• But instrumenting every memory access is very expensive, and effectiveness depends on the accuracy of the pointer analysis behind it.
• The gap. Every defence was placed along the routes an attacker might travel, which means paying everywhere and still missing routes nobody modelled. Nobody had asked what could be checked at the one point every attack must reach — where the check runs rarely, and where the attacker has run out of alternatives.

How three checks at one point replace three defences everywhere

The idea

Stand where the attacker must arrive, not where they might travel. Every defence in the comparison is placed along a route: control-flow integrity on transfers, data-flow integrity on memory accesses, pointer integrity on pointers. That has two consequences. The cost is paid continuously, because routes are travelled constantly. And the protection is only as complete as the enumeration of routes, which is why one published attack works by corrupting an array INDEX — not a pointer, so nothing guarding pointers reacts, yet the lookup runs past the end of a function-pointer table and lands on a system call. A check at the system call itself is indifferent to how the attacker got there. It also runs astonishingly rarely: the authors found that NGINX makes its sensitive system calls almost entirely during startup and hardly ever while serving requests, so the monitor is barely triggered at steady state. That is the whole reason three expensive-sounding checks cost under one percent. Choosing to act at the point where a problem becomes unavoidable, rather than everywhere it might originate, is the same intervene-at-the-chokepoint reasoning covered elsewhere on this site where a design identifies the moment at which harm becomes possible and does its work only there.

Each check is a known defence with its scope collapsed. The three contexts are not new inventions so much as existing techniques narrowed until they become affordable. Checking how a call was made is system-call filtering, but per call site instead of per process, distinguishing not merely allowed from forbidden but callable-directly from callable-by-pointer from never-callable. Checking how control arrived is control-flow integrity, but evaluated only along paths that reach a system call and only at the moment one is attempted, by walking the real stack rather than instrumenting every branch. Checking the arguments is data-flow integrity, but restricted to system call arguments and the variables they derive from — and checking VALUES against a shadow copy rather than tracking flows. The measurements show why the narrowing matters: instrumentation reaches a maximum of 5,287 points in NGINX, and stacks at system call sites average 5.2 frames deep, so unwinding them is cheap. Taking a technique that is correct but unaffordable at full scope and making it affordable by restricting where it applies is the same shrink-the-domain-not-the-guarantee move covered elsewhere on this site where an analysis becomes tractable by being asked a narrower question.

The three checks are not redundant, and the paper’s table proves it. It would be easy to assume the argument check subsumes the others, or that control-flow checking makes call-type checking unnecessary. The evaluation shows otherwise, and it is the most informative result in the paper. Classic return-oriented attacks slip past the call-type check, because they reach calls the program does use in the way it uses them, and are caught by the other two. One attack against Apache slips past the call-type check because that program does legitimately call an exec function through a pointer, and is caught by the control-flow check. And three of the most sophisticated techniques — including one against Chrome that builds its attack entirely from the program’s own legitimate control flow — slip past both, and are caught only by argument integrity. Each layer earns its place by catching something the others cannot. Demonstrating that the components of a defence are individually necessary, rather than asserting defence in depth, is the same show-your-ablation discipline covered elsewhere on this site where a paper’s own decomposition reveals which parts are actually load bearing.

An attack that never corrupts a pointer

the call site reads roughly as calling the handler stored at position index in the array, with the request and two other values from the same entry
the attacker is assumed to have a memory corruption bug giving arbitrary read and write, with the usual defences already deployed

Why it works

• Every attack must reach a system call to affect anything outside the process → a check placed there is indifferent to the route taken → so an attack that corrupts an array index rather than a pointer is caught the same as one that does not.
• Sensitive system calls are used almost entirely during startup and rarely while serving requests → the monitor is triggered very seldom at steady state → giving 0.60 percent overhead on NGINX despite doing control-flow and data-integrity work.
• Control flow is verified by unwinding the real stack at the call rather than instrumenting every branch → the cost scales with stack depth at system calls, averaging 5.2 frames → instead of with the number of branches the program takes.
• Only variables that become system call arguments are tracked → instrumentation totals at most 5,287 points → rather than every load and store as whole-program data-flow integrity requires.
• The monitor runs as a separate process reached through kernel-level trapping → an attacker controlling the application cannot disable the checks → so full control of the target does not translate into evading them.

What it costs (trade-offs and scope)

• It protects twenty calls, not all of them. The set is chosen to cover arbitrary execution, memory permissions, privilege escalation and networking, and is extensible, but an attack achieving its goal through a call outside the set is outside the design.
• It assumes several defences are already deployed. The threat model requires data execution prevention, address space layout randomisation and a hardware shadow stack, and trusts the kernel and hardware. Attacks against those are out of scope.
• Compile-time analysis is required, so source is needed. This is a compiler pass plus a runtime monitor, not something applied to an existing binary.
• The response to any violation is to kill the program. Appropriate for a security monitor and consistent with how such defences behave, but it means a false positive is an outage. The authors argue their construction admits neither false positives nor false negatives.
• The evaluation is three server programs. They are well chosen — widely deployed, frequently attacked and system-call heavy — but the low overhead depends on the pattern of using sensitive calls mostly at startup, which may not hold for every workload.

The evaluation

Two questions were asked. Against attacks, the system was tested on seven real vulnerabilities in ffmpeg, php, sudo, nginx, libtiff and python, plus five published advanced techniques designed specifically to defeat modern defences, including one that bypasses pointer integrity and one built entirely from a program’s own legitimate control flow. For cost, NGINX, SQLite and vsftpd were measured with each context enabled in turn, on an eight-core AMD machine with address space layout randomisation and the hardware shadow stack active, averaged over five runs.

Check yourself

Sources

• Protect the System Call, Protect (Most of) the World with BASTION (author PDF) — ASPLOS 2023; the full 14-page paper, read in full for this explainer. Published open access under a Creative Commons licence.
• ACM Digital Library (DOI 10.1145/3582016.3582066) — published version.