Beyond Good and Evil: Formalizing the Security Guarantees of Low-Level Compartmentalization

Compartmentalization is widely regarded as good security-engineering practice: if we break up a large software system into mutually distrustful components that run with minimal privileges, restricting their interactions to conform to well-defined interfaces, we can limit the damage caused by low-level attacks such as control-flow hijacking. But the formal guarantees provided by such low-level compartmentalization have seen surprisingly little investigation. We propose a new property, secure compartmentalization, that formally characterizes the security guarantees provided by low-level compartmentalization and clarifies its attacker model. We rationally reconstruct the secure compartmentalization property starting from the well-established notion of fully abstract compilation, by identifying and lifting three important limitations that make standard full abstraction unsuitable for compartmentalization. The connection to full abstraction allows us to prove secure compartmentalization for language implementations by adapting established proof techniques, we illustrate this for a simple unsafe imperative language with procedures and a compiler from this language to a compartmentalized abstract machine.