I really wish they would have expanded on this paragraph in their article further:
> The configuration used here had no jailer. Firecracker's production containment includes dropping privileges and restricting the process through namespaces and seccomp. A deployment using those controls requires a separate assessment of the privileges and resources available after compromising its VMM.
If I'm understanding all of this correctly, a properly jailed and configured firecracker instance would greatly reduce the blast radius of this escape, and I feel as though it would have been good for them to explain what jailing and dropping privileges would do to help. I'd hope most production firecracker deployments would be doing this already.
Not saying that this is any less severe and the root issue is not specific to Firecracker, but to a flaw in the kernel's network code -- they just used an insecure firecracker configuration to demonstrate it. However, I feel like some are taking this as a flaw with firecracker when its not. This is an issue in the kernel networking stack.
That's a good catch. Yes, since they've only proven this up to the jailer boundary, you'd need a second and relatively powerful LPE vulnerability to get real host control out of this.
I am more convinced by the day that the Nova Microhypervisor approach is correct.
Do virtualization with a tiny formally verified microkernel and have complex kernels like Linux run underneath instead of letting a huge, complex, and brittle linux kernel and vm ecosystem control the hypervisor layer.
How is this different from Xen? It's the same approach as Xen or Hyper-V where the hypervisor is its own component, then you boot into a virtualized Dom0 or Root Partition (in Xen and Hyper-V terminology respectively) that can manage the hypervisor underneath it.
This is different from KVM where your hypervisor runs in your Dom0/Root Partition.
Yeah, all security design guarantees are as good as the implementation at the end of the day. All the fancy hashing and encryption math in the world isn't gonna save you from an erroneous `"" == ""` check or an invalid reference check.
I was replying to OP saying it's the same "approach" as Xen, if your preference is "approach" based. It's fair though to point to Nova's smaller code base. It's becoming an important metric for me in general (how big is the code base). It's not a bullet proof thing, but having a smaller code base is a positive sign in 2026 for me.
This is a much lighter weight (~100k) alternative to Xen focused on baremetal TPM anchored remote attestation, formal verification, and trusted computing applications.
Anyone considering Xen should probably also consider Nova depending on requirements.
Is there a well tested L4 hypervisor solution with hardware accelerated virtualization and raw TPM support formally verified to a similar or greater level than Nova with similar remote attestation and security features?
Most of the L4s have hypervisor situations. I don't know about TPM support. Generally the vibe I get from systems security people is a bit of eye-rolling about the formal verification of things like L4; the real win is just drastic attack surface reduction.
And then, of course, L4s live in millions and millions of shipped products, which is a reason to consider L4 instead of something novel, but I'm not saying you should or shouldn't.
I for one do not want any code in the TCB that is not formally verified, tiny, and easy to audit with every line of code justified.
SeL4 is the only open, mature, and formally verified L4 solution I am aware of that can run on modern servers, but IIRC given vCPUs are not first class objects and any low level CPU virtualization features are handled in user space, it comes at a real performance cost making it only suitable for single-core or non-performance-critical applications.
Nova by contrast was built to handle the expensive and trust critical TPM and vCPU handling directly for high performance remotely attestable multi-core applications without having to reach for something as heavy as Xen, though formal verification is admittedly not as complete as SeL4 yet AFAICT, though lots of recent progress here.
Those are pretty big tradeoffs, but if you or anyone else knows other viable xen alternatives this general shape with a tiny easily-auditable footprint, please chime in!
Like some of what I'm saying is just that if you tell a systems security person "we're using seL4 because of its formal verification", you're going to get eye rolls. Whatever else you want to say about it, the formally-verified configuration of seL4 doesn't actually do much, meaning whatever system you're building on top of it is going to be where all the security drama is anyways.
I'm certainly not telling you to use Xen. I wouldn't use Xen for anything untrusted.
> the formally-verified configuration of seL4 doesn't actually do much
As a systems security engineer myself I would disagree and know several others that would as well. All depends on your circles, applications, and threat model I suppose.
Xen, Hyper-V, and KVM have all had major security flaws in the form of logic violations, spacial memory corruptions, use-after-free, and pointer corruptions that allowed hypervisor trust boundary violations. I expect most if not all of those classes of bugs could have been prevented by formal verification.
Xen likely has a bad reputation with both of us because of a history of preventable flaws like these.
In my applications every single daemon gets its own VM, some with network, some without network, different filesystem and vsock access for maximum damage control and minimal trust between daemons. The hypervisor is the most important security boundary to me especially when isolating user-controlled workloads. The more confidence I can get that it is correct the better.
This is one of the most useful web tools I've ever seen.
I work on connectors for datasources (mostly DB's. Being able to quickly test things like sorting behavior and how the same function can be emulated across DB's would have made my life so much easier.
Running our microVM's inside of a jailed zone, at least gives me some peace of mind.. It's far from perfect, as a zone escape could be lurking.. but at least it's a layer of defense.
As another commenter noted, the PoC in this article doesn't defeat the Firecracker jailer, which does essentially what a "zone" would do for you. In both cases, that's cold comfort, because even in an unprivileged sandbox environment you still have a huge kernel attack surface to contend with LPEs in.
This is not a new vulnerability, it's just a description of how to exploit the PwnEscape vulnerability in practice on firecracker VMs (disclosed multiple weeks ago and discussed in some less upvoted posts on HN)
The actual vulnerability got very little discussion. It's clever. Because Linux networking people are obsessed with checksum performance, virtio-net plumbs checksum offload metadata through the host/guest driver interface. Guests can thus deliver the host packets along with instructions on what to checksum. The vulnerability is a missing sanity check that allows checksum instructions to instruct the host to overwrite the header length field in the IPv4 header, rather than the checksum field.
Crucially, this happens in the middle of packet processing, so the kernel code is forced to work with two different ideas of how long the IPv4 header is. All hell predictably breaks loose.
Additional color: per the commit, the bug was found by Paulos Yibelo, who runs PwnAI (the author of this piece), and worked on security at AWS for 4 years prior.
It's a settled enough part of HN that you'd do better emailing [email protected] about it than you will litigating it in public. I don't care either way, I'm just explaining the behavior to you.
I posted a similar query on the Replit sandbox engine article, but..
How do people feel about the hardware isolation provided by microVMs versus the stronger end of software isolation techniques like gvisor?
Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up leaning more on patchable software isolation, similar to the rise of software RAID (though, in userspace this time).
You need something more interesting to say than this, because this is an article by the discoverers of this vulnerability, the first general-purpose Firecracker escape that doesn't depend on nested virtualization in recent memory, along with in-depth discussion of exploitability. It's inherently interesting and on-topic and there isn't an alternate source for the same content.
> The configuration used here had no jailer. Firecracker's production containment includes dropping privileges and restricting the process through namespaces and seccomp. A deployment using those controls requires a separate assessment of the privileges and resources available after compromising its VMM.
If I'm understanding all of this correctly, a properly jailed and configured firecracker instance would greatly reduce the blast radius of this escape, and I feel as though it would have been good for them to explain what jailing and dropping privileges would do to help. I'd hope most production firecracker deployments would be doing this already.
Not saying that this is any less severe and the root issue is not specific to Firecracker, but to a flaw in the kernel's network code -- they just used an insecure firecracker configuration to demonstrate it. However, I feel like some are taking this as a flaw with firecracker when its not. This is an issue in the kernel networking stack.
Do virtualization with a tiny formally verified microkernel and have complex kernels like Linux run underneath instead of letting a huge, complex, and brittle linux kernel and vm ecosystem control the hypervisor layer.
https://hypervisor.org/
This is different from KVM where your hypervisor runs in your Dom0/Root Partition.
I was replying to OP saying it's the same "approach" as Xen, if your preference is "approach" based. It's fair though to point to Nova's smaller code base. It's becoming an important metric for me in general (how big is the code base). It's not a bullet proof thing, but having a smaller code base is a positive sign in 2026 for me.
He is an ex Intel CPU engineer and the depth of his thought into the security design had me questioning my life choices.
Anyone considering Xen should probably also consider Nova depending on requirements.
SeL4 is the closest contender on my radar atm.
And then, of course, L4s live in millions and millions of shipped products, which is a reason to consider L4 instead of something novel, but I'm not saying you should or shouldn't.
SeL4 is the only open, mature, and formally verified L4 solution I am aware of that can run on modern servers, but IIRC given vCPUs are not first class objects and any low level CPU virtualization features are handled in user space, it comes at a real performance cost making it only suitable for single-core or non-performance-critical applications.
Nova by contrast was built to handle the expensive and trust critical TPM and vCPU handling directly for high performance remotely attestable multi-core applications without having to reach for something as heavy as Xen, though formal verification is admittedly not as complete as SeL4 yet AFAICT, though lots of recent progress here.
Those are pretty big tradeoffs, but if you or anyone else knows other viable xen alternatives this general shape with a tiny easily-auditable footprint, please chime in!
I'm certainly not telling you to use Xen. I wouldn't use Xen for anything untrusted.
As a systems security engineer myself I would disagree and know several others that would as well. All depends on your circles, applications, and threat model I suppose.
Xen, Hyper-V, and KVM have all had major security flaws in the form of logic violations, spacial memory corruptions, use-after-free, and pointer corruptions that allowed hypervisor trust boundary violations. I expect most if not all of those classes of bugs could have been prevented by formal verification.
Xen likely has a bad reputation with both of us because of a history of preventable flaws like these.
In my applications every single daemon gets its own VM, some with network, some without network, different filesystem and vsock access for maximum damage control and minimal trust between daemons. The hypervisor is the most important security boundary to me especially when isolating user-controlled workloads. The more confidence I can get that it is correct the better.
To save people the time:
"This code is experimental and not feature complete. If it breaks, you get to keep both pieces."
See his latest talk which represents a lot of impressive recent work: https://cfp.3mdeb.com/media/bsmconf/submissions/KW3SYT/resou...
I am working on a production OS using it as a foundation.
https://github.com/udosteinberg/NOVA/commit/9e35d89c0e7014f1...
It hosts hundreds of database engines - many of them have vulnerabilities, and some of them, like Pandas, are basically open RCEs regardless.
We can PoC the Firecracker VM Escape on that service.
I work on connectors for datasources (mostly DB's. Being able to quickly test things like sorting behavior and how the same function can be emulated across DB's would have made my life so much easier.
Running our microVM's inside of a jailed zone, at least gives me some peace of mind.. It's far from perfect, as a zone escape could be lurking.. but at least it's a layer of defense.
Couldn't find it. Care to provide link.
Crucially, this happens in the middle of packet processing, so the kernel code is forced to work with two different ideas of how long the IPv4 header is. All hell predictably breaks loose.
https://github.com/torvalds/linux/commit/6785011f8b16abf40b1...
https://www.pwn.ai/blog/pwnescape
(Doesn't HN usually deduplicate links? I'm confused)
How do people feel about the hardware isolation provided by microVMs versus the stronger end of software isolation techniques like gvisor?
Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up leaning more on patchable software isolation, similar to the rise of software RAID (though, in userspace this time).