Fault Escaping: Improving Robustness of DPU Enhanced Platform with Mutual Assisted VM Recovery

Cloud servers now offload virtualization onto a data processing unit (a smart card with its own chip), which is faster but splits each virtual machine's state and the hypervisor's job across two independent parts — the DPU and the host — so a crash in either one takes down the whole machine; this paper (Alibaba — Chao Zhang, Yibin Shen, Jiesheng Wu, and colleagues) proposes fault escaping, treating the two split hypervisors as redundant peers so that when one side fails, the VM's verified state is migrated to the surviving side and keeps running, instead of trying to recover the crashed hypervisor.

The argument in one glance

Problem

Offloading virtualization to a DPU splits the machine into two crash points

To run virtual machines (VMs) efficiently, cloud providers increasingly use a data processing unit (DPU): a smart network card with its own system-on-chip (SoC) that takes over much of the virtualization work — device emulation, network and storage handling — that used to run on the host's CPUs. Offloading this overhead to the DPU frees the host to do more useful work, which is why it boosts performance. But it changes the shape of the machine. In a traditional server, one monolithic hypervisor on the host runs everything, so the machine is a single system with a single point of failure. With a DPU, the hypervisor's functions are split and distributed across the DPU and the host, and each VM's state is likewise spread between them — some on the DPU SoC, some on the host. The server is no longer one system; it is an orchestration of multiple self-managed processing units that must cooperate. The downside is reliability: there are now two independent parts that can fail, the DPU SoC and the host, and an error in either one can crash the entire machine and all its VMs. So the very split that bought performance raised the failure rate, and the existing mechanisms for VM fault tolerance and isolation — designed for a single monolithic hypervisor — no longer fit this two-part world.
What people did

Recover the failed hypervisor in place, as monolithic servers do

The conventional response to a hypervisor failure is to try to recover the failed component in place: detect the fault, restart or repair the crashed hypervisor, and bring the VMs back on the same machine. This mirrors how fault tolerance worked on a traditional monolithic server, where there was one hypervisor to restore.
Why it fell short

In-place recovery does not fit a split, two-failure-domain system

Recovering the failed hypervisor in place is a poor fit for the DPU-enhanced platform. The machine is now two self-managed parts, the DPU SoC and the host, and a failure of either can bring down the whole thing, so the failure surface has grown rather than shrunk. Trying to resurrect a crashed hypervisor on the side that just failed is slow and unreliable — the failed side may not come back cleanly, and while it is being repaired the VMs are down. Worse, because each VM's state is split across the two sides, recovering one side in isolation must somehow reconcile with the state held on the other, which the old monolithic recovery machinery was never built to do. The deeper observation is that the split platform actually contains something a monolithic server did not: a second, independent processing unit that is still alive when the first one dies. In-place recovery ignores that survivor and stubbornly tries to fix the corpse. What was missing was a fault-tolerance design that uses the surviving side of the split system to keep the VM running, rather than depending on reviving the side that failed.
This paper's idea

Treat the two sides as redundant peers and let the VM escape a failure

The paper proposes fault escaping, a scheme built on a reframing: instead of viewing the DPU SoC and the host as two fragile halves of one machine, treat the split hypervisors on them as redundant peers, each capable of standing in for the other. The key enabler is hybrid synchronization: the design keeps the entire VM state fully accessible on either the SoC or the host, but does so with minimal synchronization overhead — on the order of kilobytes — rather than continuously copying everything, because most of the VM's large state (such as its memory) is already reachable or reconstructible and only a small amount of critical state needs to be actively kept in sync. Then, when one side fails, fault escaping does not try to recover the crashed hypervisor at all. Instead, it migrates the verified VM state to the surviving side, and the VM continues running there — it escapes the failure rather than waiting to be rescued from it. Because the full state is already accessible on both peers, this migration is fast and the VM survives a crash of either the DPU SoC or the host, turning the extra failure domain that the DPU introduced from a liability into a backup. (The paper's specific mechanisms and quantitative results are in its full text, which was not available for this write-up; see the note below.)

Why this problem is worth solving

DPU offload is becoming standard in large clouds, and at that scale a higher per-machine failure rate translates into a lot of disrupted VMs.

Data processing units are now a mainstream way to build cloud servers, because offloading virtualization overhead from the host to the DPU meaningfully improves performance and lets providers sell more of each machine’s CPU to customers. But reliability is not optional in the cloud: customers run real workloads on their VMs, and an unexpected crash of the machine takes down every VM on it, which is exactly the kind of event a cloud provider works hardest to avoid. The trouble is that the architectural change that improved performance also increased the probability of such crashes, by creating a second independent component (the DPU SoC) whose failure is fatal to the whole machine. At the scale of a large fleet, even a modest increase in per-machine failure rate means many more customer VMs disrupted over time. So making the DPU-enhanced platform robust is not a niche concern; it is about ensuring that the performance gains of offloading do not come at the cost of the reliability that cloud customers depend on. Getting fault tolerance right here is what lets providers adopt DPUs without degrading the service.

The subtlety is that the split, which caused the problem, also supplies the cure if you use it correctly.

The DPU split turns one machine into two cooperating, independently-failing parts, and the naive reading is that this simply doubles the ways the machine can die. That is true under the old fault-tolerance model, which assumes a single hypervisor to recover in place. But the same split quietly provides something valuable: when one part fails, the other part is usually still alive and functioning. A monolithic server had no such spare — if its one hypervisor died, there was nothing else on the machine to fall back on. The insight is to stop treating the two sides as fragile halves and start treating them as redundant peers, so a failure of one becomes survivable by moving the VM to the other. The obstacle to doing this is that each VM’s state is split across the two sides, so before the surviving side can take over, it needs access to the whole VM state, and keeping a full, continuous copy on both sides would be prohibitively expensive given how large VM state (especially memory) is. The design problem is therefore to make the entire state available on either side cheaply — synchronizing only what truly must be synchronized — so that failover is both possible and fast. Solving that is what converts the extra failure domain from a pure liability into a genuine backup.

What people did before, and where it broke

Recover the failed hypervisor in place. Detect the fault and restart or repair the crashed hypervisor on the same side. Where it broke. The failed side may not come back cleanly and the VMs are down while it is repaired, which is slow and unreliable.
Apply monolithic-server fault tolerance to the split platform. Reuse mechanisms designed for one hypervisor. Where it broke. They assume a single failure domain and a single place to hold state, so they cannot reconcile state that is split across the DPU and the host.
Continuously replicate the full VM state to both sides. Keep a complete live copy everywhere so either side can take over. Where it broke. VM state, especially memory, is very large, so continuously copying all of it would impose prohibitive synchronization overhead.
Accept the higher failure rate as the price of DPU performance. Take the reliability hit in exchange for offloading gains. Where it broke. At fleet scale the extra crashes disrupt many customer VMs, undermining the reliability the cloud depends on.
The gap. No design used the surviving side of the split system as a redundant peer to keep the VM running, with cheap enough synchronization to make failover fast.

How fault escaping works

Turning the DPU's extra failure domain into a backup

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Survive a partial failure by having a peer take over. Fault escaping’s core stance is that when one part of a distributed system dies, the rest should keep the service running rather than the whole machine going down — the DPU SoC and the host back each other up. Designing a system so that a failure of one component is absorbed by the others, keeping the workload alive instead of collapsing, is the same undertaking the work on cooperative graceful degradation in containerized clouds covered elsewhere on this site pursues, keeping the healthy parts serving when one part fails. Fault escaping applies this to the two halves of a DPU-split hypervisor, so a crash of either becomes survivable rather than fatal.

Move the running workload instead of repairing the broken host. Rather than trying to resurrect the crashed hypervisor, fault escaping relocates the VM to the surviving side — it fixes the situation by migration, not by in-place repair. Responding to a failing or degraded location by migrating the running workload elsewhere, instead of struggling to restore the failed spot, is the same principle the work on service migration driven network renascence (NetRen) covered elsewhere on this site is built on, treating migration as the recovery mechanism. Fault escaping migrates the verified VM state to the healthy peer, so the VM continues there while the failed side is left behind.

Build on the DPU offload that made the machine faster. The whole setting exists because virtualization work was offloaded from the host CPUs onto a specialized processing unit, the DPU, for performance. Moving infrastructure work off the general-purpose host and onto a dedicated device to gain efficiency is the same idea the work on in-storage domain-specific acceleration for serverless computing covered elsewhere on this site develops, offloading work to a smart device close to where it belongs. Fault escaping accepts that offload as its foundation and makes the resulting two-part machine reliable, so the performance benefit of the DPU does not have to be paid for in fragility.

Following a failure through fault escaping

A VM surviving a crash of one side of the split

An illustrative sketch, based on the paper's abstract, of how a VM escapes a failure. It is not a detailed mechanism from the full paper, which was not available; specifics should be read there.

Step 1 of 5

A VM runs across the DPU SoC and the host

Under DPU offload, the VM's state and the hypervisor's functions are split across the DPU SoC and the host, each a self-managed processing unit that handles part of the virtualization.

This split is what gives performance but adds a second failure domain.

Why it works

The surviving peer is a spare a monolithic server never had. A single hypervisor has nothing to fall back on when it dies → but the DPU split leaves a second, independent processing unit alive → so a failure of one side can be survived by the other.
Migrating beats reviving the failed side. Restarting a crashed hypervisor in place is slow and may not succeed → but moving the VM to the healthy peer sidesteps the broken side entirely → so the VM comes back faster and more reliably.
Hybrid synchronization makes failover affordable. Continuously copying the whole large VM state to both sides would be prohibitive → but synchronizing only the small critical part while the rest stays reachable → keeps the full state available on both peers for kilobytes of overhead.
Full state on both peers makes migration fast. A failover that first had to fetch the whole state would be slow → but because hybrid synchronization already made the state accessible on the survivor → the verified state can be migrated and resumed quickly.
Reframing the split turns a liability into a backup. Two failure domains naively means more ways to crash → but treating them as redundant peers → converts the extra domain into redundancy that improves robustness instead of harming it.

What it costs (trade-offs and scope)

It needs the two sides to be able to back each other up. Fault escaping assumes the surviving side can host the VM and access its full state; this requires the hybrid-synchronization machinery and enough capability on each peer to take over, which is design and runtime complexity a monolithic server avoids.
Some synchronization overhead is unavoidable. Even minimized to kilobytes, keeping the critical state consistent on both sides costs some continuous work during normal operation, in exchange for the ability to fail over — a small but non-zero tax.
It targets DPU-enhanced platforms specifically. The scheme is built around the particular split of VM state and hypervisor functions between a DPU SoC and a host; a traditional monolithic server, or a different offload architecture, would not have the same two-peer structure to exploit.
Migration must preserve correctness. The approach migrates verified VM state so the VM resumes correctly on the survivor; ensuring the migrated state is complete and consistent after a partial failure is a correctness-critical part of the design, which the paper details.
This explainer is drawn from limited source material. The full paper text was not accessible for this write-up; the description here is based on the paper’s abstract, so the precise synchronization and migration mechanisms, the isolation guarantees, and the quantitative results should be confirmed in the paper.

How it compares

VM fault tolerance on a DPU-enhanced server

Property	Monolithic recovery	Full continuous replication	Fault escaping
Uses the surviving side
Synchronization overhead
Recovery approach
Fits the split platform
Outcome on a one-side failure

A qualitative comparison, based on the paper's abstract, of VM fault-tolerance approaches on a DPU-enhanced server. Fault escaping treats the DPU SoC and the host as redundant peers, keeps the full VM state accessible on both with only kilobytes of synchronization, and on a failure migrates the verified VM state to the survivor rather than recovering the crashed hypervisor — so the VM escapes the failure. Specific mechanisms and quantitative results should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract, not its full text, which was not accessible for this write-up (the ACM version was not retrievable and no author preprint or open copy was found). The problem and approach are stated by the authors: modern cloud servers offload virtualization overhead onto data processing units (DPUs), which splits VM state and distributes hypervisor functions across the DPU and the host, transforming the server from a single system into an orchestration of multiple self-managed processing units; and because an error from either the DPU or the host can crash the entire machine, the failure rate rises, requiring a revamp of VM fault tolerance and isolation. The paper proposes treating the split hypervisors on the DPU SoC and the host as redundant peers, and a fault-escaping scheme that enables VMs to escape from a failing SoC or host: with hybrid synchronization, the entire VM state becomes fully accessible on either the SoC or the host with minimal synchronization overhead (on the order of kilobytes), and instead of recovering the failed hypervisor, the scheme migrates the verified VM state so the VM escapes the failure. The abstract available for this write-up ends there; the paper’s specific synchronization and migration mechanisms, its isolation guarantees, its handling of correctness under partial failure, and all quantitative results were not available and are not asserted here. They should be read from the paper. This page is marked as a draft for that reason.

Background

Why splitting systems for performance keeps reshaping fault tolerance

It helps to see fault escaping within a recurring pattern: performance pushes systems to split apart, and every split forces fault tolerance to be redesigned. For decades a server was one machine with one operating system, and reliability meant keeping that one system up or restarting it. As systems have disaggregated — across cores, across accelerators, across networks, and now across a host and a data processing unit — the number of independently-failing parts has grown, and the old assumption of a single failure domain no longer holds. The naive consequence is that more parts means more ways to fail, and indeed each split tends to raise the raw failure rate. But splits also create something the monolith lacked: when one part dies, others are still alive and can carry the load, if the system is designed to let them. The productive response, seen again and again, is to stop treating the pieces as a fragile chain and start treating them as peers that can cover for one another, which turns added components from pure risk into potential redundancy. Doing so hinges on making the necessary state cheaply available where the takeover must happen, without paying to replicate everything. The broader craft of keeping services running through disruption without taking the system down is the same terrain the work on efficient scheduler live update for the Linux kernel covered elsewhere on this site occupies, sustaining a running system through change rather than restarting it. Fault escaping brings this discipline to the DPU-split server, arguing that the second processing unit the offload introduced should be embraced as a backup, so the machine tolerates a partial failure by moving the VM to safety instead of crashing.

If you remember only three things

1. Cloud servers offload virtualization onto data processing units (DPUs) for performance, which splits each VM's state and the hypervisor's functions across the DPU's system-on-chip and the host. This turns one machine into two cooperating, independently-failing parts, and because an error on either the DPU or the host can crash the whole machine, the failure rate rises. Monolithic fault tolerance — recovering a single crashed hypervisor in place — no longer fits, because the failed side may not come back cleanly and each VM's state is now split across two sides.
2. The paper proposes fault escaping: reframe the DPU SoC and the host as redundant peers that can back each other up. Hybrid synchronization keeps the entire VM state accessible on either side while actively synchronizing only a small, critical part (on the order of kilobytes), because most large state such as memory is already reachable or reconstructible. When one side fails, fault escaping does not recover the crashed hypervisor; it migrates the verified VM state to the surviving peer, and the VM keeps running there — it escapes the failure. This converts the extra failure domain the DPU introduced into a backup.
3. The trade-offs and caveats: the approach needs the hybrid-synchronization machinery and enough capability on each peer to take over, pays a small continuous synchronization tax, targets DPU-enhanced platforms specifically, and must preserve correctness when migrating state after a partial failure. This explainer is a draft based on the abstract; the precise synchronization and migration mechanisms, isolation guarantees, and quantitative results should be confirmed in the paper.

Check yourself

Check your understanding

1. Why does offloading virtualization to a DPU raise a cloud server's failure rate?
2. What is the core reframing behind "fault escaping"?
3. How does the scheme keep the full VM state available on both sides without huge overhead?
4. Predict: the DPU SoC of a server crashes while a VM is running. Under fault escaping, what happens, and why is this better than in-place recovery?

Sources

The paper is “Fault Escaping: Improving Robustness of DPU Enhanced Platform with Mutual Assisted VM Recovery” by Chao Zhang, Tao Xu, Junming Liu, Pai Liu, Zhilang Xu, Kang Yan, Shuo Shi, Jintao Li, Jinhu Li, Chen Liang, Wenhui Shu, Feifei Fan, Yibin Shen, Hang Yang, Jianming Song, Xudong Zheng, Jiesheng Wu, and Jian Li (Alibaba), published at ASPLOS 2025, DOI 10.1145/3676642.3736124.

This explainer is written from the paper’s abstract, because the full paper text was not accessible for this write-up (the ACM version was not retrievable and no author preprint or open copy was found). The authors state that modern cloud servers offload virtualization overhead onto data processing units (DPUs), which splits VM state and distributes hypervisor functions across the DPU and the host, transforming the server from a single system into an orchestration of multiple self-managed processing units; and that because errors from either the DPU or the host can crash the entire machine, the failure rate has significantly increased, necessitating a comprehensive revamp of VM fault tolerance and isolation. The paper explores treating the split hypervisors on the DPU SoC and the host as redundant peers, and proposes a fault-escaping scheme that enables VMs to escape from a failing SoC or host: with hybrid synchronization, the entire VM state becomes fully accessible on either the SoC or the host with minimal synchronization overhead (on the order of kilobytes), and instead of recovering the failed hypervisor, the scheme migrates the verified VM state so the VM escapes the failure. The specific synchronization and migration mechanisms, isolation guarantees, and quantitative results were not available for this write-up and are not asserted here; they should be confirmed in the paper. This page is marked draft because the full text was unavailable.