Vela: A Virtualized LLM Training System with GPU Direct RoCE

Apoorve Mohan, Robert Walkup, Bengi Karacali, Ming-Hung Chen, Abdullah Kayi, Liran Schour and 19 others

Training a large language model spreads the work across hundreds or thousands of GPUs that must exchange data constantly, so AI clusters have traditionally been run on bare metal because the overhead of virtualization was assumed to ruin performance; this experience paper from IBM shows the opposite — by combining standard Linux virtual machines with hardware features that let a VM's network card talk directly to GPU memory and carry remote-memory traffic over ordinary Ethernet, their Vela system runs large-model training inside VMs at roughly 80 percent of ideal efficiency across about 1,500 GPUs, using entirely off-the-shelf hardware and software.

The argument in one glance

Problem

Big-model training needs fast GPU communication, but clouds want VMs

Training a large language model splits work across many GPUs that must constantly exchange data, so performance depends on fast, low-latency communication between them, even across machines. Cloud providers want to deliver this in virtual machines for isolation and manageability.
What people did

Run AI clusters on bare metal

Because virtualization was assumed to add too much overhead for communication-heavy workloads, high-performance AI and scientific clusters have traditionally run directly on bare-metal hardware, without a hypervisor.
Why it fell short

Bare metal gives up the cloud, and the VM tax seemed too high

Bare metal sacrifices the isolation, security, multi-tenancy, elasticity, and manageability of virtual machines. But naive virtualization mediates every network packet through the hypervisor and forces GPU-to-network data through host memory and the CPU, which was thought to make large-scale training in VMs impractically slow.
This paper's idea

Virtualize with hardware passthrough and GPU-direct networking (Vela)

Vela, per its abstract, builds a cloud-native LLM training system from off-the-shelf hardware, Linux KVM virtualization, and a virtualized RDMA-over-Converged-Ethernet network, with VMs that support peer-to-peer DMA directly between GPUs and an SR-IOV network interface. On an IBM Cloud A100 deployment it reached about 80 percent of ideal throughput training a 50-billion-parameter model across about 1,500 GPUs.

Why this problem is worth solving

Training modern large language models is one of the most communication-intensive computations in all of computing, which is why the infrastructure it runs on matters so much.

A large language model is too big and too compute-heavy to train on a single GPU, so the work is spread across many GPUs using distributed strategies — splitting the data across replicas, or splitting the model itself across devices (model parallelism) when it does not fit. Whatever the split, the GPUs cannot work in isolation: at every step they must exchange large amounts of data, such as gradients that have to be summed across all replicas, or activations that flow between the pieces of a partitioned model. This communication happens not only between GPUs in one machine but across machines over the network, and because every GPU often has to wait for this exchange to finish before proceeding, the speed of the whole training job is gated by how fast and how low-latency that communication is. A cluster of thousands of fast GPUs delivers its potential only if the network connecting them keeps up; if communication stalls, the expensive GPUs sit idle. This is why the design of the interconnect and the data path between GPUs and the network is a first-order concern for AI infrastructure.

The desire to deliver this in a flexible cloud collides with a long-held belief that virtualization is too costly for it.

Cloud computing runs workloads inside virtual machines because virtualization brings enormous operational benefits: strong isolation between tenants, security boundaries, the ability to pack multiple users onto shared hardware, elasticity to grow and shrink, and uniform management and provisioning. Yet high-performance computing and large-scale AI training have historically run on bare metal — directly on the hardware with no hypervisor — out of a well-founded fear of virtualization overhead. The concern is specifically about the parts that communication-heavy training hammers. In a naively virtualized system, the hypervisor sits in the path of network I/O, inspecting and forwarding packets, which adds latency and consumes CPU. And getting data from a GPU onto the network traditionally means copying it from GPU memory into host memory and having the CPU shepherd it to the network card, a detour that wastes bandwidth and adds delay — doubly bad when a hypervisor is also involved. For a workload where thousands of GPUs are constantly exchanging data and any stall idles them all, even a few percent of overhead per operation was thought to compound into an unacceptable loss, which is why the question of whether serious AI training could live in the cloud at all remained open.

What people did before, and where it broke

Run AI and HPC clusters on bare metal. Give training jobs direct, un-virtualized access to the hardware to avoid any hypervisor overhead. Where it broke. This forfeits the isolation, security, multi-tenancy, elasticity, and manageability that make cloud infrastructure attractive and operable at scale.
Virtualize naively, with the hypervisor mediating I/O. Put training in ordinary VMs where the hypervisor handles network traffic. Where it broke. Mediating every packet through the hypervisor adds latency and CPU cost that is punishing for the constant, latency-sensitive communication of distributed training.
Move GPU data through host memory and the CPU. Copy data from GPU memory to host memory and let the CPU send it to the network card. Where it broke. This detour wastes bandwidth and adds delay on the critical GPU-to-GPU path across machines, becoming a bottleneck at scale.
Assume the cloud simply cannot host large-scale training. Conclude that the virtualization tax rules out serious AI training in VMs. Where it broke. It leaves AI infrastructure locked to specialized bare-metal deployments, out of reach for a broader community that would benefit from cloud delivery.
The gap. No widely-shared design showed that off-the-shelf hardware and standard virtualization could host large-scale model training near bare-metal efficiency. Demonstrating that, and reporting how, is what Vela does.

How Vela works

From bare-metal AI clusters to virtualized training at near-native speed

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Virtualize for the cloud’s benefits, but pass the hardware through to keep the speed. Vela’s central bet, as described in its abstract, is that you do not have to choose between the operability of virtual machines and the performance of bare metal: you can have the VMs and still hand the performance-critical hardware almost directly to them. Rather than let the hypervisor mediate the network, Vela uses SR-IOV, a hardware feature by which one physical network card presents itself as several virtual ones that a VM can drive nearly directly, so the guest gets close-to-native network performance while remaining a managed, isolated cloud VM. Giving a virtual machine fast, direct, isolated access to a physical device instead of routing everything through the hypervisor is the same move the exit-less VM access work covered elsewhere on this site makes, letting a guest reach hardware on the fast path so that virtualization keeps its isolation without paying the usual mediation cost on every operation.

Cut the CPU and host memory out of the GPU-to-network path. The other half of the performance story is the data path between a GPU and the network. Naively, sending a GPU’s data to another machine means copying it into host memory and having the CPU push it to the network card — a detour that wastes bandwidth and adds latency on exactly the path distributed training stresses most. Vela’s VMs support peer-to-peer DMA directly between the GPUs and the SR-IOV network interface, so data flows straight from GPU memory onto the wire, GPU-direct, even inside a virtual machine. Building a direct, low-overhead path for moving data between the accelerators of a GPU cluster, so communication does not bottleneck the computation, is the same concern the GPU-cluster communication work covered elsewhere on this site addresses, optimizing how data travels between GPUs across a cluster so that the interconnect keeps the expensive accelerators fed rather than leaving them waiting.

Prove it at the scale where large models actually train. An infrastructure claim only matters if it holds when thousands of GPUs are working together on a real model, because that is where communication overheads compound and where the virtualization tax, if it were fatal, would show up. Vela is reported not as a small prototype but as a system operated in an IBM Cloud data center over roughly two and a half years and evaluated at around 1,500 GPUs, training a 50-billion-parameter decoder model with model parallelism at about 80 percent of ideal throughput. Validating a distributed-training system at genuine cluster scale, where the interactions and inefficiencies of large-scale parallelism become visible, is the same discipline the large-model training work covered elsewhere on this site follows, demonstrating its approach on real large models across many devices rather than assuming that what works small will hold when the parallelism and communication grow.

Following a training job through Vela

Running distributed LLM training inside Vela's VMs

A conceptual trace based on the paper's abstract and standard virtualization and RDMA background; specific implementation details were not available.

Step 1 of 5

Provision VMs on off-the-shelf GPU servers

Vela provisions Linux KVM virtual machines on ordinary GPU servers in an IBM Cloud data center, each VM getting GPUs and a network interface, with all the isolation and manageability of cloud infrastructure.

The hardware and hypervisor are standard, not specialized.

Why it works

SR-IOV removes the hypervisor from the packet path. Because the VM drives a virtual function of the network card almost directly → the hypervisor no longer mediates every packet → so the network latency and CPU cost that made virtualization seem too slow are largely avoided.
Peer-to-peer DMA removes the host-memory detour. Because data moves straight between GPU memory and the network card → it does not have to be copied through host memory and shepherded by the CPU → so the GPU-to-GPU path across machines keeps its bandwidth and low latency.
RoCE brings RDMA to standard Ethernet. Because remote direct memory access runs over ordinary Converged Ethernet → machines can read and write each other’s memory with minimal CPU involvement on commodity networking → so the cluster gets low-overhead communication without exotic hardware.
Off-the-shelf parts make it broadly reproducible. Because Vela uses ordinary hardware and standard KVM virtualization → the design is not tied to bespoke appliances → so the demonstrated capability is accessible to a broader community, which is the paper’s stated aim.
Near-ideal scaling shows the tax is manageable. Because about 80 percent of ideal throughput is sustained at roughly 1,500 GPUs → the compounded virtualization overhead did not prove fatal at scale → so cloud-hosted large-model training is shown to be practical, not just possible.

What it costs (trade-offs and scope)

This explainer is based on the abstract, not the full paper. The full text was not openly available at the time of writing, so the specific architecture, the network topology and tuning, the exact virtualization and driver mechanisms, the reliability and operational lessons, and the detailed measurements are not reflected here beyond what the abstract states, combined with standard background.
It is not zero-overhead. The reported roughly 80 percent of ideal throughput and roughly 70 percent per-GPU FLOPS versus a single VM indicate a real, if acceptable, cost relative to a perfect or bare-metal baseline; virtualized, cloud-delivered training trades some efficiency for its operational benefits.
The results are specific to one deployment. The numbers come from an NVIDIA A100 deployment in an IBM Cloud data center at around 1,500 GPUs on a 50-billion-parameter model; other GPUs, network fabrics, model sizes, and parallelism strategies would shift the outcomes.
It depends on specific hardware features. The approach relies on SR-IOV-capable network cards, GPUDirect-style peer-to-peer DMA between GPU and NIC, and RoCE support; hardware lacking these would not achieve the same result, even though such features are increasingly common.
It is an experience report, not a single new algorithm. Much of the contribution is the integration, operation, and lessons of building this system over about two and a half years; its value is in demonstrating and documenting a working design rather than proposing one narrow mechanism, and the paper also notes improvement opportunities for future systems.
Scaling efficiency depends on the workload’s communication pattern. The 80 percent figure is for a particular model and parallelization; workloads that communicate more, or scale to far more GPUs, could see different efficiency as communication overheads grow.

The evaluation

Ways to host large-scale LLM training

Property	Bare-metal cluster	Naive virtualization	Vela (virtualized, GPU-direct)
Cloud benefits (isolation, elasticity)
Network path	Direct to hardware
GPU-to-network data path	GPU-direct possible
Inter-machine transport	RDMA fabric	Overhead-laden
Demonstrated scaling efficiency	High (baseline goal)	Assumed too low	About 80 percent of ideal at ~1500 GPUs

Ways to host large-scale LLM training, as framed by the Vela abstract (the full paper was not openly available, so this reflects the abstract's claims and standard background rather than the paper's detailed evaluation). Bare-metal clusters avoid virtualization overhead but forfeit the cloud's isolation, multi-tenancy, elasticity, and manageability. Naively virtualizing training keeps those benefits but routes network packets through the hypervisor and forces GPU data to detour through host memory and the CPU, overheads long assumed fatal for communication-heavy training. Vela, per its abstract, is a cloud-native system built from off-the-shelf hardware, Linux KVM virtualization, and a virtualized RDMA-over-Converged-Ethernet network, whose VMs support peer-to-peer DMA directly between the GPUs and an SR-IOV network interface — keeping the hypervisor off the packet fast path and sending data straight from GPU memory to the network card. Deployed on NVIDIA A100 GPUs in an IBM Cloud data center and evaluated at roughly 1,500 GPUs, Vela trained a 50-billion-parameter decoder model with model parallelism at about 80 percent of ideal throughput, and reached about 70 percent per-GPU FLOPS compared to a single VM on the High-Performance Linpack benchmark, demonstrating that off-the-shelf, standard virtualization can host large-scale AI training near bare-metal efficiency.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract, because the full text was not openly accessible when it was written, so read it as a faithful account of the abstract’s claims plus standard, well-established background on distributed training, virtualization, SR-IOV, RDMA, RoCE, and GPUDirect — not as a report of the paper’s detailed architecture or measurements. With that caveat, the contribution is clear and valuable. Large-model training is intensely communication-bound: many GPUs must constantly exchange data, and the whole job stalls if that communication is slow, which is why serious AI clusters were run on bare metal, out of a belief that virtualization’s overheads — the hypervisor in the network path, and GPU data detouring through host memory and the CPU — would be fatal. Vela’s report, from IBM, is essentially an existence proof that this belief is too pessimistic: using off-the-shelf hardware, standard Linux KVM virtual machines, SR-IOV to give VMs near-direct network access, peer-to-peer DMA so GPUs talk straight to the network card, and virtualized RDMA over Converged Ethernet for the inter-machine fabric, they ran real large-model training in the cloud at about 80 percent of ideal throughput across roughly 1,500 NVIDIA A100 GPUs, with roughly 70 percent per-GPU Linpack FLOPS versus a single VM. The framing as a two-and-a-half-year experience paper is a strength: this is a system that was actually operated, not just benchmarked. What is not available here is exactly how the network is designed and tuned, how the virtualization and drivers are configured, what reliability and operational challenges arose, and where the remaining efficiency is lost — all the specifics that would let one judge and reproduce it. The durable idea, clear even from the abstract, is that the long-standing tradeoff between the operability of the cloud and the performance of bare metal is not fundamental for AI training: with the right hardware passthrough and direct data paths, commodity virtualization can host large-scale training at close to native efficiency, which makes serious AI infrastructure far more broadly accessible.

Background

Why the cloud keeps absorbing workloads once thought too demanding for it

It helps to see Vela within a recurring pattern in systems: workloads once considered too performance-sensitive for virtualization are, one by one, brought into the cloud as hardware and software learn to remove the overheads that made virtualization costly. Databases, high-performance networking, and now large-scale AI training have each followed this arc, and the enabling trick is consistently the same — let the guest reach the performance-critical hardware directly rather than through the hypervisor, while keeping the isolation that makes it a cloud. The virtualized-cloud memory work covered elsewhere on this site reflects the same frontier from the memory-management side, driving down the cost that virtualization imposes on a fundamental operation so that virtualized systems approach native performance. The common thread is that the value of the cloud — elasticity, isolation, multi-tenancy, manageability — is enormous, so there is persistent pressure to make even the most demanding workloads run well inside it, and the way forward is repeatedly to co-opt hardware features (SR-IOV, direct memory access, RDMA) that let virtual machines bypass the software layers that used to stand between them and the metal. Vela is a data point that large-scale AI training has now crossed that threshold, and as models and clusters grow, the pressure to deliver them through flexible cloud infrastructure rather than rigid bare-metal silos is only likely to intensify.

If you remember only three things

1. Training a large language model is a tightly-coupled distributed computation: the model and data are split across many GPUs that must continually exchange gradients and activations, within and across machines, so the whole job's speed is gated by fast, low-latency GPU-to-GPU communication. Cloud providers want to deliver this in virtual machines for isolation, security, multi-tenancy, elasticity, and manageability, but high-performance AI and scientific clusters have traditionally run on bare metal, because virtualization was believed to add too much overhead for communication-heavy training — the hypervisor mediating every network packet, and GPU data detouring through host memory and the CPU to reach the network card. The open question Vela addresses is whether a virtualized cloud built from off-the-shelf hardware and standard software can host large-scale model training near bare-metal efficiency.
2. Vela, per its abstract, is a cloud-native LLM training system built from off-the-shelf hardware, Linux KVM virtualization, and a virtualized RDMA-over-Converged-Ethernet (RoCE) network. Its two performance-critical ideas are hardware passthrough and a direct GPU-to-network data path. It uses SR-IOV, a hardware feature that lets one physical network card present virtual functions a VM can drive almost directly, keeping the hypervisor off the packet fast path so the VM gets near-native network performance while staying an isolated, managed cloud VM. And its VMs support peer-to-peer DMA directly between the GPUs and the SR-IOV network interface, so data flows straight from GPU memory onto the network — GPU-direct networking inside a VM — instead of being copied through host memory and shepherded by the CPU. The inter-machine traffic runs over virtualized RoCE, giving remote direct memory access over standard Ethernet with minimal CPU involvement.
3. Deployed on NVIDIA A100 GPUs in an IBM Cloud data center and operated over roughly two and a half years, Vela was evaluated at around 1,500 GPUs, training a 50-billion-parameter decoder model with model parallelism at about 80 percent of ideal throughput, and reaching about 70 percent per-GPU FLOPS compared to a single VM on the High-Performance Linpack benchmark — evidence that commodity virtualization can host large-scale AI training near bare-metal efficiency. Important caveat: this explainer is based on the abstract, because the full paper was not openly available, so the detailed architecture, network design and tuning, virtualization mechanisms, operational lessons, and full measurements are not reflected here. The durable idea, clear even from the abstract, is that the long-assumed tradeoff between the operability of the cloud and the performance of bare metal is not fundamental for AI training: with hardware passthrough (SR-IOV) and direct data paths (peer-to-peer DMA, RoCE), standard virtual machines can run large-scale training at close to native efficiency, making serious AI infrastructure more broadly accessible.

Check yourself

Check your understanding

1. Why was large-scale AI training traditionally run on bare metal rather than in virtual machines?
2. What are the two performance-critical mechanisms Vela uses to virtualize training without crippling it?
3. What is RDMA over Converged Ethernet (RoCE), and why does it matter for Vela?
4. What did Vela demonstrate, and what should you keep in mind about the evidence?

Sources

The paper is “Vela: A Virtualized LLM Training System with GPU Direct RoCE” by Apoorve Mohan, Robert Walkup, Bengi Karacali, Ming-Hung Chen, Abdullah Kayi, Liran Schour, Shweta Salaria, Sophia Wen, I-Hsin Chung, Abdul Alim, Constantinos Evangelinos, Lixiang Luo, Marc Dombrowa, Laurent Schares, Ali Sydney, Pavlos Maniotis, Sandhya Koteshwara, Brent Tang, Joel Belog, Rei Odaira, Vasily Tarasov, Eran Gampel, Drew Thorstensen, Talia Gershon, and Seetharami Seelam, published at ASPLOS 2025, DOI 10.1145/3676641.3716280. The authors are at IBM.

This explainer is based on the paper’s abstract. The full text was not openly accessible at the time of writing — the ACM Digital Library copy was gated, there was no arXiv or author preprint available (the OpenReview and IBM Research pages did not expose a downloadable PDF), and the Semantic Scholar, OpenAlex, and Unpaywall records pointed only to the gated version — so this page reflects the abstract’s stated claims (that Vela is a cloud-native LLM training system built from off-the-shelf hardware, Linux KVM-based virtualization, and a virtualized RDMA-over-Converged-Ethernet network; that its VMs support peer-to-peer DMA between the GPUs and an SR-IOV-based network interface; that it was deployed on NVIDIA A100 GPUs in an IBM Cloud data center and developed and operated over about two and a half years; and that at roughly 1,500 GPU scale it achieved about 80 percent of ideal throughput training a 50-billion-parameter decoder model with model parallelism, and about 70 percent per-GPU FLOPS compared to a single VM on the High-Performance Linpack benchmark) together with standard, well-established background on distributed model training, GPU-to-GPU communication, hardware virtualization, SR-IOV, RDMA, RDMA over Converged Ethernet, and GPUDirect peer-to-peer DMA, provided to make the abstract’s contribution understandable. The specific architecture, network topology, virtualization and driver mechanisms, operational and reliability lessons, and detailed measurements are not reflected here beyond what the abstract states; the flow and step-through sections are conceptual readings of the abstract plus standard background rather than details drawn from the full paper. This page is marked as a draft for that reason and should be confirmed against the full paper when it becomes available.