PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption

Running a language model inside a hardware-protected GPU enclave means every byte crossing between processor and accelerator gets encrypted, and since large models constantly swap weights and cached attention state in and out of GPU memory, that encryption costs up to 88 percent of throughput; PipeLLM (Shanghai Jiao Tong and Northeastern) hides it by guessing what the GPU will ask for next and encrypting it early — with the twist that a wrong guess poisons an entire chain of anti-replay counters, and the twist on the twist that the paper's own ablation shows the guessing is worth only 8 percent.

Why this problem is worth solving

The security model here is newly available and immediately expensive.

The mechanism that makes it secure is also what makes it hard to speed up.

What people did before, and where it broke

• Do the encryption on the fly, inside the copy. What the vendor’s implementation does — the library detects it is in confidential mode and encrypts as part of the memory copy call. Where it broke. The GPU sits idle for the whole encryption, and the measurements show it plainly: transfer latency stays flat with size when encryption is off and grows in direct proportion when it is on, because the cryptography is welded to the call.
• Trust the vendor’s overhead numbers. Reported as negligible for a language model. Where it broke. That was a small model that fits comfortably in memory. Large models do not, so they swap constantly, and swapping is exactly what encryption taxes. The measured drop reaches 88 percent for serving and 36 percent for fine-tuning.
• Keep the accelerator out of the trusted base entirely and use the processor for the sensitive layers. A software approach for edge deployments that avoids trusting the GPU hardware at all. Where it broke. It imposes between 2 and 15 times overhead on inference, because layers that cannot be transformed have to run on the processor.
• Insert a trusted software layer that validates GPU commands. Another software-only route to isolation. Where it broke. The runtime checks needed for indirect memory access are expensive, and indirect memory access is heavily used by exactly the serving systems this targets.
• Keep the encrypted copy on the processor and resend it unchanged. The obvious optimisation, since the swapped-out data is never modified while it sits in host memory. Where it broke. Reusing a ciphertext lets an attacker recognise that a transfer matches an earlier one, and more seriously opens the system to replay attacks. The counter scheme exists specifically to prevent this.
• The gap. The encryption is load-bearing and cannot be removed, only moved. Nobody had tried to get it off the critical path, and the obstacle is a genuine one — the encryptor does not know what to encrypt until the GPU asks, at which point it is already too late.

How PipeLLM hides the encryption

The idea

Mandatory work cannot be deleted, so move it off the critical path and start it early. This is the oldest trick in processor design applied somewhere new. Encryption is not overhead in the sense of waste — it is the entire security guarantee, so the only available move is to overlap it with something else. The obstacle is that a pipeline needs to know what work is coming, and the encryptor learns what to encrypt only when the GPU demands it, at which point everything is already blocked. The resolution is that language model serving is unusually predictable in exactly the dimension that matters. Systems that offload weights walk the layers in a repeating order, and since every layer has identical structure and size there is no reason for a system to offload one in some iterations and keep it in others. Systems that swap cached attention state use policies simple enough to name — evict layers in order and restore them in order, or evict the lowest-priority request and restore it last. Speculating in order to get obligatory work started before it is formally requested is the same manoeuvre as the speculative logging work covered elsewhere on this site, applied to encryption instead of durability.

The anti-replay counter is what makes this hard, and the repairs work on the counter rather than the data. Each encryption consumes a counter value that both ends track without ever transmitting, which is what stops an attacker replaying a captured transfer. It also chains the encryptions together, so a single wrong guess does not waste one block — it invalidates the counter values every subsequent pre-encrypted block was built on. The paper’s response is to find where the chain has slack. Serving systems submit swaps in batches and only enforce ordering between batches, so a block predicted in the wrong position within a batch can simply be reordered at no cost. More cleverly, a one-byte dummy transfer advances the counter almost for free, which creates a deliberate asymmetry — guessing the counter too high is repairable by padding, while guessing too low is unrecoverable. So the predictor biases every guess upward on purpose. Working around a protection mechanism whose granularity and assumptions were set without this workload in mind is the same situation as the trusted-execution granularity work covered elsewhere on this site.

Speculating changes what an observer can see, and the paper leaves that open. The security analysis is careful about what is preserved — no ciphertext reaches shared memory until its prediction has been validated, the padding transfers carry only dummy bytes, and the permission checks run inside the protected virtual machine where an attacker cannot reach them. Confidentiality and integrity are intact. But the paper then states plainly that mis-speculation introduces side channels the original design did not have, because the padding transfers are visible. An observer can tell when the system is swapping at all, and can profile how often predictions fail, which leaks something about the application’s swapping behaviour. The authors write that the implications remain unclear and need further research, which is the honest position and also an unresolved one. That speculation creates observable effects distinct from the results it produces is the recurring lesson of the speculation contract work covered elsewhere on this site.

Repairing a mispredicted swap

This is the failure mode that makes the whole scheme delicate. The counter is never transmitted — both ends simply count — so a mismatch is not detected and corrected, it just produces wrong plaintext. Getting it right is a precondition, not a check.

Why it works

• The two halves of the system are idle at different times. The GPU waits while the processor encrypts, and the processor is free while the GPU computes → so the encryption can be moved into the window where the processor has nothing else to do → and the transfer becomes something the GPU never observes waiting for.
• Serving systems swap in patterns simple enough to name. Weight offloading repeats a fixed layer order, and attention cache swapping is either first-in first-out or last-in first-out → so a short hand-written function reproduces the sequence from the recent history → and the swaps can be told apart from ordinary traffic purely by size and by what the model definition implies.
• A dummy transfer is a cheap way to skip a counter value. Advancing the counter costs one byte of traffic, while decrementing it is impossible → so predicting the counter too high is repairable and predicting it too low is not → which makes it correct to bias every guess upward rather than to aim for accuracy.
• Batches give the reordering slack for free. Serving systems require ordering between groups of copies but impose none within a group → so a block predicted in the wrong position inside a batch just moves → and only errors that cross a batch boundary force the pipeline to be abandoned.
• Page permissions answer the validity question without touching the data. Write access is revoked on each pre-encrypted region, so any modification announces itself as a fault → replacing a comparison against a saved copy that would have run on the critical path and tripled memory use → and since the swapped data is never modified in practice, the faults essentially never fire.

What it costs (trade-offs and scope)

• The prediction contributes far less than the name suggests. The paper’s own ablation runs a version whose sequence predictions never succeed, and it loses only 8.3 percent — the pipelining, the reordering and the counter padding recover almost everything on their own. This is the most important number in the paper and it is easy to miss.
• The predictors are hand-written per swapping policy. Three patterns cover the systems evaluated. A serving system with a different policy needs a person to recognise the pattern and write a new prediction function. Learning the predictor automatically is named as future work.
• A floor remains that pipelining cannot reach. Even with processor-side encryption entirely hidden, the transfer rate under confidential computing is capped around 40 gigabytes per second against 64 without it, plus control-plane overhead. That is why the result is under 19.6 percent rather than near zero.
• New side channels are introduced and left unresolved. The dummy transfers reveal whether the system is swapping and how often predictions fail, potentially exposing swapping patterns. The paper states the implications remain unclear and need further research.
• User transparency rests on hooking library internals. Applications need no changes, but the implementation intercepts the vendor’s memory copy interface and the cryptographic routines it calls underneath. Transparent to the application, tightly coupled to specific library implementations.
• The whole problem may be a transitional one, as the paper acknowledges. Next-generation confidential VMs add dedicated hardware for line-rate encryption. The paper’s counter-argument is a question rather than a measurement — whether that hardware can sustain the eight accelerators a typical server holds.
• The simpler alternative is rejected for reasons the paper calls a trade-off, not a law. Caching and reusing ciphertext for read-only swap data would eliminate the problem, and is unsafe only because the current scheme prioritises simplicity and transparency. Future hardware could support it directly.
• The validity check assumes swapped data is never modified on the processor side. True of the systems evaluated. A system that did modify it would take page faults on the critical path.
• Scope is one machine and one model family. A single accelerator passed through to a single virtual machine, no multi-GPU or distributed setting, and every workload is an OPT model, with the largest evaluated only in quantised form.

The evaluation

Check yourself

Sources

The paper is “PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption” by Yifan Tan, Cheng Tan, Zeyu Mi and Haibo Chen, at Shanghai Jiao Tong University and Northeastern University, published at ASPLOS 2025 Volume 1, pages 843-857, DOI 10.1145/3669940.3707224. The implementation is open-source.

This explainer is written from the full text, read from the authors’ copy on arXiv (2411.03357). Everything above comes from the paper’s own sections 2 through 9: the description of confidential virtual machines paired with an accelerator enclave and the counter-based encryption scheme protecting the link between them; the section 3 measurement study, including the microbenchmark showing transfer latency flat with size when encryption is disabled and proportional when enabled, and the three case studies covering weight offloading during serving, cached attention state swapping under increasing request rates, and fine-tuning, with the identification that the bottleneck is the processor encrypting on the swap-in path while the accelerator idles; the section 4 statement of the prediction problem, the observations that swaps are distinguishable by size, that swapped data is read-only, and that a one-byte transfer advances the counter cheaply; the section 5 design covering the three swapping patterns observed in current systems, the predictor and its inputs, the page-permission validator with the rejected comparison-based alternative, the batch reordering and padding recovery mechanisms with their worked example, and asynchronous decryption; the section 6 implementation details including the decision to hold unvalidated ciphertext in private rather than shared memory; the section 7 evaluation across three serving and fine-tuning systems, including the multi-threaded control experiment and the ablation with zero prediction success; and the section 8 discussion, which analyses the new side channels introduced by padding transfers and states their implications remain unclear, explains why reusing ciphertext for read-only swap data would break replay protection while calling the current design a trade-off rather than a fundamental limitation, and compares the approach against forthcoming hardware encryption support.