Running a neural network on encrypted data means translating it into an instruction set with exactly three operations — SIMD add, SIMD multiply, cyclic rotate — over tens-of-thousands-slot vectors with a metered multiplication budget; Orion compiles unmodified PyTorch models into that world with Toeplitz-formulated convolutions costing one level each, bootstrap placement solved as shortest paths on level graphs (37 bootstraps where prior work placed 58; 351 where HeLayers placed 8,480), and automatic scale management — 2.38× the state of the art on ResNet-20, ImageNet-scale ResNet-50, and the first FHE object detection.
The argument in one glance
-
Problem
Private inference exists in theory; networks don't fit it in practice
Cloud services run neural networks on your medical scans, finances, and photos — in the clear, server-side. Fully Homomorphic Encryption computes directly on encrypted data: the server executes the whole network without ever decrypting, and only you can read the result. The catch beyond raw cost: FHE's CKKS scheme offers exactly three operations — element-wise add, element-wise multiply, cyclic rotation — over encrypted vectors of 2¹⁴–2¹⁷ slots, with a finite multiplication budget per ciphertext. Nothing about a convolution, a ReLU, or a residual block maps naturally onto that.
-
What people did
Hand-crafted networks and early compilers
Experts hand-packed specific networks (Lee et al.'s multiplexed ResNet-20 became the community benchmark). Compilers emerged: EVA automated scale management, Fhelipe (PLDI '24) compiled tensor programs to FHE primitives, IBM's HeLayers reached ImageNet with tile tensors. Each solved a slice of the problem.
-
Why it fell short
Two levels per strided convolution, thousands of bootstraps, and no PyTorch
The state-of-the-art packing spent TWO multiplication levels on every strided convolution (one to convolve, one to mask-and-collect the strided outputs) and supported limited parameter combinations. Bootstrap placement — deciding where to pay the scheme's most expensive operation — was either greedy (provably wasteful in residual networks), exhaustively searched (hours), or profligate: HeLayers places 8,480 bootstraps in a ResNet-50 where 351 suffice. Scaling factors were tracked by manually inspecting per-layer value ranges. And everything lived far from the PyTorch world where models are actually built — capping the field at CIFAR-scale networks.
-
This paper's idea
PyTorch in, FHE out — with the two hard problems solved structurally
Orion is a fully-automated framework: models written (or loaded from torchvision) as orion.nn modules mirroring torch.nn compile to CKKS programs on the Lattigo backend. Two core techniques: SINGLE-SHOT MULTIPLEXED packing formulates any convolution (arbitrary stride, padding, groups, dilation) as a Toeplitz matrix whose rows are permuted so the strided mask fuses into the preprocessed weights — one level instead of two, with baby-step-giant-step and double-hoisting cutting rotations up to 6.41×; and bootstrap placement becomes a SHORTEST-PATH problem on "level digraphs," scaling linearly with depth (1.94 s for ResNet-20, 8–1270× faster than prior algorithms) while placing fewer bootstraps (37 vs Fhelipe's 58). Results: 2.38× on ResNet-20, ResNet-50 on ImageNet in 8.98 single-threaded hours, and the first FHE object detection — a 139M-parameter YOLO-v1 at 448×448, added in 60 lines.
Why this problem is worth solving
fully homomorphic encryption — encryption that
permits computation directly on ciphertexts: the server applies
functions to encrypted data and produces an encrypted result
only the key holder can read. The data, and every intermediate
value, stays opaque to the machine computing on it. Orion uses
CKKS, the FHE scheme built for approximate arithmetic on real
numbers — the natural fit for neural networks
The model is alien in three compounding ways. CKKS ciphertexts
are giant SIMD vectors — tens of thousands of slots encrypted
together, with no way to index one element; every operation
hits all slots, so performance is determined by how densely
you pack meaningful data and how cleverly you use cyclic
rotations to align it. Each ciphertext carries a finite
CKKS’s metered budget: every
multiplication consumes one “level” (technically, a prime limb
of the ciphertext modulus is dropped during rescaling); at
level 0 no further multiplication is possible. Operations also
run FASTER at lower levels — smaller ciphertexts — so levels
are both a budget and a cost model
the operation that makes FHE “fully”
homomorphic: a homomorphic re-encryption that resets a
ciphertext’s level budget. It is by far the most expensive
primitive — its own latency grows superlinearly with the level
it restores to — and consumes 13–15 levels itself, so a
ciphertext bootstraps back to an effective level well below the
maximum
pip install away from PyTorch.
What people did before, and where it broke
- Hand-crafted milestones. Lee et al. packed ResNet-20 with multiplexed parallel convolutions — interleaving channels within a ciphertext to handle strides — establishing the standard benchmark. But their strided convolutions burn two levels (convolve, then mask-and-collect), parameters beyond basic stride/padding aren’t supported, and everything is bespoke.
- Compilers, each partial. EVA automated rescaling and scale management for small kernels (LeNet-scale). Fhelipe (PLDI ‘24), the closest prior work, compiles tensor programs down to primitive FHE operations — an abstraction level that, Orion argues, discards the structure needed for the best cryptographic optimizations (hoisting), and generates encoded plaintexts on the fly during inference. HeLayers (IBM) reaches ImageNet via tile tensors but pays staggeringly in bootstraps (8,480 on ResNet-50) and needs FHE-specific training tricks (removing skip connections, range-aware losses).
- Bootstrap placement as an afterthought. Greedy “bootstrap only when forced” placement backfires in residual networks (extra bootstraps at merge points); search-based placement matches good counts but takes minutes to hours; and count-minimization itself is the wrong objective, since bootstrap cost grows superlinearly with the level it restores, and every op is cheaper at lower levels.
- Scale management by inspection. Bootstrapping and polynomial activations need inputs in [−1, 1], so prior work manually tracked each network’s largest intermediate values and inserted scale-downs by hand — per network, per dataset.
How Orion works
From torchvision weights to an encrypted inference
-
orion.nn mirrors torch.nn new
Models are written exactly like PyTorch modules — on.Conv2d, on.BatchNorm2d, on.SiLU(degree=127) — trainable with existing PyTorch scripts, loading pretrained torchvision weights directly. Activations take their polynomial degree as an argument; outputs are validated against the cleartext PyTorch run (typically ~8 bits of matching precision).
-
net.fit(): automatic range estimation new
Bootstrapping and Chebyshev activation polynomials require inputs in [−1, 1]. Orion sweeps the training set, computes per-layer scaling factors, and inserts the scale-down multiplications into the graph automatically — replacing the manual value-tracking of prior work. ReLU becomes a minimax composite polynomial (depth 14); SiLU needs only depth 7, halving activation cost for a small accuracy trade.
-
Single-shot multiplexed convolutions new
Any convolution — arbitrary stride, padding, groups, dilation — is formulated as a Toeplitz matrix acting on the flattened image, evaluated by the diagonal method with baby-step-giant-step (rotations drop from O(f) to O(√f) in filter size) and double-hoisting (key-switch costs amortized across rotations). Because Toeplitz ROWS can be permuted freely, the strided mask-and-collect step fuses into the preprocessed weight matrix: one multiplicative level per convolution, always, and densely packed outputs tracked by a gap parameter.
-
Bootstrap placement as shortest paths new
With bootstraps restricted to layer boundaries, the network unrolls into a LEVEL DIGRAPH: vertices are (layer, level) states, edges are "run this layer at this level" (cost = measured latency) or "bootstrap here" (cost = level-dependent bootstrap latency). The minimum-latency level-management policy is literally the shortest path — found by topological relaxation in O(L²·d), linear in depth. Residual blocks are single-entry single-exit regions solved independently and black-boxed.
-
Errorless scale management new
Instead of letting scales drift and correcting approximately, Orion encodes each layer's plaintext weights at precisely the prime q_j that the subsequent rescale will divide by — maintaining the invariant that every ciphertext sits at exactly scale Δ between layers, eliminating a compounding error source in deep networks.
-
Compile-time precomputation + Lattigo backend
All rotation keys and encoded matrix diagonals are generated and stored at compile time (to disk if needed) rather than during inference — encoding involves iFFT and NTT work worth avoiding on the critical path. Execution targets the single-threaded Lattigo library; the blocked matrix-vector structure is embarrassingly parallel for future multi-threaded/GPU backends.
The idea
Convolutions are matrices; matrices have one great FHE algorithm; so make every convolution that matrix. Orion’s packing insight is a chain of three observations. First, any convolution — whatever its stride, padding, groups, or dilation — is exactly a Toeplitz matrix–vector product on the flattened image, so the best homomorphic matrix-vector machinery (diagonal encoding + baby-step-giant-step + double-hoisting) applies to all convolutions uniformly, cutting rotations from O(f) to O(√f) in filter size. Second, Toeplitz rows can be permuted freely without changing the computation — which liberates the output layout from raster-scan order. Third, that freedom lets the strided convolution’s mask-and-collect step — a whole multiplicative level in the state of the art — be fused into the preprocessed cleartext weights, where it costs nothing. Every convolution now consumes exactly one level, halving depth where it hurts most, and on shallow networks the halved depth even permits a smaller ring degree (LoLA’s MNIST inference drops from 19 seconds to 0.23 — 83×).
Bootstrap placement is a pathfinding problem once you see the state space. The field treated bootstrap placement as a combinatorial search or a greedy heuristic; Orion’s reframing is that once bootstraps are restricted to layer boundaries, the entire level-management decision — where to bootstrap, and at what level to run every layer — is a shortest-path query on a DAG whose vertices are (layer, level) pairs and whose edge weights are measured latencies. This gets the objective right for free: because bootstrap cost grows superlinearly with the restored level and every homomorphic op is cheaper at lower levels, minimizing bootstrap count is not minimizing latency — the shortest path balances both automatically (the MobileNet-vs-ResNet-18 result shows the algorithm choosing more expensive convolutions to buy fewer bootstraps in one network and the reverse in the other). Complexity is linear in depth; residual blocks black-box as single-entry-single-exit subproblems; placement takes seconds where prior algorithms took minutes to hours, with equal or better counts.
Meet ML people where they live. The third contribution is deliberate un-cleverness: orion.nn classes subclass PyTorch’s, so pretrained weights load directly, training scripts run unchanged, and FHE outputs are validated against the cleartext model. The YOLO case study is the argument made flesh — the largest FHE computation ever run, added in 60 lines of code that read like ordinary PyTorch. Where HeLayers needed FHE-aware retraining (skip connections removed, range-aware losses), Orion fine-tunes stock torchvision models only to swap ReLU→SiLU and max-pool→average-pool.
An encrypted ResNet-20 inference, end to end
From CIFAR-10 image to encrypted classification
Slot = one element position in a CKKS ciphertext vector (2¹⁵–2¹⁶ here). Level = remaining multiplication budget. Δ = the scaling factor all ciphertexts hold between layers. BSGS = baby-step-giant-step rotation decomposition. Numbers from the paper's ResNet-20 evaluation vs Fhelipe (PLDI '24) on identical hardware.
Step 1 of 6
Define the model as if FHE didn't exist
The ResNet block is written with orion.nn layers mirroring torch.nn line for line; SiLU(degree=127) replaces ReLU (half the multiplicative depth); weights come from a normal PyTorch training run. net.fit() sweeps the training data, records per-layer value ranges, and inserts scale-down multiplications so every bootstrap and polynomial sees inputs in [−1, 1].
code delta from PyTorch: import orion.nn as on range estimation: automated; prior work inspected by hand
Why it works
- Any convolution is a Toeplitz matrix-vector product → the best FHE linear-algebra stack (diagonals + BSGS + double-hoisting) covers every conv uniformly → O(√f) rotations, arbitrary parameters, and the benefits GROW with network size (AlexNet: 86× the parameters of ResNet-20, only 1.76× the rotations).
- Toeplitz rows permute freely → the strided mask-and-collect folds into cleartext weight preprocessing → strided convolutions cost one level, not two; total depth drops, and shallow networks fit smaller (faster, still-secure) ring degrees.
- Restricting bootstraps to layer boundaries makes the level-management state space an enumerable DAG → shortest path IS the optimal policy under the true cost model (where bootstrap latency and op latency both depend on level) → fewer bootstraps AND lower latency than count-minimizing or greedy schemes, found in seconds, linear in depth.
- Encoding weights at the exact rescaling prime keeps every ciphertext at precisely Δ → approximation error stops compounding with depth → 50-layer encrypted networks still match PyTorch to ~8 bits.
- Inheriting from PyTorch rather than replacing it → pretrained weights, training scripts, and developer intuition carry over → the demonstrated marginal cost of a NEW model (YOLO) is 60 lines, which is the difference between a benchmark suite and a usable field.
What it costs
- FHE is still FHE. The headline systems results sit atop hours-scale latencies: 8.98 h for one ResNet-50 inference, 17.5 h for one YOLO detection — single-threaded, to be fair, and embarrassingly parallel, but “far from practical” is the authors’ own phrase. Orion removes the programmability wall, not the compute wall (that’s the accelerator literature’s job).
- Approximation is intrinsic. Activations become polynomials (SiLU trades accuracy for half of ReLU’s depth), max-pool becomes average-pool, CKKS itself is approximate arithmetic — FHE accuracy tracks cleartext closely but is certified here empirically, per network, not analytically.
- Levels of trust. The threat model protects the INPUT; weights are cleartext (server-owned), and ImageNet-scale validation is a single encrypted inference per network due to cost — statistical accuracy claims ride on cleartext proxies.
- Single backend, single thread. Evaluation targets Lattigo single-threaded; OpenFHE, GPU HEaaN, and multi-threading are future work, and comparisons against 32-thread HeLayers runs are necessarily indirect (bootstrap-count comparisons stand in).
- Boundary-restricted bootstraps. The shortest-path formulation depends on placing bootstraps only between layers — a restriction the paper argues is natural (linear layers consume one level anyway) but a restriction nonetheless.
The evaluation
Four datasets (MNIST, CIFAR-10, Tiny ImageNet, ImageNet) and eleven networks from MLPs to ResNet-50, with baselines (Lee et al., Fhelipe) re-run on the same GCP instance, single-threaded Lattigo throughout, and FHE outputs validated against cleartext PyTorch (~8 bits of precision; full test sets for MNIST, sampled sets for larger benchmarks). Beyond the speedups, the evaluation is a scaling argument: the same automation spans LoLA (0.23 s) to YOLO-v1 (139M parameters), a range no prior system covers.
Getting a neural network into FHE
| Property | Hand-crafted (Lee et al.) | Prior compilers (EVA / Fhelipe / HeLayers) | Orion |
|---|---|---|---|
| Programming model | |||
| Convolution packing | |||
| Bootstrap placement | Manual | ||
| Scale & range management | Manual inspection of per-layer value ranges | EVA automates rescaling at small scale; ranges still per-network craft elsewhere | |
| Demonstrated ceiling | ResNet-20 / CIFAR-10 (2.53 h ResNet-50 on 32 threads in follow-up work, at 8,480 bootstraps) | Fhelipe: CIFAR-scale (ResNet-20 largest); HeLayers: ImageNet with retraining |
Check yourself
Check your understanding
1. CKKS gives you SIMD addition, SIMD multiplication, and cyclic rotation over ~65,536-slot encrypted vectors — nothing else. Predict the THREE distinct walls this raises for running a stock neural network.
2. Prior bootstrap-placement work minimized the NUMBER of bootstraps. Orion's shortest-path formulation often places fewer anyway — but why is count the wrong objective in the first place?
3. The state-of-the-art strided convolution cost two multiplicative levels: one to convolve, one to mask-and-collect the strided outputs. How does Orion's single-shot multiplexing cut this to one?
4. Fhelipe compiles networks down to primitive FHE operations; Orion deliberately stops at a higher abstraction level (linear transforms). The measured consequence: Orion's convolutions run 11.2× faster despite only 1.71× fewer rotations. What explains the gap?
Sources
- arXiv preprint 2311.03470 (v3) — full text used for this explainer
- ACM DL: Orion: A Fully Homomorphic Encryption Framework for Deep Learning — ASPLOS 2025
- Orion on GitHub