Cinnamon: A Framework for Scale-Out Encrypted AI

Computing on encrypted data is about ten thousand times slower than computing on plain data, and the accelerators built to fix that have grown into single chips carrying half a gigabyte of on-chip memory — while still only running models small enough to encrypt in one ciphertext; Cinnamon (Carnegie Mellon) argues the field should spread the work across several small chips instead, which required removing the one subroutine whose data dependencies made that impossible, and then runs a language model in 1.67 seconds rather than 17 hours.

Why this problem is worth solving

The capability on offer is unusual enough to be worth the engineering.

The internal representation is what makes it hard to parallelise.

What people did before, and where it broke

• Run it on processors with optimised libraries. The baseline, and the libraries are good. Where it broke. Over four orders of magnitude slower than plaintext. A single refresh operation takes 33 seconds; one inference of a small language model takes over 17 hours.
• Build a large accelerator and parallelise across polynomial coefficients. What most published designs do, since the transforms and permutations are where the arithmetic is. Where it broke. Those operations have all-to-all dependencies, so the on-chip network needs 10 to 20 terabytes per second. That is hard on one chip and much harder across several, which is precisely why the designs became monolithic.
• Make the chip bigger. The consistent response to needing more compute and more cache. Where it broke. The measurements here show 256 megabytes of cache is a balanced point for one refresh and a further gigabyte adds only 28 percent. Meanwhile the chips reach 400 to 700 square millimetres, where yield falls to around a third, and models have grown far faster than caches have.
• Optimise for refreshing one ciphertext at a time. Reasonable when the target workload fits in one ciphertext. Where it broke. A language model on 128 tokens needs three ciphertexts and about 1,400 refreshes, so a design that handles one at a time degrades linearly — eight refreshes take not four but nearly nine times as long as one at the standard cache size.
• Split across chiplets and broadcast the pieces to resolve dependencies. The prior state of the art for parallelising across limbs. Where it broke. Three broadcasts per invocation of the problematic subroutine. It works while per-chip memory bandwidth is the binding constraint, but raising memory bandwidth to the level these designs need would demand roughly 16 terabytes per second of network — unattainable. Measured here, it is 2.14 times slower than not parallelising at all.
• The gap. Everyone treated the cross-limb dependencies as a bandwidth problem to be paid for. They are an algorithmic problem: the subroutine communicates at both its start and its end, and only one of those can be batched across repetitions, so confining communication to a single end changes the asymptotic cost rather than the constant.

How Cinnamon spreads the work

The idea

The field has been scaling up when it should have been scaling out, and the evidence is in its own numbers. Two curves tell the story. Model parameters have grown by orders of magnitude over a decade, while on-chip cache in encrypted-computing accelerators has crept from 64 megabytes to about half a gigabyte. So the largest workload any published architecture accelerates is a small image classifier whose inputs fit in a single ciphertext and which needs roughly fifty refresh operations — against a modest transformer needing three ciphertexts and about fourteen hundred. Worse, all of those designs optimise for refreshing exactly one ciphertext at a time, which is the wrong target, because larger models are wider and produce genuinely independent refreshes. The paper’s own study makes the bind concrete: at the standard cache size, eight refreshes take nearly nine times as long as one, and fixing that by growing the cache to a gigabyte helps five times more for eight than for one, because the shared plaintext matrices and evaluation keys finally stop spilling. Either models slow you down linearly or the chip has to keep growing. Getting real deep learning models to run under encryption rather than demonstrating on small ones is the same ambition as the encrypted deep learning framework work covered elsewhere on this site.

The blocker was one subroutine, and it is an algorithmic problem rather than a bandwidth problem. Almost everything about a ciphertext decomposes cleanly — additions, transforms, multiplications and permutations all act on each limb independently, which is exactly what a scale-out design wants. The exception is keyswitching, which sits inside every multiplication, every rotation and every refresh, and whose basis conversions compute each output piece from all input pieces. Prior limb-parallel work resolved this by broadcasting, three times per invocation, and the paper shows why that dead-ends: raising per-chip memory bandwidth to the level these designs need would demand around 16 terabytes per second of network. The insight is that repeated communication can be hoisted and merged across invocations, but only at one end — so a formulation that communicates at both ends will always leave a linear term. Both of Cinnamon’s algorithms therefore confine communication to a single point. One broadcasts inputs at the start and duplicates a small temporary basis across chips so that the finish needs nothing; the other redefines the algorithm’s own partition to coincide with the existing chip partition, so nothing is needed at the start, and reorders two commuting steps so only the finish aggregates. A run of rotations then costs one broadcast rather than one per rotation, and a standard matrix multiplication pattern drops from communication growing with the square root of the size to constant. Restructuring the encrypted computation itself so that a compiler can batch what was previously per-operation work is the same manoeuvre as the encrypted circuit vectorisation work covered elsewhere on this site.

At four chips the win is cost, not speed, and the paper is straightforward about it. This deserves stating plainly because the headline numbers can obscure it. Against a monolithic chip built from the same components and scaled up to comparable resources, four Cinnamon chips are slightly slower on the refresh benchmark and on logistic regression, faster on the image classifier, and identical on the language model. The stated goal is to match a large monolithic chip using four small ones — and what that buys is a die of 223 square millimetres instead of 720, which is 66 percent yield instead of 31, which is a tape-out cost of 3.5 million dollars instead of 25. The performance gains arrive only when there is enough parallelism to use more chips, which the language model has and the small benchmarks do not. The paper is equally clear about where scaling stops: the standard refresh configuration gains 4.18 times at four chips and only 4.98 at twelve, because communication becomes the limit again. A deeper refresh configuration with twice the compute scales considerably better, which is a genuine finding and also a different configuration. Trading against the cost structure of bootstrapping rather than treating it as fixed is the same axis as the bootstrapping management work covered elsewhere on this site.

Making keyswitching parallel

The prior approach broadcasts at both ends, three broadcasts per invocation. That is workable when memory bandwidth is the binding constraint, but the paper shows it becomes 2.14 times slower than running on a single chip once the links are the limit, because the communication cost swamps the parallelism gain.

Why it works

• Almost everything about a ciphertext already decomposes. Additions, transforms, multiplications and permutations act on each limb independently → so distributing limbs across chips costs nothing for the vast majority of operations → which means the whole scale-out question reduces to handling the one subroutine that does not decompose.
• Batching can absorb one communication point, not two. Repeated invocations let a compiler hoist and merge communication → but merging is only possible at one end, so an algorithm communicating at both keeps a cost proportional to the number of invocations → and confining it to one end turns a run of rotations into a single broadcast.
• Duplicating a small basis is cheaper than communicating a large one. The contraction at the end of the subroutine reads only the temporary basis → so computing that basis redundantly on every chip removes the need to exchange it later → trading a little duplicated compute and storage for eliminating an entire round of communication.
• The algorithm partition and the hardware partition can be made the same thing. The subroutine is free to choose how it splits its input, and the choice does not affect the mathematics → so choosing it to match how limbs already sit on chips means no redistribution is needed at the start → and reordering the final contraction with the summation, which commute, leaves only an aggregation.
• These workloads never need the general form of basis conversion. They always convert from few input pieces to many output pieces → so sizing the unit by input count rather than output count shrinks its multipliers roughly tenfold and its buffers fivefold → and lets the buffers be single-ported, since each is either only written or only read.

What it costs (trade-offs and scope)

• At four chips it matches the monolithic design rather than beating it. Against a single large chip built from the same components, four Cinnamon chips are slower on the refresh benchmark and on logistic regression, faster on the image classifier, and equal on the language model. The stated goal was to match, and the win at that point is manufacturing economics.
• The cost analysis is explicitly optimistic and one competitor’s figure is understated. The authors say yield data is an industry secret and adopt what they call a very optimistic defect assumption. They also state that the chiplet competitor’s cost covers a single chiplet and is therefore underestimated, since no public data existed for the interposer — a disclosure that works against their own comparison.
• The comparison spans three process nodes. Cinnamon is synthesised at 22 nanometres; the competing designs are at 14 and 7. Area, yield and wafer cost are compared across all of them using public pricing.
• The standard refresh configuration stops scaling after four chips. It gains 4.18 times at four, 4.78 at eight and 4.98 at twelve, because communication becomes limiting again. A configuration refreshing more levels, with roughly twice the compute, reaches 8.81 at twelve — better, and a different configuration.
• Everything is simulated. A cycle-accurate simulator driven by timing from synthesised register-transfer code, with a separate emulator used to check correctness. The register-transfer implementation is real; no chip is.
• The headline speedup has no accelerator to compare against. The 36,600 times figure for the language model is against a 48-core processor. No prior architecture reports that workload at all, so there is no accelerator baseline for the paper’s flagship result.
• Utilisation falls as chips are added. Twelve chips show lower compute and memory utilisation, because parallelising the wide sections leaves the narrow ones as a larger share of the runtime.
• One competing optimisation is ruled out for this workload rather than compared against. A design reducing precision to 36 bits is noted as tested only on small models; the language model here needs 50 to 80 bits. The authors describe that work as orthogonal and combinable, which is fair, but it is not evaluated together.
• Both new formulations trade something. The first duplicates compute and on-chip storage across chips. The shrunken conversion unit halves its lanes, trading throughput for half the area and power.

The evaluation

Check yourself

Sources

The paper is “Cinnamon: A Framework for Scale-Out Encrypted AI” by Siddharth Jayashankar, Edward Chen, Tom Tang, Wenting Zheng and Dimitrios Skarlatos at Carnegie Mellon University, published at ASPLOS 2025 Volume 1, pages 133-150, DOI 10.1145/3669940.3707260. It is open access under a Creative Commons licence and the framework is described as the first end-to-end open source framework for encrypted computing.

This explainer is written from the full text of the published paper. Everything above comes from its own sections 2 through 8: the background on the encryption scheme, the residue number decomposition into limbs, the transforms, basis conversion and its non-decomposability, digits, the multiplicative budget and bootstrapping, and the keyswitching routine with its expansion, key multiplication and contraction; the section 3 study showing that larger models produce both wider and deeper programs, the cache and compute sweep establishing that eight refreshes take nearly nine times one at the standard cache size while a gigabyte helps five times more for eight than for one, the bandwidth requirements of coefficient-level parallelism, and the analysis of why the prior chiplet approach does not scale; the section 4 design with both keyswitching formulations described step by step, the argument that batching can absorb only one communication point, the compiler pass that detects patterns and selects an algorithm, the reported sevenfold and 9.81-fold communication reductions, the ring and switch topologies, and the base conversion unit sized by input rather than output limbs with its resource reductions; the section 5 implementation with its process node, die area, power and component-wise area breakdown; and the section 7 evaluation including the execution time table across all configurations and baselines, the yield and tape-out cost analysis with its stated optimistic assumptions and the acknowledgement that the chiplet cost is underestimated, the keyswitching speedup breakdown across link bandwidths, the scaling study for two bootstrap configurations, the utilisation and sensitivity results, and the discussion of a competing precision-reduction technique as orthogonal but untested at the precision this workload requires.