Systems research on CPU+FPGA machines was trapped inside narrow, closed commercial platforms where even "is cache coherence a good idea?" couldn't be tested — so ETH Zürich built its own computer; Enzian pairs a 48-core ARM server CPU with the largest available FPGA as two coherent NUMA sockets speaking the CPU's native protocol (re-implemented from scratch in FPGA logic), overengineered and instrumented everywhere, open-source down to the board-management firmware — and it matches the specialized platforms it replaces while enabling experiments none of them can run.
The argument in one glance
-
Problem
Systems research lost its common machine
For decades, OS, database, storage, and networking research shared one platform — the commodity PC server — which made results comparable, ideas transferable, and deployment plausible. That era is over: clouds now run smart NICs, smart storage, programmable switches, TPUs, and above all hybrid CPU+FPGA machines. These hybrids are widely deployed and full of open design questions — yet there is no consensus even on basics like whether CPU–FPGA cache coherence is worth having.
-
What people did
Do research on whatever industry ships
Researchers adopted the available platforms: PCIe FPGA cards (Amazon F1, Xilinx Alveo, Alpha Data), coherent-attach research programs (Intel's QPI/UPI Xeon+FPGA, IBM's CAPI), and bump-in-the-wire designs (Microsoft Catapult). A wave of "operating systems for FPGAs" research grew on top of them.
-
Why it fell short
Cost-optimized, closed, quirky, and mutually incomparable
Each platform is engineered for one narrow use case and priced for a market, not a lab. PCIe cards bake in the GPU-style batch-offload model — copy data over, compute, copy back — which forecloses fine-grained interaction studies. The coherent platforms had small FPGAs, NDA-bound protocols, or were discontinued. Nothing is instrumented (power and thermal data are inaccessible), the baseboard management firmware is a closed black box with total control of the machine, and every platform's quirks mean results from one cannot be compared with results from another or transferred forward.
-
This paper's idea
Build the research computer industry won't
Enzian is a from-scratch two-socket server where socket two is a huge FPGA: a 48-core ARM ThunderX-1 and a Xilinx XCVU9P joined by the CPU's NATIVE inter-socket cache-coherence protocol, re-implemented by the group in FPGA logic (ECI). Every design choice is "overengineer and instrument": maxed-out parts, 480 Gb/s of network, 25 monitorable voltage regulators, an open programmable BMC, everything open source and remotely usable. It reproduces the surveyed platforms' roles at comparable performance — and runs experiments (coherence-message access, custom memory controllers, zero-overhead runtime verification) that no commercial machine permits.
Why this problem is worth solving
This is an unusual ASPLOS paper: its contribution is not a
technique but a machine, and its thesis is about research
methodology. Hybrid platforms — a conventional CPU beside an
field-programmable gate array — a chip whose logic
is configured after manufacture, letting developers implement
custom hardware (accelerators, NICs, controllers) as reloadable
“bitstreams” instead of fixed silicon
The stakes compound because research builds on research. When every FPGA-OS project (Coyote, AmorphOS, and kin) targets a different quirky platform, results cannot be compared, artifacts cannot be reused, and the community cannot converge. The paper’s inspiration is explicit: Berkeley Unix, Emulab, PlanetLab, and NetFPGA — moments when shared, open research infrastructure unlocked a decade of work. Enzian asks whether a single academic group can still do that at the level of a full server-class computer, pandemic and 150-amp power rails included — and documents what it costs to try.
What people did before, and where it broke
The paper surveys the platform landscape researchers had to choose from:
- PCIe accelerator cards (Amazon F1, Alveo, Alpha Data, Intel D5005). The dominant model, patterned after GPUs: batch data onto card DRAM by DMA, compute, copy back. PCIe has fine bandwidth for bulk transfers but poor latency for small ones — so the whole research field inherits the batch-offload assumption baked into the interconnect. Cloud instances add another wall: the shell logic is proprietary and portions of the device are off-limits.
- Coherent-attach platforms (Intel Xeon+FPGA via QPI/UPI, IBM CAPI/OpenCAPI). Exactly the interesting design point — the FPGA sees the CPU’s memory coherently — but Intel’s program shipped small FPGAs under NDA to a select few and was discontinued; CAPI tied to POWER systems with a fixed proxy cache design. You could use coherence, but not study it: the protocols themselves are sealed.
- Bump-in-the-wire (Microsoft Catapult): the FPGA sits on the network path in front of the NIC — hugely influential inside Microsoft, unavailable outside it.
- SoC hybrids (Xilinx Zynq): CPU and FPGA on one die, but with embedded-class cores unsuited to server workloads.
Common to all: no power/thermal instrumentation, closed
baseboard management controller — the hidden
processor on every server board that sequences power and clocks,
runs its own OS, has network access, and answers to nobody: not the
host OS, not the hypervisor. Commercial BMCs are closed firmware;
research has largely ignored them despite their total control over
the machine
What Enzian is
The Enzian machine
-
CPU socket: 48-core ARM server SoC
A Cavium/Marvell ThunderX-1 — 48 ARMv8 cores at 2.0 GHz with up to 512 GiB of DDR4 (50–70 GiB/s) and 2×40 GbE. Chosen not for single-thread speed (it's mostly in-order) but because it was essentially the ONLY server-class part whose inter-socket coherence protocol the group could practically implement themselves on an FPGA.
-
FPGA socket: the biggest part available
A Xilinx Virtex UltraScale+ XCVU9P-3 — chosen to max out logic and high-speed transceivers — with its own DDR4 (tens of GiB, ~60 GiB/s), 4×100 GbE, NVMe, PCIe x16, SATA, and an FMC expansion slot. Total board network bandwidth: 480 Gb/s, sized so racks of Enzians can form clusters or prototype switches.
-
ECI: the coherence link, built from scratch new
The two sockets speak the CPU's NATIVE inter-socket protocol over 24 lanes (~30 GiB/s) — re-implemented on the FPGA as the Enzian Coherence Interface: a MOESI-style protocol with 128-byte cache lines, formally specified in layers, with generated checkers. The FPGA is a NUMA node, not a peripheral — and researchers get explicit access to raw coherence messages. A PCIe crossover cable remains available to reproduce conventional-platform experiments.
-
Instrumentation everywhere new
25 voltage regulators feeding 30 rails, each queryable for voltage, current, and temperature over PMBus in ~5 ms — enabling fine-grained power measurement and even undervolting studies of CPU, FPGA, and DRAM. Four serial consoles route through one USB port.
-
An open, programmable BMC new
A Zynq SoM running OpenBMC with firmware the group wrote themselves — by necessity, since nothing else could sequence 25 regulators (a mistake risks a short across a 150 A rail). Unlike every commercial server, the BMC is open for experimentation and exports all telemetry without touching the main CPU. Writing it spawned research lines of its own: solver-generated provably-correct power sequencing, a formally verified I2C stack, and an ongoing seL4 port.
-
Open source, remotely usable new
Board design, ECI, firmware, and software are open; the boot chain (BDK → ARM Trusted Firmware → UEFI) exposes knobs to throttle lanes, clocks, and bandwidth — letting one machine imitate many lesser ones. Enzians at ETH are available for remote research use; the ASPLOS artifact is on Zenodo.
The idea
Design hardware from the software researcher’s viewpoint — then overengineer it. Enzian’s design principles invert product economics. A product minimizes unit cost for a known market; a research machine exists in small numbers, so flexibility per board matters and cost per board doesn’t. Hence: max out every part (largest FPGA then available, nearly all transceivers wired), favor bandwidth over capacity (one DIMM per channel), instrument everything, and skip the industry standard where it constrains — the deepest example being the interconnect. A maxed-out machine can always be throttled down to emulate a lesser one (the boot firmware exposes lane counts and clocks for exactly this), but a cost-reduced machine can never be scaled up. Coverage beats optimality: Enzian aims to subsume the “convex hull” of existing platforms, not to beat any of them at its own game.
Make the FPGA a NUMA peer, not a peripheral. The load-bearing
decision is connecting CPU and FPGA with the CPU’s native
the protocol by which multiple
caches/sockets keep a consistent view of shared memory — here
MOESI-style messages moving 128-byte cache lines between the
ThunderX-1 and the FPGA, exactly as between two CPU sockets in a
NUMA server
The machine is also the test harness. Enzian can be split into “system under test” and “measurement apparatus”: the BMC samples every power rail while workloads run; the FPGA can consume the CPU’s hardware program-trace stream and check temporal-logic assertions about OS and application behavior in real time with zero overhead. Commercial machines hide exactly the layers — regulators, firmware, coherence messages — where this observability lives; Enzian opens all of them.
One machine, many machines: the demonstrated roles
Six platforms Enzian can be — each a demonstrated use case
ECI = Enzian Coherence Interface (native coherence link, 128 B cache lines, ~30 GiB/s). Coyote = the group's FPGA 'OS' shell providing services on the FPGA. BMC = the open baseboard controller. Numbers from the paper's Section 5 evaluations.
Step 1 of 6
A PCIe accelerator card (the F1/Alveo role)
Running a previously-published gradient-boosting decision-tree inference workload under Coyote, Enzian delivers throughput in the same range as Intel's Harp-v2, Amazon F1, and a Xilinx VCU118 card — tens of millions of tuples per second. The point is not winning but parity: research done on Enzian transfers to the platforms people actually deploy.
same workload, four platforms, one open machine performance "in the ballpark" is the stated bar — and met
Why it works
- Research economics differ from product economics → maxing out parts and ignoring unit cost buys coverage of the whole design space with one board → a throttleable superset machine can emulate the platforms it replaces, so results are comparable at last.
- Coherence protocols move cache lines with low latency; PCIe moves bulk with setup cost → attaching the FPGA by the CPU’s native protocol serves the small-transfer regime AND enables peer-socket roles (memory controller, cache instrument) → the platform covers both the conventional accelerator model (via PCIe crossover) and everything beyond it.
- The group implemented ECI themselves and specified it formally → the protocol is open to researchers down to individual messages, with generated checkers → “is coherence a good idea?” and near-data-processing questions become experiments rather than speculation.
- Writing the BMC firmware was unavoidable (25 regulators, 150 A rails, no vendor firmware) → the necessity became an asset: open telemetry at 20 ms granularity, and spin-off research in solver-generated power sequencing, verified I2C, and seL4-based trustworthy board control.
- Open source plus remote access → one lab’s capital expense becomes community infrastructure → the NetFPGA/PlanetLab model, applied to server-class hybrid computing.
What it costs
- Years of a group’s effort on non-research engineering. Board bring-up by oscilloscope, firmware written from scratch, an 18-layer PCB outsourced for signal-integrity expertise, prototype boards that a single regulator misconfiguration could destroy — the paper is candid that this is a massive detour from “publishing papers.”
- A weak, aging CPU. The ThunderX-1 was the only viable choice given the open-protocol requirement, and it trades single-thread performance for 48 in-order cores — an ARM part from the mid-2010s anchors a platform meant to represent the future.
- ECI bandwidth below modern PCIe. ~30 GiB/s over the full link; single-link experiments run lower, and several benchmarks (write throughput especially) are visibly link-constrained. Latency, not bandwidth, is where ECI shines.
- Not a product. Small board counts, no vendor support ecosystem, and a learning curve of its own — the openness bet only pays if a community actually adopts the machine.
- The landscape moves. CXL — an open, industry-standard coherent interconnect — was arriving as Enzian shipped, and commercial coherent-FPGA attach may commoditize part of what Enzian uniquely offered.
The evaluation
The evaluation is a coverage argument in three movements: (1) parity — Enzian reproduces published workloads from specialized platforms at comparable performance (decision-tree inference vs Harp-v2/F1/VCU118; 100 Gb/s TCP-in-FPGA vs a Xeon+Mellanox pair; RDMA against FPGA DRAM; Catapult’s topology by re-cabling); (2) superset — experiments impossible elsewhere (a computational memory controller under a real CPU workload; explicit coherence-message access); (3) instrumentation — regulator-level power tracing of the machine’s own boot and stress sequence.
Where hybrid-systems researchers could work
| Property | PCIe accelerator cards (F1, Alveo, Alpha Data) | Coherent research platforms (Intel Xeon+FPGA, IBM CAPI) | Enzian |
|---|---|---|---|
| CPU–FPGA connection | |||
| FPGA size and role | Large FPGAs, but locked into the peripheral role; cloud shells proprietary | ||
| Demonstrated performance | Baseline for accelerator workloads | Baseline for coherent workloads | |
| Instrumentation & firmware | None user-visible; closed BMC | None user-visible; closed BMC | |
| Openness & availability | Proprietary shells; cloud regions of the device off-limits | NDAs; limited academic allocation |
Check yourself
Check your understanding
1. Nearly every hybrid platform connects the FPGA over PCIe. Enzian instead uses the CPU's native inter-socket cache-coherence protocol. Predict what this choice enables beyond raw performance.
2. Enzian's design principles include "don't worry about unit cost" and "if in doubt, overengineer" — the exact opposite of product engineering. Why is this rational for a research machine?
3. The group had to write Enzian's baseboard-management firmware from scratch — a pure engineering burden. What did that necessity yield?
4. Why did Enzian end up with a Cavium ThunderX-1 — a mostly in-order ARM CPU with modest single-thread performance — at its heart?
Sources
- Author PDF (enzian.systems) — full text used for this explainer
- ACM DL: Enzian: an open, general, CPU/FPGA platform for systems software research — ASPLOS 2022
- Enzian project site · ASPLOS artifact on Zenodo