TAPA-CS: Enabling Scalable Accelerator Design on Distributed HBM-FPGAs

Cloud FPGAs let you build custom hardware accelerators by writing C++, but the tools only target one chip, and modern designs have outgrown a single FPGA — so scaling across a networked cluster means an expert hand-partitioning the design and fighting the tools for a fast clock; this work from UCLA builds TAPA-CS, which automatically splits a large dataflow design across several networked FPGAs, mapping each piece to balance the load and minimize the slow cross-chip traffic, and pipelines the interconnect at every chip and die boundary so the clock stays fast, making 4-FPGA designs 4.4 times faster than a single chip with no manual effort.

The argument in one glance

Problem

Cloud FPGA tools program one chip, but designs have outgrown one chip

FPGAs are reconfigurable chips that clouds rent out as custom accelerators, and high-level synthesis tools let programmers describe one by writing C++ rather than low-level circuits. The catch is that these tools target a single FPGA. As Moore's law fades, accelerator designs have grown larger than one chip can hold, and clouds now offer clusters of FPGAs wired together by fast networks. But there is no tool to program that cluster as one: someone must split the design across chips by hand. Modern FPGAs make this brutal — each has multiple dies with slow boundaries between them, high-bandwidth memory exposed on only one die, and network ports whose transfers cost far more than on-chip data movement.
What people did

Partition the design across chips by hand, or with a narrow tool

The expert response is to manually decide which module of the design goes on which FPGA, wiring up the inter-chip communication in low-level RTL. A few research tools automate pieces of this: some partition convolutional neural networks specifically across FPGAs, some rely on simulation rather than real hardware, and some virtualize the FPGA into pre-placed static regions that user logic maps into.
Why it fell short

Manual partitioning does not scale, and the tools leave the clock slow

Hand-partitioning becomes impractical as a design grows to hundreds of modules. The narrow tools are each limited — tied to one workload like neural networks, confined to simulation, or forced into rigid virtualized regions — and, crucially, none of them consider the chip's physical layout when partitioning. Without that, they cannot floorplan and pipeline the interconnect, so their designs run at low frequency, often 100 to 240 megahertz, and none account for the network topology, which caps how far they scale.
This paper's idea

Automatically partition, place, and pipeline across the whole cluster

TAPA-CS takes a large dataflow design written in C++ and automatically splits it across a cluster of networked FPGAs. An optimizer maps each module to a chip to balance the compute load while minimizing the slow cross-chip traffic, aware of the network's topology. It exploits the fact that dataflow designs tolerate arbitrary communication delay, so it can freely place network links and pipeline registers. And it couples floorplanning at both the cluster and single-chip level with interconnect pipelining, giving the tool a physical view of the board so the clock stays fast.

Why this problem is worth solving

The heart of the difficulty is that a modern FPGA is not a uniform sheet of logic but a landscape of very different costs, which one must respect to get speed.

A field-programmable gate array is a grid of programmable logic that can be configured into a custom circuit, and high-level synthesis compiles a C++ program into that circuit so a programmer need not write hardware directly. But the chip is deeply non-uniform. It is split into multiple dies joined by silicon interposers, and a wire crossing a die boundary is far slower than one staying on a die. Fixed blocks of hard logic — controllers for the network ports, for the off-chip memory — sit at set locations and cannot be moved. The memory hierarchy spans on-chip memory that is tiny but delivers tens of terabytes per second, and high-bandwidth memory that holds gigabytes but is dozens of times slower and, on a typical card, is wired to only the bottom die. Two facts follow. First, where each compute module physically lands matters enormously: a wire that must reach across a die, or a module that congests the die holding the memory ports, quietly wrecks the clock frequency. Second, high-level synthesis, converting untimed C++ into a timed circuit, cannot see where modules will end up, so it inserts too few pipeline registers between them, leaves long connections underpipelined, and ships a slow design. Getting a fast accelerator therefore requires giving the compiler a physical view of the board — and once you must span several such boards across a network, the problem multiplies.

The reason this is worth automating, rather than leaving to experts, is that scaling out helps even designs that fit, and hand-partitioning cannot keep up.

It is a common misconception that spreading a design across multiple FPGAs is only for designs too big for one chip. The paper shows otherwise with a nearest-neighbor accelerator that fits on a single FPGA but runs at only 165 megahertz. The reason is a resource squeeze: to saturate the high-bandwidth memory it wants a wide port and a large buffer, but that configuration overloads the bottom die where the memory ports live, and the design fails to route. Forced to a narrower port and smaller buffer, it saturates only half the memory bandwidth and pays for many more memory accesses, each dozens of times slower than on-chip. Splitting the design across two FPGAs gives each chip’s bottom die room for the optimal wide-port configuration, and divides the input so both chips compute in parallel — the same design now runs at 300 megahertz and twice as fast. So multi-FPGA is not just a capacity story; it exposes more memory bandwidth per compute module and relieves congestion. But doing it by hand means an expert reasoning about die boundaries, memory-port placement, network topology, and load balance for a design with hundreds of modules, which does not scale. The value of a tool that partitions, places, pipelines, and wires up the network automatically is exactly that it makes this reasoning tractable — which is what makes the framework worth building.

What people did before, and where it broke

Partition the design across FPGAs by hand in RTL. An expert decides the module-to-chip mapping and writes the inter-chip communication in low-level hardware description. Where it broke. It demands deep reasoning about die boundaries, memory ports, topology, and load balance, and becomes impractical as designs grow to hundreds of modules.
Automate partitioning for a specific workload like neural networks. Build a tool that splits convolutional or deep networks across FPGAs. Where it broke. It is tied to that one class of workload and does not generalize to stencils, graph algorithms, or other accelerators, and several such tools still reach only 190 to 240 megahertz.
Partition and evaluate in simulation. Use a simulator to explore multi-FPGA mappings without real hardware. Where it broke. Simulation is useful for prototyping but is not a substitute for real execution, and these tools often still require the user to supply the module-to-chip mapping.
Virtualize the FPGA into pre-placed static regions. Map user logic into fixed, pre-placed and pre-routed soft blocks connected over the network. Where it broke. The rigid regions do not scale to large real designs where each function compiles into its own state-machine-controlled circuit, and the approach yields low frequency (around 100 megahertz) partly because it ignores the physical chip layout and uses slow interconnects.
The gap. Every approach either did not scale, worked for only one workload, stayed in simulation, or ignored the physical layout and network topology — so none could couple intelligent floorplanning and interconnect pipelining with the partitioning needed for a fast, general, real multi-FPGA design. That coupling is exactly what scalable accelerator design across a cluster needs.

How TAPA-CS works

Splitting one C++ design across a cluster of FPGAs, fast

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Partition across the chips by minimizing the slow cross-chip traffic, aware of where things sit. The whole cost structure of a multi-FPGA design is that moving data on-chip is cheap, across a die is dearer, and across the network to another chip is dearest, so a good split keeps tightly-communicating modules together and pays the network cost only where it must. TAPA-CS casts this as an optimization that weighs every channel by its traffic and a topology-aware distance between the chips its endpoints land on, and it deliberately does not always take the smallest cut — it will push a module off-chip if keeping it on would congest the die and drop the frequency. Mapping a design across a multi-chip substrate by explicitly minimizing the expensive cross-boundary communication, trading fast local links against slow distant ones, is the same problem the ququart-compilation work covered elsewhere on this site solves for quantum circuits, routing operations to exploit cheap on-unit interactions and avoid the costly cross-unit ones.

Hide the network behind the computation using latency-insensitive dataflow. A cluster’s network is slow compared to on-chip wires, so if the design stalled waiting on each transfer it would never scale. TAPA-CS leans on the fact that its dataflow modules are latency-insensitive — they communicate through queues and tolerate arbitrary delay without breaking correctness — so a transfer in flight does not block a module that has other work, and computation on one chip overlaps communication with another. Overlapping slow communication with useful computation so the transfer never sits on the critical path is exactly the discipline the communication-overlapping scheduler covered elsewhere on this site applies to distributed training, keeping the expensive movement running quietly underneath the math that was going to happen anyway.

Raise the abstraction so the programmer never sees the cluster. The expert work TAPA-CS replaces — reasoning about die boundaries, memory-port placement, network topology, and load balance — is exactly the kind of physical detail a programmer should not have to hold in their head. TAPA-CS takes an ordinary dataflow C++ program and hides the whole cluster behind it, automatically partitioning, placing, pipelining, and wiring the network so the user writes as if for one abstract machine. Automating the intricate, hardware-specific mapping that once demanded a specialist, so the programmer expresses intent and the compiler handles the substrate, is the same lift the compute-in-memory compiler work covered elsewhere on this site provides, turning a hand-crafted mapping onto exotic hardware into something a compiler produces from a high-level description.

Following a nearest-neighbor accelerator through TAPA-CS

Scaling a memory-bound KNN accelerator across FPGAs

Tracing the KNN design shows how splitting the chip unlocks memory bandwidth.

Step 1 of 5

A nearest-neighbor accelerator is starved on one chip

The design computes each query's distance to millions of data points and sorts for the closest. It fits on one FPGA, but to saturate the high-bandwidth memory it needs a wide port and a large buffer — a configuration that overloads the bottom die where the memory ports sit and fails to route, so it settles for a narrow port at 165 megahertz.

The narrow buffer forces many more memory accesses, each dozens of times slower than on-chip.

Why it works

Giving the compiler the board layout and pipelining every crossing lifts the clock. High-level synthesis cannot see where modules land, so it underpipelines and the clock stays low → but floorplanning the design and inserting pipeline registers at every slot crossing keeps critical paths short → so the frequency climbs from around 165 to 300 megahertz.
Splitting the design relieves the die congestion that blocked the best memory configuration. Cramming the wide-port, large-buffer memory setup onto one chip overloads the die holding the memory ports and fails to route → but spreading the design gives each chip’s die room for that optimal setup → so memory-bound designs finally saturate the bandwidth they could not on one chip.
A topology-aware cost keeps the expensive cross-chip traffic small. Communication across the network is far slower than on-chip, so a careless split pays dearly → but weighing each channel by its traffic and the network distance between its endpoints steers the optimizer → so chatty modules stay together and the costly traffic is minimized while load stays balanced.
Latency-insensitive dataflow lets links and registers go anywhere. Ordinary circuits break if you change interconnect timing, forbidding free placement of network links → but dataflow modules communicate through queues and tolerate arbitrary delay → so pipeline registers and slow network hops can be inserted anywhere without altering correctness.
A constant inter-chip transfer as compute grows makes added chips nearly free. If communication ballooned with every added chip, scaling would stall → but for designs whose inter-chip transfer size is fixed regardless of scale, added chips run in parallel with little extra traffic → so those designs scale near-linearly or better, reaching roughly six times on four FPGAs.

What it costs (trade-offs and scope)

Scaling is governed by the compute-to-communication ratio, and varies widely. Designs whose inter-chip transfer stays constant as compute grows, like the graph-ranking and nearest-neighbor kernels, scale superlinearly (up to about six times on four FPGAs), while compute-bound designs with large or growing transfers, like a high-iteration stencil or a systolic convolution with many modules contending for the network, scale far less (around two-and-a-half times).
Some topologies force chips to run sequentially or contend. The stencil design makes later FPGAs idle while earlier ones execute, and the systolic array has many modules writing to the network at once, causing contention and idle compute — so the benchmark’s structure, not just its size, limits the gain.
The input must be a latency-insensitive dataflow program in a specific format. TAPA-CS works on designs expressed as C++ functions communicating through queues, not arbitrary code, and is tied to a particular toolchain and to Xilinx and AMD Alveo hardware.
The optimizer is exact but adds compile time that grows with size. The integer-linear-programming partitioning adds up to about fifty seconds and the floorplanning up to roughly thirty-eight seconds combined for the largest design of several hundred modules — modest, but scaling with design size.
Networking carries real, if small, overheads and limits. The reliable-Ethernet stack costs about five percent of resources plus roughly two percent per network port, supports up to a few thousand connections, and the evaluation uses only the Ethernet path, not the PCIe alternative the framework also supports.
It needs the real cluster, and the gains are workload- and setup-specific. Results come from real FPGAs on one particular cluster of Alveo cards in a ring, across four benchmark families up to eight FPGAs; the speedups depend on the workload and the cluster, and the approach presumes the hardware actually exists rather than being simulated.

The evaluation

Three ways to build a large FPGA accelerator

Property	Single-chip HLS (Vitis)	Prior multi-FPGA tools
Partitioning across chips
Sees the physical chip layout	No; underpipelines the interconnect
Design frequency	About 165 megahertz
Inter-FPGA communication	None
Speedup versus one Vitis FPGA	Reference	Limited, not generalizable

Three ways to build a large FPGA accelerator, evaluated on real Xilinx Alveo U55C cards in a ring across four benchmark families — a stencil, graph page-ranking, nearest-neighbor search, and a systolic convolution — against single-FPGA Vitis HLS and TAPA baselines. TAPA-CS automatically partitions each design across the cluster, delivering average speedups of 2.1, 3.2, and 4.4 times on 2, 3, and 4 FPGAs and raising the design frequency 11 to 116 percent, because it couples topology-aware partitioning with floorplanning and interconnect pipelining that single-chip tools and prior multi-FPGA tools lack.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

TAPA-CS is a substantial engineering result: the first framework to automatically partition a general dataflow C++ design across a cluster of networked FPGAs while achieving high frequency. Its strength is coupling three things prior work kept separate — a topology-aware optimizer that balances compute load against slow cross-chip traffic, a physical floorplanning of each chip that pipelines every die and slot crossing to keep the clock fast, and lightweight Ethernet communication with a one-microsecond round trip. It also dispels a misconception, showing that spreading even a fitting design across chips exposes more memory bandwidth per module. The results are strong: 2.1, 3.2, and 4.4 times average speedups on 2, 3, and 4 FPGAs, with frequency up 11 to 116 percent over single-chip Vitis. Read it with its scope. How well a design scales depends heavily on its compute-to-communication ratio — kernels with constant inter-chip transfer scale superlinearly while compute-bound or contention-heavy ones scale far less — and some topologies force chips to run sequentially. The input must be a latency-insensitive dataflow program in a specific format, tied to one toolchain and Alveo hardware; the exact optimizer adds compile time that grows with design size; and the gains are measured on one real cluster across four benchmarks. The durable lesson is that spanning a design across chips is worth it not only when it must be, but whenever the extra chips buy bandwidth or relieve congestion — provided a tool handles the physical partitioning the way an expert once did by hand.

Background

Why hardware acceleration keeps running into the bandwidth wall

It helps to see TAPA-CS against a pattern that recurs across accelerator design: the bottleneck is usually not the arithmetic but feeding it data. The whole point of an FPGA accelerator is to lay down exactly the compute a workload needs, yet the paper’s own examples are governed by memory, not math — a design that fits on one chip still runs slow because it cannot saturate its high-bandwidth memory, and the single biggest reason to add a second chip is to expose more memory bandwidth per compute module. On-chip memory is dozens of times faster than the high-bandwidth memory, which is itself dozens of times faster than crossing the network, so the engineering is a constant fight to keep the fast compute fed from the right tier. The deep-learning-on-GPUs retrospective covered elsewhere on this site draws the general version of this lesson, that once arithmetic units grew cheap the ceiling moved to moving the data, so the useful work shifted from accelerating compute to marshaling memory traffic. TAPA-CS is that lesson applied to reconfigurable clusters: it wins not by computing faster but by giving each piece of the design its own memory ports and overlapping the slow network behind computation. The broad point is that for accelerators as for processors, throughput is decided less by peak compute than by how well the memory and communication hierarchy is exploited — which is why partitioning, placement, and bandwidth management, not raw logic, are where the gains live.

If you remember only three things

1. Cloud FPGAs let programmers build custom accelerators in C++ via high-level synthesis, but the tools target only one chip, and as designs outgrow a single FPGA, using a networked cluster has meant an expert hand-partitioning the design across chips in low-level hardware. Modern FPGAs make this brutal: multiple dies with slow boundaries, high-bandwidth memory wired to only one die, fixed hard-logic blocks, and network transfers far costlier than on-chip movement. Because high-level synthesis cannot see where modules physically land, it underpipelines the interconnect and ships slow designs (around 165 megahertz), and prior multi-FPGA tools were workload-specific, simulation-only, or topology-blind, reaching just 100 to 240 megahertz.
2. TAPA-CS automatically partitions a latency-insensitive dataflow C++ design across a cluster: it models the design as a graph of modules connected by queues, uses an integer-linear-programming optimizer to map each module to a chip that balances compute load while minimizing cross-chip traffic weighted by a topology-aware distance, and exploits latency-insensitivity to place network links and pipeline registers freely. It then floorplans within each chip into slots bounded by dies and hard blocks (minimizing cross-die wire distance and binding memory channels to avoid congestion), pipelines every slot crossing to keep the clock fast, wires up one-microsecond Ethernet communication, and hands the result back to the vendor CAD flow. Crucially it shows multi-FPGA helps even fitting designs, by exposing more memory bandwidth per module.
3. On real Alveo U55C cards in a ring across four benchmarks (stencil, PageRank, KNN, systolic CNN), TAPA-CS delivers average speedups of 2.1, 3.2, and 4.4 times on 2, 3, and 4 FPGAs and raises frequency 11 to 116 percent over single-chip Vitis. Its limits: scaling depends on the compute-to-communication ratio — constant-transfer kernels (PageRank, KNN) scale superlinearly to about six times on four FPGAs, while compute-bound or contention-heavy ones (high-iteration stencil, systolic CNN) reach only about two-and-a-half times; some topologies force chips to run sequentially; the input must be a specific dataflow format tied to one toolchain and Alveo hardware; the exact optimizer adds compile time; and the gains are measured on one real cluster.

Check yourself

Check your understanding

1. Why does high-level synthesis produce FPGA designs with a low clock frequency?
2. Why does splitting a design across multiple FPGAs help even when it already fits on one?
3. How does TAPA-CS decide which modules go on which FPGA?
4. Which statement best reflects the scope and limits of TAPA-CS?

Sources

The paper is “TAPA-CS: Enabling Scalable Accelerator Design on Distributed HBM-FPGAs” by Neha Prakriya, Yuze Chi, Suhail Basalama, Linghao Song, and Jason Cong, published at ASPLOS 2024, DOI 10.1145/3620666.3651347. The authors are at UCLA.

This explainer is written from the full 15-page text. Everything above comes from the paper’s own sections: the introduction and background motivating cloud multi-FPGA acceleration, the physical complexity of modern FPGAs (multiple dies, high-bandwidth memory on the bottom die, hard IP blocks, varying on-die, cross-die, and cross-chip transfer costs), and why high-level synthesis underpipelines designs; the motivating KNN example showing a fitting design limited to 165 megahertz by memory-port congestion and rising to 300 megahertz and twice as fast across two chips; the TAPA-CS design with its graph model of modules and FIFO channels, its integer-linear-programming inter-FPGA partitioner minimizing topology-aware communication cost under resource constraints (with the daisy-chain and ring distance metrics and the choice not to always min-cut), the AlveoLink-based reliable Ethernet communication with a one-microsecond round trip, the intra-FPGA slot-based floorplanning and automatic high-bandwidth-memory channel binding, and the conservative interconnect pipelining with reconvergent-path balancing; and the evaluation on real Alveo U55C cards in a ring across stencil, PageRank, KNN, and systolic-array CNN benchmarks, reporting average speedups of 2.1, 3.2, and 4.4 times on 2, 3, and 4 FPGAs, frequency improvements of 11 to 116 percent (165 to 300 megahertz for the stencil, 123 to 266 for PageRank, 165 to 220 for KNN), the compute-to-communication analysis explaining superlinear scaling for PageRank and KNN versus limited scaling for the stencil and CNN, the floorplanning overheads of tens of seconds, and the comparison against prior tools like Elastic-DF and SMAPPIC.