Cohort: Software-Oriented Acceleration for Heterogeneous SoCs

Tianrui Wei, Nazerke Turtayeva, Marcelo Orenes-Vera, Omkar Lonkar and Jonathan Balkind

Programming an accelerator normally means bespoke drivers, special allocators and manual cache management — Cohort lets an accelerator sit on the other end of an ordinary lock-free queue, so the software that already knows how to hand work to another thread can hand it to hardware instead.

The argument in one glance

Problem

Every accelerator arrives with its own way of being talked to

Chips now contain many specialised units, and using one means learning its interface: special memory allocation, particular data layouts, explicit cache flushes, a bespoke driver. That knowledge does not transfer to the next accelerator.
What people did

Optimise the accelerator to the maximum and let software adapt

The prevailing philosophy is to make the hardware as fast as possible and leave any mismatch for software to absorb. It produces excellent accelerators and a programming experience that requires deep hardware knowledge to get right.
Why it fell short

The two standard interfaces both fight the software

Memory-mapped I/O forces the core to make round trips that cannot be pipelined or reordered, destroying memory-level parallelism. Coherent DMA needs special allocation, per-block API calls and explicit synchronisation. Both leak hardware into the application.
This paper's idea

Make the accelerator look like another thread on a queue

High-performance parallel software already decouples work through lock-free single-producer single-consumer queues. Those queues have a precise, agreed meaning about when data becomes visible. Build hardware that participates in exactly that contract, and an accelerator becomes something you push to and pop from.

Why this problem is worth solving

The paper opens with an unusually direct philosophical claim: our approach to acceleration focuses on the extreme, optimising the hardware to the maximum and leaving software to fix whatever mismatch results. The authors call this hardware-oriented, and their objection is not that the accelerators are bad but that the interfaces are. Using one today means changing your data formats to what it expects, allocating memory through a special routine, managing cache coherence by hand, and going through a driver written for that device and no other. Every one of those requirements is hardware knowledge that has escaped upward into the application.

The two conventional interfaces both illustrate the problem.

Memory-mapped I/O gives a device a range of addresses that the processor reads and writes directly, so talking to the device looks like touching memory — but each access must reach the device and cannot be cached, reordered or overlapped.

is simple but forces the core into serialised round trips that the compiler must not optimise away, which destroys memory-level parallelism. Direct memory access is faster in bulk but demands special allocators, per-block API calls and explicit synchronisation, so it lands squarely in the application — and it puts the burden of

Cache coherence is the hardware mechanism that keeps the private caches of different processors in agreement, so that when one writes a memory location the others are told their copy is stale.

onto the programmer, who must flush and invalidate by hand. Meanwhile, high-performance software solved the equivalent problem for itself long ago: threads decouple through lock-free queues, and the same shift has already happened in high-performance networking, where shared-memory queues replaced device register operations. Cohort’s question is why hardware accelerators should be an exception.

What people did before, and where it broke

Memory-mapped registers. Write inputs to the device, poll or take an interrupt, read results back. Each access is a round trip that cannot be pipelined, and the compiler must be prevented from optimising or reordering it. The paper’s measurements show cores stalling word by word.
Coherent DMA. Hand the accelerator a buffer and let it fetch. Much better throughput, but the buffer needs a special allocator, transfers need per-block API calls that themselves contain memory-mapped writes, and synchronisation is the application’s problem.
Vendor libraries and domain-specific languages. Wrap the details in a library or generate code from a restricted language. This helps for the class of applications the vendor anticipated and leaves everything else where it was.
Accelerator-specific drivers. The general answer, and the reason the trusted software base grows with every device added. Each accelerator variety brings its own kernel code.
Decoupled access-execute hardware. A long line of work separates data supply from computation so one unit feeds another. Valuable, and aimed at feeding cores rather than at giving applications a uniform way to reach accelerators.
The gap. Every approach asks the application to learn something about the hardware. Nobody had asked whether the abstraction software already uses to hand work between threads could be made to work when the thread on the other end is a circuit.

How Cohort puts an accelerator on a queue

From a bespoke interface to an ordinary producer-consumer queue

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

Every stage after the first is about making hardware honour a contract software had already written down.

The idea

Take the abstraction software already has, and make hardware speak it. The framing the paper argues for, software-oriented acceleration, is that the interface should be something applications already use rather than something the device dictates. Concretely, that means single-producer single-consumer lock-free queues, which parallel software uses everywhere to decouple threads and which have mature high-performance implementations. The whole user-facing API is the queue calls the program already makes, plus two: register an accelerator with an input and an output queue, and deregister it. A program that pushed to a queue for another thread now pushes to a queue for a circuit, and the code barely changes. Treating an abstraction that programmers already write as the specification the hardware must implement, rather than asking programmers to learn the hardware’s abstraction, is the same inversion behind turning existing packet-processing programs directly into network hardware, covered elsewhere on this site.

The coherence protocol was already carrying the signal. The technical core is an observation about what a queue actually guarantees. When a producer finishes an enqueue and the consumer sees the write pointer change, the data behind that pointer is guaranteed visible — a property combining cache coherence and memory consistency that the paper names queue coherence. On a coherent chip, updating that pointer necessarily invalidates the corresponding cache line elsewhere. So the invalidation the hardware already sends is the notification, and there is no need for a doorbell register or an interrupt. Cohort’s endpoints watch for invalidations on the pointer addresses, with a backoff period to avoid hammering the coherence system. Replacing explicit device registers and descriptor rings with participation in the machine’s existing cache-coherent shared memory is precisely the move made by recent work on cache-coherent network interfaces, covered elsewhere on this site, and it works here for the same reason: the coherence fabric is already delivering the message.

Give the accelerator the same memory the program has. The third piece is the least glamorous and does the most work for usability. The Cohort engine carries a full RISC-V compliant memory management unit, derived from the core’s own, with a sixteen-entry TLB and a page table walker that refills transparently and raises an interrupt to a core on a genuine fault. The consequence is that queues are ordinary memory: allocated with malloc, addressed virtually, shared with the process that owns them. There are no separate I/O page tables to maintain, no pinning, no physical addresses in the application, and if the programmer uses huge pages the engine benefits exactly as the cores do. It also collapses the driver problem — one driver serves every Cohort-attached accelerator regardless of what it computes, which keeps the trusted software base from growing with each new device. Letting an accelerator participate in the host’s existing memory and driver abstractions rather than maintaining a parallel universe alongside them is the same goal pursued by work giving accelerators first-class access to host-side infrastructure, covered elsewhere on this site.

Following a block of data through an accelerator

A program hashes data with a SHA accelerator

The application code is the producer-consumer code it would have written for another thread, plus one registration call.

Step 1 of 5

Allocate two queues and register the accelerator

The program creates an input and an output queue with the ordinary library call, then names them in a single registration.

That registration is the entire Cohort-specific API surface alongside its counterpart for deregistration. A queue descriptor tells the engine where this library keeps its read and write pointers, base address, element size and length, so any queue layout can be supported.

Why it works

Queues restore the parallelism that memory-mapped I/O destroys. A core pushing into a queue is not waiting for a device round trip → many elements can be in flight and input can overlap output → which is why the memory-mapped baseline loses by 5.44 to 8.38 times on the hashing benchmark.
Coherence traffic replaces explicit notification. The pointer update already invalidates a line elsewhere → the endpoint learns of new work with no doorbell write and no interrupt → and a backoff period keeps the watching itself from becoming the bottleneck.
Software optimisations transfer to the hardware boundary. Batching pointer updates so they happen once per N elements is a normal multithreaded technique → applied unchanged it reduces coherence pressure at the accelerator interface → and is worth 2.32 to 3.33 times on hashing and 5.30 to 8.10 times on encryption.
A native MMU removes the whole allocation and pinning problem. The engine translates virtual addresses using the same page tables as the process → queues are plain malloc-ed memory and huge pages help automatically → and no separate I/O page tables need maintaining in software.
One driver covers every accelerator. The interface is the queue, not the device → a single Cohort driver serves any attached accelerator → so the trusted software base does not grow with each new piece of hardware.

What it costs (trade-offs and scope)

The engine can be bigger than the accelerator it feeds. Against the small SHA unit, a Cohort engine uses 27% more lookup tables and 57% more registers. It is 68% of the AES accelerator’s lookup tables and 37% of the H264 encoder’s. The fixed cost only amortises for accelerators of reasonable size.
The benefit varies a lot by workload. Hashing gains 5.44 to 8.38 times over memory-mapped I/O; encryption gains only 1.83 to 2.07 times. The paper explains why: AES produces output the same size as its input, so pointer updates in both directions cause false sharing with the core, and its 41-cycle latency leaves less to hide than the 66 cycles of SHA.
Only single-producer single-consumer queues. That is the pattern with the cleanest semantics and the best library support, but multi-producer and multi-consumer arrangements are outside the design.
The backoff period is a tuned parameter. How long the endpoint waits after an invalidation before re-checking trades responsiveness against coherence traffic, and the right value is not derived.
The prototype is small and slow. An FPGA at 100 MHz with two RISC-V cores, two accelerators, 8 KiB of L1 data cache and 64 KiB of L2, running Linux. Enough to demonstrate a real system end to end, far from a production chip.
Batching is applied to Cohort and not to the baselines. That is arguably the point — the argument is that software optimisations become available once the interface is a software abstraction — but it means the batched speedups are not like-for-like against unbatched alternatives.

The evaluation

Property	Measurement	Conventional interfaces	Cohort
SHA benchmark latency	MMIO and coherent DMA baselines
AES benchmark latency	same baselines
Gain from software batching within Cohort	not applicable
Instructions per cycle	cores stall on round trips
Area of the engine	an Ariane core tile

Measured on an OpenPiton and Ariane SoC on a Xilinx Alveo U200 FPGA at 100 MHz, booting Linux 5.6-rc4, with two RISC-V cores and two accelerators.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The headline range from 1.83 to 8.38 times spans two workloads that behave very differently, and the difference is instructive rather than incidental. Hashing consumes much more than it produces, so the queue does what queues are good at: keep the producer running while the consumer works. Encryption produces exactly as much as it consumes, so both pointers are being updated constantly and the core and the engine end up contending on the same cache lines — the paper attributes the weaker result to that false sharing plus an accelerator fast enough that there is little latency left to hide. Read the numbers as a statement about which shapes of work suit a queue interface, not as a uniform speedup. The area figures deserve the same care: the engine is small next to a core but larger than a small accelerator, so this is infrastructure that pays for itself on substantial units and not on tiny ones. Finally, the batching results measure a software optimisation applied to Cohort and not to the baselines. That asymmetry is the paper’s actual argument — the point is that such optimisations only become available once the interface is a software abstraction — but it is worth naming rather than reading the batched numbers as a direct comparison.

Background

Where this sits

There is a recurring pattern in systems work where a community discovers that the device interface it inherited is the thing standing in the way. High-performance networking went through it: register writes and descriptor rings gave way to shared-memory queues, and the paper cites that transition as precedent for its own. Storage went through a version of it with byte-addressable devices. Cohort makes the same argument for on-chip accelerators, and its particular contribution is identifying which property of a software queue the hardware actually has to honour — the guarantee that observing a pointer update implies seeing the data — and then noticing that a cache-coherent machine already delivers that signal for free. The approach is not tied to a single chip either: the authors note their primitives could sit on emerging coherent interconnects rather than only on an on-chip protocol. The wider family of work on how accelerators and network devices should attach to a coherent host is covered elsewhere on this site.

Check yourself

Check your understanding

1. Why does memory-mapped I/O hurt performance so much in this setting?
2. What is queue coherence and why does it matter for hardware?
3. How does a Cohort endpoint learn that new work has arrived?
4. Why does the encryption benchmark gain much less than the hashing one?

Sources

The paper is “Cohort: Software-Oriented Acceleration for Heterogeneous SoCs” by Tianrui Wei, Nazerke Turtayeva, Marcelo Orenes-Vera, Omkar Lonkar and Jonathan Balkind, published at ASPLOS 2023 Volume 3, pages 105-117, DOI 10.1145/3582016.3582059. The authors are at UC Santa Barbara, and the hardware and software are open-sourced at github.com/cohort-project.

This explainer is written from the full text, read from the author-hosted copy on Jonathan Balkind’s site. All figures above come from the paper’s own sections 4 through 6: the SHA speedups of 5.44 to 8.38 times over memory-mapped I/O and 7.27 to 11.24 times over coherent DMA, the AES speedups of 1.83 to 2.07 and 1.69 to 2.03 times, the batching gains of 2.32 to 3.33 times on SHA and 5.30 to 8.10 times on AES, the peak instructions-per-cycle improvements of 4.42 and 2.11 times, the accelerator latencies of 41 and 66 cycles, the area figures including the 1081 lookup tables and 1206 registers of the MMU and the engine being under 4% of an Ariane tile’s lookup tables, and the prototype configuration of an OpenPiton and Ariane SoC on a Xilinx Alveo U200 at 100 MHz booting Linux 5.6-rc4. The paper is also deposited open access in the University of California eScholarship repository.