Scaling Up Memory Disaggregated Applications with SMART

Fast networks now let a machine use memory that physically lives in another box, but applications doing many small remote accesses peak at around 32 cores and then get SLOWER as you add more — on machines that today have hundreds; this paper (Tsinghua and Meituan) opens up the network card and finds three separate contention points, none of them visible to an application programmer and each only becoming the bottleneck once the previous one is fixed, starting with a hardware register nobody knew they were sharing that consumes 74 percent of execution time in a spinlock at 96 threads.

The argument in one glance

Problem

Adding cores makes it slower, not faster

Fast networks now let one machine read and write memory that physically lives in another, which is attractive for utilisation, elasticity and failure isolation. The catch is that applications built this way — key-value stores, transaction systems, caches — do enormous numbers of small remote accesses, so their speed is set by how many operations the network can complete per second rather than by the processor. Measured on real systems, their throughput PEAKS at around 32 cores and then declines as more are added. Datacentre machines today have dozens or hundreds of cores, so most of the machine sits unusable.
What people did

Give each thread its own connection

The received wisdom is that the network card struggles when many threads share connection state, so give every thread its own. This is what the strongest systems in this area do, and it is a reasonable diagnosis — prior work observed the poor scaling and attributed it to contention in a cache inside the card, where connection metadata is held. Per-thread connections remove the obvious sharing and do improve matters at moderate thread counts.
Why it fell short

The sharing you removed was not the sharing that mattered

Profiling at 96 threads shows something else entirely: up to 74 percent of execution time sits inside a spinlock. Not one the application wrote, and not one guarding the cache — it guards a DOORBELL REGISTER, a hardware structure inside the card that a thread writes to announce it has work. Connections are mapped onto a small number of these automatically, so threads with entirely separate connections still collide on one, invisibly. And this is only the first of three bottlenecks: fixing it exposes a second, and fixing that exposes a third, each requiring knowledge of the card that application developers do not have and vendors do not publish.
This paper's idea

Find all three, then hide them behind an ordinary-looking interface

This paper (Tsinghua and Meituan — Ren, Zhang, Chen, Xia, Chen, Wu) identifies each bottleneck by measurement and addresses it. Allocate doorbells per thread, which requires reverse-engineering the card driver since no interface exists to request one. Throttle outstanding requests with a limit the system searches for at runtime, because the right value depends on both thread count and workload. And back off exponentially on failed atomic operations, since a faster network makes collisions more frequent and each failure costs three more remote requests. All of it sits behind an interface resembling ordinary remote-memory operations: 16 to 48 lines of change to three existing systems, for speedups of 2 to 132 times.

Why this problem is worth solving

The workloads in question have an unusual cost structure, and the vocabulary is worth fixing before the diagnosis.

MEMORY DISAGGREGATION separates compute from memory: a machine runs the application but much of the data lives in other machines, reached over a fast network using REMOTE DIRECT MEMORY ACCESS, which lets one machine read or write another’s memory without involving the remote processor at all. Software posts a WORK REQUEST to a QUEUE PAIR, a connection endpoint held by the network card, and the card carries it out. The applications that suit this arrangement — key-value stores, transaction processors, caches — issue vast numbers of tiny operations, so what limits them is not bandwidth but OPERATIONS PER SECOND. The distinction is sharp: the authors measure that once a transfer exceeds 64 bytes the workload becomes bandwidth-limited instead, and everything in this paper concerns the smaller accesses. Their card tops out at 110 million operations per second, which is the number the whole design chases

The first bottleneck is invisible in a specific and instructive way.

Posting work to the card is not simply writing to memory. The thread must ring a DOORBELL — write to a special hardware register on the card — to announce that a request is waiting. There are very few of these registers: twelve by default on the hardware used here, and at most 512 even on a recent card, against machines with hundreds of cores and applications creating far more connections than that. Connections are assigned to doorbells automatically by the driver, and the assignment is not exposed by any interface, so a programmer who carefully gives each thread its own connection has no way to know that two of those connections landed on the same doorbell — where a lock serialises them. The contention is IMPLICIT: it happens between threads that share nothing the programmer can see. This is why the field had attributed poor scaling to the card’s internal cache instead; the real answer was a register with no name in the programming interface

What people did before, and where it broke

Share one connection between all threads. The simplest arrangement, and it keeps the card’s connection state small.
But it serialises everything. All threads contend for the same queue, so throughput is capped well below what the hardware can do.
Share each connection between a few threads, grouped by processor socket. A middle path that reduces connection count while limiting how far a thread reaches across the machine.
But the contention only shrinks rather than disappearing, and the thread count that fits is still far below what modern machines offer.
Give every thread its own connection. This is what the strongest recent systems do, and it removes the visible sharing.
But it does not remove the invisible sharing. Connections are mapped onto a small pool of doorbell registers automatically, so threads still collide — at 96 threads, up to 74 percent of execution time is spent in the resulting spinlock, and the configuration cannot scale past 24 threads at all.
Issue more requests per thread to hide network latency. Deeper pipelining is the standard way to cover round-trip time, and it genuinely helps up to a point.
But past that point it destroys the card’s cache. With 96 threads, 32 outstanding requests per thread yields only 49.5 percent of the throughput of eight, because the card’s memory traffic per request nearly doubles.
The gap. Every prior answer addressed one visible resource. The real situation is three separate contention points, each of which only becomes the bottleneck once the previous one is removed, all of them below the level the programming interface describes.

How three hidden bottlenecks get peeled apart

Fix the doorbell, then the cache, then the retries — each is exposed by fixing the last

Give each thread its own connection and pipeline requests

The standard arrangement removes visible sharing of connection state and issues several requests per thread to cover network round-trip time. It is a reasonable design and is what the strongest systems in this area already do.
↓ →
Work out which doorbell a connection will get, and group by thread new

No interface exists to bind a connection to a particular doorbell register, so read the driver source and discover the assignment is round-robin and therefore predictable. Allocate connections so that all of a thread own connections share one doorbell and no doorbell is shared between threads.
↓ →
Raise the supply of doorbells to match the core count new

Only twelve are available by default. A driver environment variable increases this, and a small driver patch is needed to lift its own cap, giving up to 512 on recent cards — comfortably more than the cores in a machine, though far fewer than the connections an application creates.
↓ →
Limit outstanding requests with a credit scheme new

Track how many requests each thread has in flight and refuse to post more beyond a limit, releasing credit as completions arrive. This keeps the card working set small enough that its cache holds what it needs rather than thrashing.
↓ →
Search for the right limit at runtime rather than fixing it new

The best limit depends on both thread count and workload, so no constant works. Periodically try five candidate values for eight milliseconds each, keep the one that completed the most operations, and run with it for the next half second before searching again.
↓ →
Back off when atomic operations collide, adaptively new

A faster network makes concurrent updates collide more often, and each failed compare-and-swap costs three more remote requests. Delay each retry by a doubling, randomised interval, and adjust the ceiling on that delay from the measured retry rate, which converges over a thousandfold range depending on skew.

Implemented as a 3,000-line C++ library with an interface resembling ordinary remote-memory operations, used to refactor a hash table, a transaction system and a B-tree with 44, 16 and 48 lines changed respectively.

The idea

The bottleneck the field assumed was not the bottleneck. Prior work had observed that these systems scale poorly and attributed it to contention in a cache inside the network card, which is a plausible story: the card caches connection metadata, and more connections means more metadata. This paper profiles the situation instead of reasoning about it, and finds up to 74 percent of execution time inside a spinlock guarding a DOORBELL REGISTER — the hardware location a thread writes to announce it has posted work. The distinction matters because it changes what the fix is. If the cache were the problem, fewer connections would help; because the doorbell is the problem, the fix is to control which connections share which doorbell, which is not something the interface lets you request. The authors read the driver source, discover the assignment is round-robin and therefore predictable, and exploit that determinism to arrange the mapping they want. Replacing a plausible explanation with a measured one, and discovering the fix lies somewhere the interface does not reach, is the same look-at-what-is-actually-happening discipline covered elsewhere on this site where a long-standing assumption about where time goes turns out to be wrong once someone profiles it.

Each fix reveals the next bottleneck, and they are not the same shape. What makes this paper useful as a case study is that removing one limit does not finish the job — it moves the limit somewhere else, and the new limit responds to a different variable. Fix the doorbells and the card’s cache does become the constraint, but keyed on the number of OUTSTANDING REQUESTS rather than the number of connections, so the fix is a credit limit on requests in flight. Fix that, and the network genuinely gets faster — which makes concurrent updates collide more often, so a hash table whose failed atomic operations each cost three more remote requests now spends its new capacity on retries. The measurement is stark: 67.9 percent of updates suffer at least one failed retry, averaging 11.5 retries per update at 96 threads. Recognising that a performance investigation is not finished when the first bottleneck falls, because the system’s behaviour changes underneath you, is the same peel-the-next-layer discipline covered elsewhere on this site where a fix in one place shifts the constraint to a resource that was previously slack.

No constant is correct, so search at runtime. Two of the three fixes need a tuning value, and in both cases the right value depends on things the library cannot know in advance. The number of outstanding requests that maximises throughput depends on both how many threads are running and what they are doing; prior work assumed a fixed depth suffices, but the authors show that even a small depth becomes a problem once there are more than a hundred threads. So the library runs a tiny online search: try five candidate limits for eight milliseconds each, keep the winner, run with it for 480 milliseconds, repeat — spending under eight percent of its time exploring. The backoff ceiling is found the same way, adjusted from the measured retry rate, and converges to values a thousandfold apart depending on whether the workload is skewed or uniform. Replacing a constant nobody can choose correctly with a cheap measurement the system takes for itself is the same let-the-running-system-decide instinct covered elsewhere on this site where a policy is enabled or disabled according to its own observed hit rate rather than fixed at design time.

Peeling three bottlenecks in order

Following the throughput as each hidden limit is removed

All figures measured on the paper's own hardware, whose card tops out at 110 million operations per second.

Step 1 of 5

Where it starts: slower with more cores

An application doing small remote reads and writes is given more threads. Throughput rises, peaks around 32 cores, and then falls. With each thread holding its own connection and sixteen requests in flight, the arrangement cannot scale past 24 threads at all — which is the configuration the strongest published systems use.

this matters because datacentre machines now have dozens or hundreds of cores, so most of the machine cannot be used
the applications affected are those limited by operation count rather than bandwidth, which for this hardware means transfers of 64 bytes or less

Why it works

Threads with separate connections still share doorbell registers, and the driver assignment is deterministic → connections can be grouped so each thread owns its doorbell → removing a spinlock that consumed 74 percent of execution time and reaching the card’s 110 million operations per second limit.
Card cache pressure tracks outstanding requests rather than connections → a credit limit on requests in flight keeps the working set small → avoiding the drop to 49.5 percent of peak throughput seen at 32 outstanding requests per thread.
The best limit depends on thread count and workload together → searching five candidates for eight milliseconds each and running the winner for 480 finds it online → giving up to five times the throughput of per-thread connections without any constant to choose.
A faster network increases collisions between concurrent updates, and each failed atomic costs three more requests → backing off exponentially with a ceiling adapted from the measured retry rate → cutting average retries per update from 11.5 to 1.1.
All three fixes need knowledge of undocumented card internals → wrapping them in an interface resembling ordinary remote-memory operations → so existing systems adopt them in 16 to 48 lines of change.

What it costs (trade-offs and scope)

It requires patching the network card driver. Only twelve doorbell registers are available by default and the driver caps its own supply, so a small modification is needed. The authors argue vendors should expose this properly.
It depends on undocumented behaviour. The doorbell assignment was found by reading driver source. It works for any vendor whose assignment is deterministic, but nothing guarantees that, and a driver update could change it.
The runtime search cannot track fast workload changes. Because it measures over epochs, a workload shifting faster than roughly half a second costs up to 13 percent throughput, since the chosen limit goes stale before it is re-evaluated.
The 132-times headline is a peak, not a typical figure. It comes from comparing against a baseline at high thread counts where that baseline actively degrades. The transaction system and B-tree gains are 5.2 and 2.0 times.
One baseline was reimplemented by the authors. The published hash table’s code is not available, so they wrote their own version to compare against, which they state plainly.

The evaluation

Three real systems were refactored: a lock-free hash table following a published design, a persistent transaction processor derived from an existing system, and a B-tree based on another. Line counts are reported honestly small — 44 of 1,078, 16 of 3,018 and 48 of 1,210. Workloads came from standard benchmark suites with write-heavy, read-heavy and read-only mixes and varying skew, plus transaction benchmarks. Alongside end-to-end numbers the paper reports a breakdown showing what each of the three techniques contributes, which turns out to depend strongly on the workload.

Scaling a small-remote-access application across many cores

Property	One connection shared by all	One connection per thread	Thread-aware, throttled, backed off
Doorbell register sharing
Threads before it stops scaling
Requests in flight per thread
Behaviour under update contention
Peak operations per second

Against per-thread connections, thread-aware allocation gives 1.0 to 4.3 times the throughput and request throttling a further gain to 5.0 times overall. Refactored applications improved by up to 132.4 times for the hash table, 5.2 times for transactions and 2.0 times for the B-tree, with median hash-table latency down 69.6 percent. The techniques matter unevenly: doorbell allocation dominates read-only workloads, throttling matters most between 8 and 32 threads, and backoff dominates skewed write-heavy ones.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Two dependencies deserve attention. The doorbell fix requires patching the network card driver, because only twelve registers are available by default and the driver caps its own supply — and more delicately, the mechanism for choosing which register a connection gets was found by reading driver source, since no interface exposes it. It works because the assignment happens to be round-robin and therefore predictable, which holds for any vendor whose policy is deterministic but is guaranteed by nothing; the authors say directly that vendors should provide an explicit binding interface. The runtime search has its own limit, measured rather than glossed: because it evaluates candidates over epochs of roughly half a second, a workload changing faster than that costs up to 13 percent throughput, since the chosen limit goes stale before being re-evaluated. On the numbers, read the 132-times figure as a peak against a baseline that actively degrades at high thread counts rather than as a typical result; the transaction and B-tree gains of 5.2 and 2.0 times are more representative of what the techniques buy. One baseline was reimplemented by the authors because the published code is unavailable, which they state plainly. Set against that, the paper is unusually good at showing where its value comes from: the breakdown reveals that each of the three techniques dominates a different workload — doorbell allocation for read-only, throttling between 8 and 32 threads, backoff for skewed writes — so none is carrying the result alone.

Background

When the resource you are contending for has no name in the interface

This paper sits in the site’s datacenter and memory systems clusters and rhymes with several ideas covered elsewhere on this site. Its central discovery — that the field had blamed the wrong structure, and that the real contention was on a hardware register the programming interface never mentions — is the same profile-before-you-theorise correction the microservice-debugging and graph-mining pages make when a long-accepted explanation turns out not to describe the system any more. Its shape is also familiar: fixing one bottleneck exposes the next, and each responds to a different variable, which is the peel-the-next-layer structure the page-table page shows when removing a sequential chain simply relocates the cost to bandwidth. That page shares a sharper parallel too, since both find that more parallelism is not free but purchased with a resource nobody was counting. Its use of a tiny online search instead of a constant, because the right value depends on things no designer can know in advance, is the same let-the-running-system-decide instinct the serverless-caching page shows when it enables and disables a policy from its own measured hit rate. And its framing — that these details are real, decisive, and nonetheless something application developers should never have to learn — is the hide-the-machine-behind-an-abstraction argument the custom-instruction page makes when it puts a processor timing model behind two numbers so that domain experts can work without it.

If you remember only three things

1. Applications that use memory in another machine over a fast network are limited by operations per second rather than bandwidth, and measurements show their throughput peaks around 32 cores and then DECLINES — leaving most of a modern many-core machine unusable. Prior work had attributed this to contention in a cache inside the network card. Profiling shows otherwise: at 96 threads, up to 74 percent of execution time sits in a spinlock guarding a DOORBELL REGISTER, the hardware location a thread writes to announce work. Only twelve exist by default, connections are assigned to them automatically, and no interface exposes the choice — so threads that carefully avoid sharing connections still collide invisibly.
2. Fixing that exposes a second bottleneck of a different shape. The card cache does then become the limit, but keyed on the number of requests IN FLIGHT rather than the number of connections: at 96 threads, 32 outstanding per thread yields 49.5 percent of the throughput of eight, with card memory traffic per request rising from 93 to 180 bytes. And fixing that exposes a third, because a genuinely faster network makes concurrent updates collide — each failed atomic compare-and-swap in the hash table costing three more remote requests, averaging 11.5 retries per update, with 67.9 percent of updates failing at least once.
3. None of the three tuning values can be a constant, since the right outstanding-request limit depends on thread count and workload together and the right backoff ceiling varies a thousandfold with skew, so the library searches for both at runtime — trying five candidates for eight milliseconds each and running the winner for 480. Throughput reaches the card own 110 million operations per second ceiling, retries fall to 1.1 per update, and because all of it hides behind an interface resembling ordinary remote-memory operations, three existing systems adopted it in 44, 16 and 48 lines of change for gains of 132, 5.2 and 2.0 times.

Check yourself

Check your understanding

1. Giving every thread its own connection removes the obvious sharing, yet the configuration still cannot scale past about 24 threads. Predict what is still shared.
2. Pipelining more requests per thread hides network latency and helps — until it does not. At 96 threads, 32 outstanding per thread gives only 49.5 percent of the throughput of eight. Predict why, and why a fixed limit cannot be the answer.
3. After both network bottlenecks are removed, the hash table gets slower rather than faster at high thread counts. Predict why making the network faster hurt the application.
4. The published hash table this paper compares against was evaluated by its own authors and showed no such problem. Predict why they did not see it.

Sources

Scaling Up Memory Disaggregated Applications with SMART (author PDF, Tsinghua MADSys) — ASPLOS 2024; the full 16-page paper, read in full for this explainer.
ACM Digital Library (DOI 10.1145/3617232.3624857) — published version.
SMART source code — the authors’ open-source release.