A Quantitative Analysis and Guidelines of Data Streaming Accelerator in Modern Intel Xeon Scalable Processors

Reese Kuper, Ipoom Jeong, Yifan Yuan, Ren Wang, Narayan Ranganathan, Nikhil Rao and 4 others

Modern datacenter CPUs burn 30-50% of their cycles on plain data movement — copying, comparing, checksumming memory — a "datacenter tax" that steals compute from real applications and pollutes the cache for everything running alongside; Intel's 4th-gen Xeon (Sapphire Rapids) ships a new on-chip Data Streaming Accelerator (DSA) to offload that work, but it is a knob-heavy device (shared vs dedicated queues, batching, engine counts, cache hints, sync vs async) whose behavior was undocumented, so this study (Illinois and Intel) measures DSA exhaustively and distills programmer guidelines — offload pays above 4 KB synchronously but as low as 256 bytes if you overlap it asynchronously, modest batching of 4-8 descriptors is the sweet spot, and the biggest surprise is that offloading memcpy mainly helps by AVOIDING cache pollution (software memcpy raised a co-running app's latency 43%) — then proves the guidelines on DPDK packet forwarding for a 1.14-2.29x speedup.

The argument in one glance

Problem

CPUs waste a huge fraction of their cycles just moving data around

In a datacenter, a large share of CPU time never runs the actual application — it is spent on plumbing: copying buffers, filling memory, comparing regions, computing checksums, moving packets. This overhead is the "datacenter tax," and for large packet processing it eats 30-50% of CPU cycles. Worse, doing these copies on the CPU also pollutes the shared cache, evicting the working sets of co-located latency-sensitive services and hurting their tail latency. Since the end of Dennard scaling means CPU cores are not getting dramatically faster, burning half of them on data movement is a direct, expensive loss — cycles that should be running paying work.
What people did

Add a dedicated data-movement accelerator on the chip

Intel's answer in its 4th-generation Xeon (Sapphire Rapids) is the Data Streaming Accelerator (DSA), an on-chip engine built to take data-movement operations off the cores. It handles memory copy, fill, compare, CRC32 checksums, delta records, and data-integrity fields, and it improves substantially on the older CBDMA/I-OAT DMA engines: much lower offload overhead, more operations, and — crucially — no need to pin memory, because it shares the process's virtual address space. Offload the plumbing to DSA, the thinking goes, and give the cores back to the applications.
Why it fell short

It is a knob-heavy accelerator, and nobody knew how to drive it

Having the hardware is not the same as using it well. DSA exposes a thicket of configuration choices, each with unclear performance implications: shared work queues (submitted lock-free with ENQCMD) versus dedicated queues (lower-latency MOVDIR64B); how many descriptors to BATCH per submission; how many processing engines to configure; whether to signal completion by polling, interrupt, or a low-power wait; whether to submit SYNCHRONOUSLY (wait for the copy) or ASYNCHRONOUSLY (overlap it); and cache-control hints that decide whether moved data lands in the cache or bypasses it. Get these wrong and offloading can be SLOWER than a plain CPU memcpy — the submission overhead alone dooms small transfers. Without a systematic characterization, a programmer had no principled way to know when DSA helps or how to configure it, so the accelerator's potential went unrealized.
This paper's idea

Measure it exhaustively, then distill actionable guidelines

This paper (University of Illinois and Intel — Kuper, Jeong, Yuan, Hu, Wang, Ranganathan, Kim) is a quantitative study that measures DSA across all its dimensions and turns the results into rules a programmer can follow. The headline findings: synchronous offload beats CPU memcpy only above 4 KB, but ASYNCHRONOUS offload (overlapping the copy with computation) wins as low as 256 bytes; modest BATCHING of 4-8 descriptors is the sweet spot; and — the biggest surprise — the main benefit of offloading is not the CPU cycles saved but the CACHE POLLUTION avoided, since a co-located software memcpy raised a memory-intensive neighbor's latency by 43% while DSA barely touches the cache. It packages these into six guidelines and validates them on a real application, DPDK packet forwarding, where applying them yields a 1.14-2.29x speedup and a flat forwarding rate across packet sizes.

Why this problem is worth solving

The value of an on-chip accelerator is only realized if software can actually use it well, and that depends on understanding a cost structure that is far from obvious. The thing DSA attacks is the

the CPU cycles consumed by ubiquitous infrastructure operations — data copying, moving, hashing, compression, encryption, serialization — rather than by the application logic that actually serves users. In large-scale services these commodity operations recur constantly (every packet copied, every buffer moved), and profiling shows they consume a large fraction of all CPU cycles — 30-50% for big packet processing. It is called a “tax” because it is overhead levied on every workload: cycles that could have run paying work instead go to plumbing. And it has a second, subtler cost — doing a large memcpy on a core drags the copied data through that core’s caches, evicting the working sets of everything co-located and inflating their latency

. So the tax is paid twice: once in stolen cycles, once in polluted caches. An accelerator that offloads this work could recover both, which is exactly why data-movement engines are appearing on server CPUs — but only if programmers can tell when and how offloading actually pays.

The reason that is hard is that offloading is not free, and its break-even point depends on details the accelerator hides. Every offload pays a

the fixed cost of handing work to the accelerator rather than doing it in-line. To offload a copy, the CPU must prepare a 64-byte work descriptor (operation type, source and destination addresses, completion-record location, cache hints), submit it to the device through a memory-mapped portal, and later check for completion. The paper’s latency breakdown shows descriptor ALLOCATION alone can be about 70% of the offload cost for a 4 KB transfer. This fixed overhead is amortized over the bytes moved, so for a small transfer it dominates and a plain CPU memcpy (which just runs) is faster, while for a large transfer it is negligible and offload wins. The crossover point — and everything about how to lower it — is the central practical question

. Because that overhead can be hidden by overlapping the copy with useful work, or amortized by batching many copies into one submission, or paid down by pre-allocating descriptors, the effective crossover moves dramatically with how you use the device. Documenting exactly how is what turns a powerful but opaque accelerator into one a programmer can wield — which is the contribution of measuring it carefully and distilling the results into rules.

What people did before, and where it broke

Do it on the CPU (software memcpy). Just run the copy in-line on the core. Zero submission overhead, so it wins for small transfers, but it burns the cycles that are the datacenter tax and pollutes the cache — raising a co-located memory-intensive workload’s latency by 43% in the paper’s measurement.
Older DMA engines (CBDMA / Intel I-OAT). Offload copies to a prior-generation on-chip DMA. It helped, but with high offload overhead, few operations, and a requirement to PIN memory (no shared virtual addressing), which made it awkward to use — DSA delivers on average 2.1x its throughput.
Use DSA naively (offload everything synchronously). Point DSA at every copy and wait for each. This loses on small transfers (below the 4 KB synchronous crossover the submission overhead dominates) and underuses the device, because the core sits idle waiting instead of overlapping work.
Guess at the configuration. Pick a queue type, batch size, and engine count by intuition. But the right choices are workload-dependent and interact — shared queues underperform with few threads, tiny batches waste the device, wrong cache hints evict neighbors — so guessing leaves large performance on the table.
The gap. No systematic, quantitative account of when DSA offload pays and how to configure it existed, so its benefits were unpredictable in practice. This paper fills that gap with measurements, six concrete guidelines, and a real-application case study that validates them.

How DSA works, and what the study measured

Submit descriptors, let engines run, and tune the knobs the study characterizes

Prepare a work descriptor

To offload an operation the CPU builds a 64-byte descriptor naming the operation (copy, fill, compare, CRC32, and so on), the source and destination addresses, where to write the completion record, and cache-control hints. Because DSA shares the process's virtual address space (via the on-device address-translation cache and IOMMU), no memory pinning is needed — a major simplification over CBDMA.
↓ →
Enqueue via a shared or dedicated work queue

The descriptor is submitted to a memory-mapped portal. A SHARED work queue (SWQ) accepts lock-free submissions from many clients via the ENQCMD instruction; a DEDICATED work queue (DWQ) serves one client with the lower-latency MOVDIR64B. The study finds DWQs (with batching, or several of them) beat SWQ unless many threads are contending, and about 32 queue entries suffice for near-peak throughput.
↓ →
Engines execute and signal completion

A group arbiter dispatches eligible descriptors to free processing engines (4 per device by default), each of which reads the source, performs the operation, writes the destination, and posts a completion record. The core learns of completion by spin-polling, an interrupt, or the low-power user-space wait UMWAIT — which lets the core sleep while DSA works, so at 4 KB and above most core cycles are spent idle in UMWAIT.
↓ →
Batch to amortize submission new

A single BATCH descriptor references an array of work descriptors and yields one completion record, so the per-submission overhead is spread across many operations. The study shows this gives large, near-exponential throughput gains for small transfers — and finds a sweet spot of 4-8 descriptors for synchronous offload, with gains flattening above 256 KB as the I/O fabric saturates near 30 GB/s.
↓ →
Overlap asynchronously to hide the overhead new

Submitting synchronously (wait for the copy) only pays off above 4 KB; submitting ASYNCHRONOUSLY — issuing the descriptor and continuing to compute, checking completion later — keeps both the core and the device busy and pushes the crossover down to about 256 bytes. This is the single most impactful guideline in the study.
↓ →
Control the cache and memory placement new

A descriptor cache-control hint chooses whether moved data is written into the last-level cache (cache-allocating) or bypasses it to memory; for latency-sensitive co-located apps, bypassing avoids evicting their working sets. Placing source and destination on different NUMA nodes gives marginally more throughput, and for slower CXL memory the study advises using CXL as the source and DRAM as the destination.

Measured on a 4th-gen Xeon (Sapphire Rapids) DSA — 8 work queues and 4 processing engines per device by default — across transfer sizes, batch sizes, queue types, engine and device counts, and cache settings, plus a DPDK Vhost packet-forwarding case study. Peak per-device throughput about 30 GB/s; up to about 90 GB/s across 4 devices before hitting cache-injection limits.

The idea

Know the break-even, and move it. Every offload trades a fixed submission cost for freeing the CPU, so it only wins when the work offloaded is large enough to amortize that cost — and the whole practical art of DSA is knowing where that line is and how to shift it. Measured plainly, synchronous offload beats CPU memcpy only above 4 KB, because descriptor preparation and submission (about 70% of the offload latency) dominate small transfers. But the line is not fixed: batching amortizes the submission across many copies, pre-allocating descriptors removes the biggest fixed cost, and overlapping the copy with computation hides the overhead entirely — which together drop the crossover to about 256 bytes. This is the amortize-and-hide-the-fixed-cost discipline the site meets wherever an operation has a setup price — the graph-mining page packing many operations behind one control block, the pointer-passing page’s minimum transfer size below which the trick does not pay — and here it is quantified precisely: know your transfer size, and choose batching, async, and pre-allocation to push the break-even beneath it.

Overlap the offload instead of waiting on it. The most consequential single finding is that HOW you wait matters more than whether you offload. Synchronous offload makes the core idle while DSA copies, so the copy has to be big to beat just doing it — but asynchronous offload issues the descriptor and lets the core keep computing, checking completion later, so the device and the core work in parallel and the copy is nearly free from the core’s perspective. That is what pushes the crossover from 4 KB down to 256 bytes and lets cores drop into low-power UMWAIT while DSA runs. This is the same decouple-submit-from-wait, hide-latency-with- overlap discipline the site sees in the edge-inference page pipelining IO against compute and the task-parallel page overlapping stages — an offload is only as good as your willingness to not block on it.

The real cost of a CPU memcpy is the cache it wrecks, not the cycles it spends. The study’s most striking result reframes why offloading is worth it. One naturally measures the benefit in CPU cycles saved — but a large software memcpy also streams the copied data through the core’s caches, evicting whatever a co-located latency-sensitive service was keeping there, and the paper measures that as a 43% latency increase for a memory-intensive neighbor. DSA offload sidesteps this almost entirely: its reads do not allocate into the cache and its writes are confined to a small cache-injection region. So even when the raw cycle savings are modest, the pollution avoided can be the dominant win. This is the measure-the-full-cost lesson the site returns to — the replicated- simulation page finding its real limit was cache, the GPU-scheduling page finding its was contention — here revealing that the true price of in-line data movement is the damage it does to everyone else’s cache, a second-order effect a naive cycle count misses entirely.

When does offloading a memcpy actually pay?

From a losing offload to a winning one

Deciding whether and how to offload a copy to DSA. Numbers from the paper's measurements.

Step 1 of 5

The naive question

Should this copy go to DSA or just run as a CPU memcpy? The honest first answer is "it depends" — because offloading is not free. The core must build a 64-byte descriptor, submit it through a memory-mapped portal, and later check for completion, and that fixed cost has to be earned back by the bytes moved.

a plain CPU memcpy has zero submission overhead — it just runs — so it is the baseline to beat
descriptor allocation alone is about 70% of the offload latency for a 4 KB transfer

Why it works

Offloading pays a fixed submission cost (descriptor allocation is about 70% of it) → that cost must be amortized over the bytes moved → synchronous offload only beats CPU memcpy above 4 KB.
Asynchronous submission overlaps the copy with computation → the submission overhead is hidden behind useful work and cores idle in low-power UMWAIT → the crossover drops from 4 KB to about 256 bytes.
A batch descriptor spreads one submission across many operations → per-copy overhead falls sharply for small transfers → a modest 4-8-descriptor batch is the synchronous sweet spot.
A CPU memcpy streams data through the core’s caches → it evicts co-located services’ working sets (raising their latency 43%), while DSA reads do not allocate to cache → offloading protects neighbors’ tail latency, often the dominant benefit.
The right queue type, engine count, and cache hint depend on the workload → the study measures each and gives decision rules → applying them to DPDK yields 1.14-2.29x and a flat forwarding rate.

What it costs

Offload has a hard floor. Below the crossover — 4 KB synchronous, about 256 bytes asynchronous — a plain CPU memcpy is faster, so DSA is not a universal replacement; tiny copies should stay on the core.
Descriptor preparation must be managed. The dominant offload cost is descriptor allocation, so the gains assume descriptors are pre-allocated and reused; a fresh allocation per copy adds significant latency and can erase the benefit.
Bandwidth saturates. A single device tops out near 30 GB/s and multiple devices plateau around 90 GB/s before hitting cache- injection (DDIO) limits, so DSA scales the datacenter tax down but is not an unlimited memory-bandwidth engine.
Shared queues need many threads. SWQ submission carries an ENQCMD round-trip latency that only pays off under high thread concurrency; with few threads, dedicated queues are better, so the right choice is workload-specific.
Enablement is manual and partial. Most applications need case-by-case source changes and recompilation to use DSA (a transparent memcpy-interception library helps but does not cover everything), and some operations (delta records, cache flush) were left unanalyzed.

The evaluation

A 4th-generation Xeon (Sapphire Rapids) DSA characterized across transfer size, batch size, queue type (shared vs dedicated), engine and device counts, completion-signaling method, and cache-control settings, plus a DPDK Vhost packet-forwarding case study applying the derived guidelines.

Moving memory: CPU versus DSA offload

Property	CPU memcpy	DSA synchronous offload	DSA asynchronous + batched
When it wins (transfer size)
CPU cycles during the copy
Cache pollution on neighbors
Throughput ceiling
Programmer effort

The guidelines follow the cost structure: below the crossover keep copies on the CPU; above it, submit asynchronously and batch modestly (4-8 descriptors), pre-allocate descriptors, prefer dedicated queues unless many threads contend, and use cache-control hints to protect co-located services. Applied to DPDK Vhost, these give a 1.14-2.29x forwarding speedup and a flat rate across packet sizes.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

DSA is not a universal memcpy replacement: below the crossover (4 KB synchronous, about 256 bytes asynchronous) a plain CPU copy is faster, and the measured wins assume descriptors are pre-allocated and reused, since a fresh allocation per copy is the dominant cost. Bandwidth saturates — near 30 GB/s per device and about 90 GB/s across four before cache-injection limits — so DSA reduces the datacenter tax rather than removing the memory wall. Shared work queues only pay off under heavy thread concurrency, so the best queue configuration is workload-specific, and enabling DSA generally requires case-by-case source changes and recompilation (a transparent memcpy-interception library helps but is not comprehensive), with some operations left unanalyzed. The results are also specific to Sapphire Rapids DSA and its I/O fabric. None of this dulls the study’s value — it converts an opaque, knob-heavy accelerator into something programmers can actually use, by quantifying the offload break-even and how to move it (batch, overlap, pre-allocate), by revealing that the chief benefit of offloading data movement is avoiding cache pollution rather than saving cycles, and by proving the guidelines on a real application for a 1.14-2.29x gain.

Background

Amortize the fixed cost, overlap the offload, and measure the cost you cannot see

This paper sits in the site’s datacenter-systems and accelerator cluster and rhymes with several ideas covered elsewhere on this site. Its first lesson is a break-even discipline: an offload trades a fixed submission cost for freeing the CPU, so it only pays above a transfer-size threshold, and the practical craft is amortizing and hiding that cost — batching many copies into one submission, pre-allocating descriptors, overlapping the copy with computation — which is the same amortize-a-fixed-cost-over-many-uses move the graph-mining page makes when it packs many cheap units behind one control block, and the same minimum-payload-to-justify-the-trick caveat the pointer-passing page attaches to its handle-passing. Its second lesson is decouple-submit-from-wait: asynchronous offloading overlaps the copy with useful work and drops the crossover from 4 KB to 256 bytes, the same hide-latency-with-overlap discipline the edge-inference page uses to pipeline IO against compute and the task-parallel page uses to overlap stages. Its most striking lesson is a measure-the-full-cost one: the real price of an in-line CPU memcpy is not the cycles it spends but the cache it pollutes for co-located services (a 43% neighbor latency hit), a second-order effect a cycle count misses — the same look-past-the-obvious-metric insight the replicated-simulation page reaches when it finds its true bottleneck is cache and the GPU-scheduling page when it finds its is unisolated contention. And as a genre it is the characterize- then-distill-guidelines contribution the site values in the measure-the-optimum-first work: understand a complex artifact empirically, then hand programmers decision rules — with no single universal setting, only choices matched to the workload, the same no-universal-best conclusion the GNN-scheduling and graph-mining pages reach.

If you remember only three things

1. Datacenter CPUs spend 30-50% of their cycles on data movement (copy, fill, compare, checksum) — a "datacenter tax" that both steals compute from applications and pollutes the shared cache, hurting co-located latency-sensitive services — so Intel's Sapphire Rapids adds an on-chip Data Streaming Accelerator (DSA) to offload it, with far lower overhead than the old CBDMA and no memory pinning; but DSA is knob-heavy (shared vs dedicated queues, batching, engines, cache hints, sync vs async) and using it well was undocumented.
2. This study measures DSA exhaustively and distills six programmer guidelines around its cost structure: offloading pays a fixed submission cost (descriptor allocation is ~70% of it), so synchronous offload only beats CPU memcpy above 4 KB — but overlapping the copy ASYNCHRONOUSLY with computation, batching modestly (4-8 descriptors), and pre-allocating descriptors push the crossover down to about 256 bytes, prefer dedicated queues unless many threads contend, and cache-control hints decide whether moved data lands in or bypasses the last-level cache.
3. The most striking finding is that the chief benefit of offloading data movement is avoiding CACHE POLLUTION rather than saving cycles: an in-line software memcpy raised a memory-intensive neighbor's latency 43%, while DSA reads do not allocate to cache. Applied to DPDK Vhost packet forwarding, the guidelines flatten the forwarding rate across packet sizes for a 1.14-2.29x speedup — with honest limits (a hard offload floor, ~30 GB/s per device and ~90 GB/s across four, manual enablement). The durable lessons: amortize and hide the fixed offload cost, decouple submit from wait, and measure the full cost of in-line data movement, not just its cycles.

Check yourself

Check your understanding

1. The study finds synchronous DSA offload only beats a CPU memcpy above 4 KB, but asynchronous offload wins as low as 256 bytes. Predict why the submission method moves the break-even so dramatically.
2. The paper argues the chief benefit of offloading a memcpy to DSA is not the CPU cycles saved but the cache pollution avoided (a 43% latency hit to a co-located neighbor from software memcpy). Predict why this reframing matters.
3. The DPDK Vhost case study applies the guidelines — asynchronous pipeline, batching matched to 32-packet bursts, dedicated queues, cache hints — plus a reordering array. Predict why the reordering array is necessary specifically because the offload is asynchronous.
4. This paper is a measurement-and-guidelines study rather than a new accelerator design. Predict why exhaustively characterizing DSA and distilling six guidelines is a valuable contribution.

Sources

arXiv:2305.02480 — A Quantitative Analysis and Guidelines of Data Streaming Accelerator in Modern Intel Xeon Scalable Processors — ASPLOS 2024 (full text read via ar5iv).
ACM DL: A Quantitative Analysis and Guidelines of Data Streaming Accelerator (ASPLOS 2024) — published version.