Load and MLP-Aware Thread Orchestration for Recommendation Systems Inference on CPUs

Rishabh Jain, Teyuh Chou, Onur Kayiran, John Kalamatianos, Gabriel H. Loh, Mahmut T. Kandemir and 1 other

Recommendation models decide what you see across the internet and are among the most-run programs in datacenters, and while they are trained on GPUs, companies increasingly run the cheaper inference step on ordinary server CPUs — but a big part of that work is looking up rows in enormous tables, which strains the memory system, and existing schedulers leave the many-chiplet CPU unbalanced, over-assign cores, and waste memory bandwidth; this work from Penn State and AMD builds Balance, a thread scheduler that gives each lookup table just enough cores, schedules by how each table uses memory, and steals and reorders work to keep every chiplet busy, running the same models up to 1.67 times faster.

The argument in one glance

Problem

Serving recommendation models on CPUs is memory-bound and uneven

Recommendation models are among the most-run machine-learning models in datacenters, and running their inference on cheap server CPUs matters. But much of the work is looking up rows in enormous embedding tables in main memory, which strains the memory system, and a modern server CPU is built from several chiplets that are hard to keep evenly busy.
What people did

Parallelize the lookups across the CPU cores

Existing techniques split the embedding-table lookups and the neural-network work across the many cores of the server CPU, aiming to use all of them at once.
Why it fell short

They imbalance chiplets, misallocate cores, and waste bandwidth

These techniques fall short three ways. They spread work unevenly across the CPU's chiplets, so some finish while others lag. They allocate cores to embedding tables poorly, wasting cores. And they use the machine's memory-level parallelism — its ability to keep many memory requests in flight — inefficiently, so the memory bandwidth that limits this workload sits underused.
This paper's idea

Schedule with load and memory-parallelism awareness (Balance)

Balance is a thread scheduler with three moves: give each embedding table the fewest cores that keep it busy, so more tables run at once; schedule tasks according to how much memory parallelism each table needs, so bandwidth stays saturated; and combine work stealing with table reordering to even out load across chiplets. On real hardware with production traces and 96 cores, Balance runs up to 1.67 times faster than prior state-of-the-art techniques.

Why this problem is worth solving

The workload is enormous in volume, and its cost is dominated by memory, not math.

Recommendation models decide which items to show a user — products, videos, posts — and they are among the most frequently executed machine-learning models in datacenters, because a recommendation is computed for essentially every user interaction. The deep learning recommendation model, or DLRM, is a widely-used design. It has two kinds of input. Dense features (numeric values) go through a small neural network. Sparse features (categorical values, like which user or which item) are handled by embedding tables: for each category, the model stores a learned vector, and inference looks up the vectors for the categories present in a request. These embedding tables are huge — collectively tens or hundreds of gigabytes — and the lookups gather many scattered rows from them. The looked-up vectors and the dense output are then combined and passed through more neural-network layers to produce a score. Such models are trained on GPUs, where the training compute is heavy. But for inference — serving predictions live — CPUs are often the cost-effective choice, because inference is dominated by the memory-heavy embedding lookups rather than by dense arithmetic, and because CPUs are already plentiful in datacenters. Since recommendation inference runs at massive scale, even modest per-query speedups translate into large savings, so optimizing DLRM inference on CPUs is a problem with real business value.

The reason it is hard is that the bottleneck is memory bandwidth on a fragmented CPU.

The embedding lookups make DLRM inference a memory-bound problem: the model reads many rows scattered across large tables in main memory, so performance is limited by how fast and how many memory requests the machine can service, not by the cores’ arithmetic. Two hardware realities make this delicate. First, memory-level parallelism, or MLP: a core hides the long latency of a memory access by issuing many requests and keeping them in flight at once, and a workload only reaches full memory bandwidth if enough requests are outstanding — but issue too many from too many cores and they contend, wasting bandwidth. So how tasks are scheduled directly determines how well the memory system is used. Second, modern server CPUs are not monolithic; they are built from several chiplets, each a cluster of cores with its own path to memory. Work must be spread evenly across the chiplets, because if one chiplet is handed more work than another, the busy chiplet sets the pace while the idle one wastes its cores and its share of memory bandwidth. Embedding tables differ widely in size and access pattern, so a naive split leaves some cores over-assigned, some chiplets overloaded, and the memory system underfed. Existing parallelization schemes, the paper argues, stumble on exactly these points: load imbalance across chiplets, poor core allocation per table, and inefficient use of memory-level parallelism.

What people did before, and where it broke

Split embedding lookups evenly across all cores. Divide the tables’ lookups across the CPU’s cores to use every core. Where it broke. A memory-bound table cannot profitably use many cores, so a generous split over-assigns cores that then sit idle waiting on memory, wasting them.
Parallelize without regard to memory-level parallelism. Schedule tasks to keep cores busy without modeling how many memory requests they put in flight. Where it broke. Too few outstanding requests leaves bandwidth idle; too many contend for it; either way the memory system, which limits this workload, is used inefficiently.
Statically partition work across the CPU. Assign fixed shares of the work to parts of the CPU up front. Where it broke. On a multi-chiplet CPU with tables of very different sizes, a static split leaves some chiplets overloaded and others idle, so the slowest chiplet sets the pace.
The gap. No scheduler jointly fixed core over-allocation, memory-parallelism usage, and chiplet load imbalance for DLRM inference. Doing all three together — minimal core allocation per table, memory-parallelism-aware scheduling, and work stealing plus table reordering — is what Balance does.

How Balance works

From a naive parallel split to a load- and memory-aware schedule

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Give each table only the cores it can actually use. Because embedding lookups are limited by memory, not compute, piling more cores onto one memory-bound table does not speed it up — the extra cores stall waiting on memory while other tables wait for cores. Balance instead allocates the fewest cores per table that keep it busy, so the freed cores serve other tables and overall utilization rises. Matching the amount of resource handed to a piece of work to how much that piece can actually exploit, rather than splitting evenly, is the same instinct the sequence-parallelism work covered elsewhere on this site follows, giving short sequences small groups and long ones big groups so each gets the resources it needs and no more.

Schedule to keep the memory system saturated, not flooded. The machine reaches full memory bandwidth only when the right number of memory requests are in flight, and each embedding table generates a different amount of that memory-level parallelism. Balance reads each table’s characteristics and schedules tasks so the memory channels stay busy without so many concurrent requests that they contend. Driving a scarce resource to full utilization by scheduling around how much of it each task demands is the same goal the idle-flash-harvesting work covered elsewhere on this site pursues, arranging accesses so the memory or storage bandwidth that would otherwise sit idle is put fully to work.

Rebalance dynamically instead of trusting a static split. Tables differ in size and cost, and a modern CPU is several chiplets, so any fixed division of work will leave some chiplets idle while others lag. Balance reorders the tables and lets idle chiplets steal work from busy ones, so the load evens out as the computation runs rather than being locked in ahead of time. Letting idle workers pull work from busy ones to smooth out uneven load is the same mechanism the manycore work-stealing work covered elsewhere on this site is built on, having threads that run dry steal tasks so no part of the chip stays idle while another is overloaded.

Following a batch of lookups through Balance

Scheduling embedding lookups on a many-chiplet CPU

A high-level trace of the approach as described in the abstract; internal algorithm details are beyond the abstract this page is based on.

Step 1 of 5

A batch of requests needs many lookups

A batch of recommendation requests arrives, each needing lookups of scattered rows from several large embedding tables that live in main memory. This is the memory-heavy heart of the work.

The tables differ widely in size and in how their rows are accessed.

Why it works

Embedding lookups are memory-bound. Because DLRM inference gathers scattered rows from huge tables in main memory → the work is limited by memory bandwidth and latency, not arithmetic → so the scheduler must be built around the memory system, not the cores alone.
Over-allocating cores to a table wastes them. Because a memory-bound table cannot use more cores than its bandwidth allows → giving it extra cores leaves them stalling on memory while other tables wait → so allocating the fewest cores per table lets more tables run at once and raises utilization.
Memory parallelism must be fed, not flooded. Because a core hides memory latency by keeping many requests in flight, yet too many concurrent requests contend for bandwidth → scheduling tasks by each table’s memory-parallelism profile → so the memory channels stay saturated without thrashing.
Chiplets go idle without dynamic balancing. Because a server CPU is many chiplets and a static split lets some finish before others → letting idle chiplets steal work and reordering the tables → so all chiplets stay busy and the slowest one no longer sets the pace.
The gains compound on real workloads. Because the three fixes target distinct losses — wasted cores, underused bandwidth, and imbalance → together they recover more than any one alone → so on production traces Balance reaches up to 1.67 times and a steady 1.22 times across batch sizes.

What it costs (trade-offs and scope)

It targets DLRM inference on CPUs specifically. Balance is designed for the memory-bound embedding-lookup pattern of recommendation models on multi-chiplet server CPUs. Workloads dominated by dense compute, or run on GPUs, are outside its focus.
Scheduling adds its own overhead. Deciding core allocations, profiling memory-level parallelism, and running work stealing and table reordering all cost some cycles; the design bets these are far outweighed by the utilization they recover, but they are not free.
It depends on knowing table characteristics. The memory-parallelism-aware scheduling relies on understanding each embedding table’s access behavior; tables whose behavior is hard to characterize, or shifts at run time, may be scheduled less well.
The benefit tracks the imbalance available to fix. The gains come from removing wasted cores, underused bandwidth, and chiplet imbalance. A configuration that was already well balanced, or a CPU with few chiplets, leaves less for Balance to improve.
Results are on particular hardware and traces. The reported up to 1.67 times and 1.22 times figures come from real hardware with production DLRM traces at 96 cores; different CPUs, memory systems, or models would shift the numbers.
This page is written from the abstract. The full text was not available when this explainer was prepared, so the internal scheduling algorithms and the detailed evaluation are summarized only at the level the abstract states, without invented specifics.

The evaluation

Three ways to parallelize DLRM inference on a server CPU

Property	Even core split	Prior state-of-the-art
Cores per embedding table		Suboptimal
Uses memory-level parallelism	Ignored
Load across chiplets	Imbalanced
Speedup on 96 cores	Slower	Baseline
Across batch sizes	Varies	Baseline

Three ways to parallelize deep learning recommendation model inference on a modern multi-chiplet server CPU, as described in the paper's abstract. Splitting embedding lookups evenly across cores over-assigns cores to memory-bound tables and ignores how the memory system is used. Prior state-of-the-art techniques still allocate cores to tables poorly, use memory-level parallelism inefficiently, and leave load imbalanced across the CPU's chiplets. Balance instead gives each table the fewest cores it needs, schedules tasks according to each table's memory-parallelism profile so bandwidth stays saturated, and combines work stealing with table reordering to even out chiplet load. On real hardware with production DLRM traces and 96 cores, Balance reports up to a 1.67 times speedup over prior techniques, and a consistent about 1.22 times gain across a range of batch sizes. These figures are from the abstract; the full paper was not available for this explainer.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is written from the paper’s abstract, because the full text was not available when it was prepared, so it stays at the level the abstract supports and fills in only standard, well-known background about recommendation models and CPU memory systems. The core of the contribution is clear and sensible. Recommendation-model inference on CPUs is dominated by memory-heavy embedding-table lookups, and on a modern server CPU made of several chiplets, existing schedulers waste cores on memory-bound tables, use the machine’s memory-level parallelism poorly, and leave the chiplets unevenly loaded. Balance addresses all three at once: it hands each table the fewest cores it can actually use, schedules tasks according to how much memory parallelism each table needs so bandwidth stays saturated, and uses work stealing and table reordering to rebalance load across chiplets as the computation runs. The reported gains — up to 1.67 times over prior state-of-the-art on a 96-core machine, and about 1.22 times consistently across batch sizes, on real hardware with production traces — are meaningful for a workload that runs at enormous scale. Read the specifics with care: the internal scheduling algorithms, how core counts and memory-parallelism needs are decided, and the full evaluation are in the paper and not reproduced here. The durable idea is that on memory-bound workloads the scheduler’s job is to match cores and concurrency to the memory system and to keep every part of a fragmented CPU busy, not simply to spread work evenly.

Background

Why the scheduler must match the hardware's shape

It helps to see Balance beside a broader theme: getting performance out of a modern CPU increasingly means scheduling with the machine’s physical structure in mind, not treating it as a uniform pool of cores. A server processor is a collection of chiplets with non-uniform paths to memory, and a memory-bound workload lives or dies by how well the scheduler feeds those memory paths. The cooperative-scheduling work covered elsewhere on this site makes the same argument in the virtualization setting, showing that a scheduler blind to the CPU’s cache-cluster and load topology makes poor decisions, and that exposing and respecting that structure unlocks large gains. Balance carries the lesson into recommendation inference: the right number of cores per table, the right amount of memory-level parallelism, and even load across chiplets all depend on the shape of the hardware and the shape of the workload. The general point is that as CPUs grow more internally heterogeneous — chiplets, multiple memory channels, non-uniform latencies — the software that schedules work onto them has to become topology-aware and workload-aware to keep the machine full. A scheduler that ignores the structure leaves performance stranded exactly where the hardware has the most to give.

If you remember only three things

1. Recommendation models like the deep learning recommendation model (DLRM) are among the most-run machine-learning models in datacenters, and although trained on GPUs, their inference is often served on cheaper server CPUs. The bottleneck is memory, not math: inference looks up scattered rows in enormous embedding tables in main memory, so it is limited by memory bandwidth. On a modern CPU built from several chiplets, existing parallelization wastes cores on memory-bound tables, uses memory-level parallelism (keeping many memory requests in flight) inefficiently, and leaves load unevenly spread across chiplets.
2. Balance is a thread scheduler that fixes all three at once. It allocates the fewest cores per embedding table that keep it busy, so more tables run in parallel and no cores idle waiting on memory. It schedules tasks according to each table's memory-parallelism profile, so enough requests stay in flight to saturate memory bandwidth without contending. And it combines work stealing — idle chiplets take tasks from busy ones — with table reordering to even out load across the CPU's chiplets as the computation runs, rather than committing to a static split.
3. On real hardware with production DLRM traces and 96 cores, Balance reports up to a 1.67 times speedup over prior state-of-the-art parallelization techniques, and a consistent about 1.22 times gain across a range of batch sizes. Its scope: it targets the memory-bound embedding-lookup pattern of recommendation inference on multi-chiplet CPUs, the benefit tracks how much imbalance and waste there is to remove, and the scheduling itself adds some overhead. Note: this explainer is written from the paper's abstract, as the full text was not available, so internal algorithm and evaluation details are summarized only at the level the abstract states.

Check yourself

Check your understanding

1. Why is deep learning recommendation model (DLRM) inference on a CPU limited by memory rather than by the cores' arithmetic?
2. Why does giving a memory-bound embedding table more cores not make it faster?
3. What is "memory-level parallelism" (MLP), and why does Balance schedule with it in mind?
4. A datacenter runs Balance on a workload that is already perfectly balanced across chiplets, uses full memory bandwidth, and assigns cores well. What does the paper's reasoning predict?

Sources

The paper is “Load and MLP-Aware Thread Orchestration for Recommendation Systems Inference on CPUs” by Rishabh Jain, Teyuh Chou, Onur Kayiran, John Kalamatianos, Gabriel H. Loh, Mahmut T. Kandemir, and Chita R. Das, published at ASPLOS 2025, DOI 10.1145/3676641.3716003. The authors are affiliated with Pennsylvania State University and AMD Research. The proposed thread scheduler is named Balance.

This explainer is written from the paper’s abstract, because the full text was not available at the time of writing: the publisher’s version was behind automated-access protection, the paper’s open-access status was closed, and no preprint or author copy was found. The problem statement (deep learning recommendation model inference being memory-bound and dominated by embedding-table lookups, CPUs being cost-effective for inference, and the three shortcomings of existing parallelization: load imbalance across CPU chiplets, suboptimal core allocation for embedding tables, and inefficient use of memory-level parallelism) and the description of the Balance scheduler (minimizing core allocation per embedding table to maximize core utilization, memory-parallelism-aware task scheduling based on table characteristics, and combining work stealing with table reordering to reduce chiplet load imbalance) come directly from the abstract, as do the results (evaluation on real hardware with production DLRM traces, up to a 1.67 times speedup over prior state-of-the-art parallelization techniques on 96 cores, and a consistent about 1.22 times gain across batch sizes). The surrounding explanation of what a deep learning recommendation model is, how embedding lookups stress memory, what memory-level parallelism means, and why multi-chiplet CPUs need balanced scheduling is standard, widely-known background provided to make the abstract’s claims understandable, and is not specific to this paper. The internal scheduling algorithms and the detailed evaluation are not reproduced here because they were not available.