Occamy: Elastically Sharing a SIMD Co-processor across Multiple CPU Cores

Zhongcheng Zhang, Yan Ou, Ying Liu, Chenxi Wang, Yongbin Zhou, Xiaoyu Wang and 7 others

Modern CPUs give each core its own vector unit — hardware that does the same math on many numbers at once — but when one core runs a workload stuck waiting on memory, its vector lanes sit idle while a compute-heavy workload on a neighboring core is starved for exactly that hardware; this work from the Chinese Academy of Sciences and UNSW builds Occamy, a shared vector co-processor that several cores plug into and that hands its lanes elastically to whichever workload can use them right now, together with a compiler that produces vector code able to run at any width, raising vector-hardware utilization and performance at almost no extra chip area.

The argument in one glance

Problem

Per-core vector units sit idle while a neighbor starves

Each CPU core has its own SIMD unit — vector hardware that does the same operation on many numbers at once. When cores co-run different workloads, a memory-bound workload leaves its vector lanes idle while stalled on memory, but a compute-bound workload on another core, hungry for vector hardware, cannot use those idle lanes because each unit is private to its core.
What people did

Give every core a fixed, private SIMD unit

The standard design provisions each core with its own SIMD lanes, sized for the average case, and each workload vectorizes to that fixed width. Some designs allow a scalable vector length, but the unit still belongs to one core.
Why it fell short

Static, private lanes waste the machine's vector capacity

Because the lanes are private and statically assigned, a compute-intensive workload cannot borrow the underutilized lanes allocated to a memory-intensive workload on another core. The total vector capacity of the machine is underused exactly when some workload could exploit it, so overall performance falls short.
This paper's idea

Share one vector co-processor elastically across cores

Occamy is a SIMD co-processor shared by multiple CPU cores, so their co-running workloads spatially share its lanes. It partitions all the lanes across workloads dynamically, based on their phase behavior, so each runs in variable-length SIMD mode — more lanes when it can use them, fewer when it cannot. An Occamy compiler generates vector code that works at any length. This raises SIMD utilization and performance over representative SIMD architectures, at negligible chip-area cost.

Why this problem is worth solving

Vector hardware is powerful but only if it is kept busy, and per-core units often are not.

SIMD stands for Single Instruction, Multiple Data: a vector instruction applies the same operation to many data elements at once, across parallel hardware lanes. This exploits data-level parallelism — the fact that many programs do the same arithmetic to long arrays of numbers — and it is widely built into multi-core CPUs. Each core is given its own SIMD unit with some number of lanes. The efficiency of this hardware depends entirely on how busy the lanes are, and that varies a great deal by workload. A compute-intensive workload, doing heavy arithmetic on data already in the cache, can keep many lanes busy every cycle. A memory-intensive workload is different: it spends much of its time stalled, waiting for data to arrive from memory, and during those stalls its SIMD lanes do nothing. Now consider a realistic machine where different cores co-run different workloads at the same time — a common situation in shared servers. The memory-intensive workload leaves the lanes on its core idle for long stretches. The compute-intensive workload on another core, by contrast, has more vector work than its own lanes can serve, and would gladly use more. But it cannot, because the idle lanes belong to a different core’s private SIMD unit. The machine has vector capacity going to waste on one core while another core is starved for it, which is exactly the inefficiency this work targets.

Fixing it means rethinking where the vector hardware lives and how code uses it.

The obvious wish is to let the compute-intensive workload borrow the memory-intensive workload’s idle lanes. But that is hard for two reasons. First, the hardware: a SIMD unit is physically part of one core, tightly coupled to it, so there is no path for another core to reach its lanes. Sharing requires restructuring where the vector hardware sits, so that several cores can draw on a common pool of lanes rather than each having a private, unreachable unit. Second, the software: traditional vector code is compiled for a fixed vector length — a program is built to use, say, exactly eight lanes, and its instructions assume that width. If the number of lanes a workload is granted can change from moment to moment, code compiled for a fixed width cannot use it. Sharing therefore also requires that a workload be able to run in a variable-length mode, using however many lanes it holds at that instant, and that requires a compiler able to produce vectorized code that is correct and efficient across a range of vector lengths. And because a workload swings between memory-bound and compute-bound phases over its execution, the right allocation of lanes is not fixed but keeps changing, so the sharing must be dynamic. Meeting all of this — a shared pool of lanes, dynamic phase-aware partitioning, and length-agnostic compiled code — without paying a large hardware cost is the challenge Occamy takes on.

What people did before, and where it broke

Private per-core SIMD units (fixed length). Give each core its own vector unit and vectorize code to that fixed width. Where it broke. The lanes are private, so a compute-bound workload cannot use the idle lanes of a memory-bound workload on another core; the machine’s total vector capacity is wasted when co-running mixed workloads.
Scalable-length vector architectures. Support a vector length that can vary, so code need not be tied to one width. Where it broke. This helps a single workload adapt, but the unit is still owned by one core, so it does not let cores share lanes with each other.
Provisioning more lanes per core. Add vector width to each core to serve peaks. Where it broke. It grows chip area for lanes that still sit idle whenever that core’s workload is memory-bound, so it is expensive and does not fix the underutilization.
The gap. No design let several cores elastically share one pool of SIMD lanes, reallocating them to whichever workload can use them as phases change, with compiler support to run each workload at whatever width it holds. Providing that is what Occamy does.

How Occamy works

From private idle lanes to an elastically shared vector pool

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Pool the idle lanes so a hungry workload can borrow them. The root waste is that each core’s SIMD unit is private, so when a memory-bound workload leaves its lanes idle, no one else can use them. Occamy’s first move is to gather the SIMD lanes into a shared co-processor that several cores connect to, turning many private units into one common pool. Then the lanes a stalled workload cannot use become available to a compute-hungry neighbor. Taking capacity that would otherwise sit idle under one tenant and making it available to another that can put it to work is the same harvesting the idle-flash-bandwidth storage work covered elsewhere on this site performs, pooling a resource so that whatever one tenant leaves unused is reclaimed by another rather than wasted.

Allocate the shared lanes to whoever can use them, moment by moment. A pool alone is not enough, because which workload deserves the lanes changes constantly as each swings between memory-bound and compute-bound phases. Occamy tracks that phase behavior and dynamically repartitions the lanes, granting more to a workload in a compute-heavy phase and fewer to one stalled on memory. Continuously steering a shared resource toward the participant that can currently make the most of it, based on live behavior rather than a fixed split, is the same load-aware dynamic scheduling the cooperative-virtualization work covered elsewhere on this site performs, reading each workload’s current state and directing the machine’s resources where they will actually be used.

Make one program run at any width. Elastic partitioning only works if a workload can actually execute on whatever number of lanes it holds at a given instant, but traditional vector code is fixed to one width. Occamy’s compiler produces variable-length vectorized code — code that is correct and efficient whether it is handed a few lanes or many — so a workload adapts as its allocation grows and shrinks. Building one flexible artifact that gracefully handles a varying extent instead of assuming a single fixed size is the same accommodation the variable-sequence training work covered elsewhere on this site makes, handling short and long inputs with right-sized groupings rather than forcing everything into one rigid shape.

Following two co-running workloads through Occamy

Elastically sharing lanes between a memory-bound and a compute-bound workload

A high-level trace of the approach as described in the abstract; the exact hardware and partitioning internals are beyond the abstract this page is based on.

Step 1 of 5

Two cores co-run different workloads

Core A runs a memory-bound workload that is often stalled waiting on memory. Core B runs a compute-bound workload full of vector arithmetic that would use as many lanes as it can get.

This mix of workloads is common on a shared server.

Why it works

Per-core SIMD is underused. Because a memory-bound workload spends much of its time stalled on memory → its private vector lanes sit idle → so the machine’s total vector capacity is wasted even when another core needs it.
Idle capacity can serve a hungry neighbor. Because a compute-bound workload has more vector work than its own lanes can handle → the idle lanes of a memory-bound neighbor could do that work → so pooling the lanes lets the hungry workload borrow them.
Phases make the right split change over time. Because a workload alternates between memory-bound and compute-bound phases → the best allocation of lanes keeps shifting → so the partitioning must be dynamic and phase-aware rather than fixed.
Variable length lets one program use any share. Because the compiler produces vectorized code that runs at any vector length → a workload can execute on however many lanes it holds at that instant → so the elastic partition can change without recompiling.
Sharing raises utilization cheaply. Because the pooled lanes are kept busy by whichever workload can use them → total SIMD utilization rises without adding more lanes → so performance improves over representative SIMD architectures at negligible chip-area cost.

What it costs (trade-offs and scope)

It is a hardware redesign, not a software patch. Occamy moves the SIMD lanes into a shared co-processor and connects multiple cores to it, which changes the processor’s structure — a design choice for a chip maker, not something added to an existing CPU.
Sharing has coordination overhead. Pooling lanes across cores, tracking phases, and repartitioning add machinery and some latency; the abstract reports the chip-area cost is negligible, but coordinating a shared resource is not free.
It needs the special compiler. The benefit depends on the Occamy compiler generating length-agnostic vectorized code; existing binaries compiled for a fixed width do not automatically gain, and code that vectorizes poorly offers less to share.
The gain tracks the imbalance available. The win comes from one workload being memory-bound and idle while another is compute-bound and hungry. When co-running workloads are similar, or all keep their lanes busy, there is little idle capacity to reallocate.
Phase prediction can be imperfect. Elastic partitioning relies on reading phase behavior; if phases shift faster than the system can track, or are misjudged, the allocation can lag the workloads’ real needs.
This page is written from the abstract. The full text was not available when this explainer was prepared, so the microarchitecture, the partitioning algorithm, the compiler internals, and the exact evaluation numbers are not reproduced; the discussion stays at the level the abstract states.

The evaluation

Three ways to provide SIMD to co-running workloads

Property	Private per-core SIMD	Scalable-length SIMD
SIMD unit shared across cores		No, still per core
Reuses a neighbor's idle lanes		No
Variable vector length	No, fixed width
Allocation adapts to workload phase	No	No
Utilization with mixed co-running workloads		Low

Three ways to provide vector (SIMD) hardware to workloads co-running on different cores, as described in the paper's abstract. A private per-core SIMD unit strands its idle lanes when its workload is memory-bound, unreachable by a compute-bound workload on another core. A scalable-length vector architecture lets a single workload vary its width but still keeps the unit private to one core, so it does not enable cross-core sharing. Occamy makes the SIMD lanes a co-processor shared by multiple cores, dynamically partitions those lanes across the co-running workloads based on their phase behavior so each runs at a variable vector length, and provides a compiler that generates length-agnostic vector code. The paper reports that this improves SIMD utilization and, consequently, performance over three representative SIMD architectures, at negligible chip-area cost. Specific numbers and the hardware details are in the full paper, which was not available for this explainer.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is written from the paper’s abstract, because the full text was not available when it was prepared, so it stays at the level the abstract supports and adds only standard, well-known background about SIMD and vectorization. The core idea is clear and appealing. Each CPU core normally has its own vector unit, and when cores co-run a mix of workloads, a memory-bound one leaves its lanes idle while a compute-bound one on another core is starved for exactly that hardware. Occamy fixes this by making the SIMD lanes a co-processor that several cores share, dynamically partitioning the lanes across workloads according to their phase behavior so each runs at whatever vector length it is granted, and adding a compiler that produces length-agnostic vector code so a workload can use any share of lanes. The abstract reports this raises SIMD utilization and performance over three representative SIMD architectures at negligible chip-area cost. Read the specifics with care. The exact hardware design, how phases are detected and lanes are partitioned, how the compiler generates variable-length code, and the quantitative results are in the full paper and not reproduced here. As trade-offs, the approach is a hardware redesign requiring compiler support, its benefit depends on there being idle vector capacity to reallocate (workloads that all keep their lanes busy offer little), and phase-based allocation can lag if behavior shifts quickly. The durable idea is that a specialized functional unit provisioned privately per core is often idle, and pooling it into a shared, elastically-partitioned resource can reclaim that waste.

Background

Why pooling a private resource so often pays off

It helps to see Occamy as one instance of a pattern that recurs across computer systems: a resource dedicated privately to each unit is frequently idle, and pooling it so it can be shared reclaims that idleness. Here the resource is vector lanes; each core has its own, and they go unused whenever that core’s workload stalls on memory. The manycore work-stealing work covered elsewhere on this site makes the same argument for cores themselves, letting idle cores pull work from busy ones so no part of the chip sits unused while another is overloaded — dynamic sharing beating static, private assignment. The common thread is that static, per-unit provisioning has to be sized for each unit’s own peak, which guarantees waste whenever units are imbalanced, whereas a shared pool can shift capacity to wherever it is needed at the moment. The general lesson is that when a valuable resource is replicated privately across many units whose demand for it fluctuates and differs, there is usually efficiency to be won by pooling it and allocating it dynamically to whoever can use it — provided the coordination to do so, and any software changes it requires, can be made cheap enough that the reclaimed utilization is worth it.

If you remember only three things

1. Modern multi-core CPUs give each core its own SIMD (Single Instruction, Multiple Data) vector unit to exploit data-level parallelism, but when cores co-run different workloads the hardware is used poorly: a memory-bound workload stalls on memory and leaves its vector lanes idle, while a compute-bound workload on another core has more vector work than its own lanes can serve and cannot reach the idle ones, because each unit is private to its core. The machine's total vector capacity is wasted precisely when some workload is hungry for it.
2. Occamy makes the SIMD lanes a co-processor shared by multiple CPU cores, so their co-running workloads spatially share the lanes. It enables elastic spatial sharing by dynamically partitioning all the lanes across workloads based on their phase behavior, so each executes in variable-length SIMD mode — granted more lanes in a compute-heavy phase and fewer when memory-bound. An accompanying Occamy compiler supports this by analyzing phase behavior and generating vectorized code that runs correctly and efficiently across varying vector lengths, so a workload can use whatever share of lanes it currently holds.
3. The paper reports that Occamy improves SIMD utilization and, consequently, performance over three representative SIMD architectures, at negligible chip-area cost. Its scope: it is a hardware redesign that pools the lanes and needs the special compiler to produce length-agnostic code; its benefit tracks how much idle vector capacity there is to reallocate (workloads that all keep their lanes busy gain little); and phase-based allocation can lag if behavior shifts quickly. Note: this explainer is written from the abstract, as the full text was not available, so the hardware design, partitioning algorithm, compiler internals, and exact numbers are summarized only at the level the abstract states.

Check yourself

Check your understanding

1. Why are per-core SIMD units often underused when different workloads co-run on different cores?
2. What is Occamy's central hardware idea?
3. Why does Occamy need a special compiler that produces "variable-length" vector code?
4. A server co-runs several workloads that are all compute-bound and keep their SIMD lanes fully busy. What does Occamy's reasoning imply?

Sources

The paper is “Occamy: Elastically Sharing a SIMD Co-processor across Multiple CPU Cores” by Zhongcheng Zhang, Yan Ou, Ying Liu, Chenxi Wang, Yongbin Zhou, Xiaoyu Wang, Yuyang Zhang, Yucheng Ouyang, Jiahao Shan, Ying Wang, Jingling Xue, Huimin Cui, and Xiaobing Feng, published at ASPLOS 2023, DOI 10.1145/3582016.3582046. The authors are affiliated with institutions including the Institute of Computing Technology of the Chinese Academy of Sciences and the University of New South Wales.

This explainer is written from the paper’s abstract, because the full text was not available at the time of writing: although the paper is open access (gold), the publisher’s copy was behind automated-access protection, no preprint was available, and no separately hosted author copy could be retrieved. The problem statement (SIMD extensions exploiting data-level parallelism in multi-core processors, and the inefficiency that a compute-intensive workload cannot use the underutilized SIMD lanes allocated to a memory-intensive workload co-running on another core) and the description of the approach (Occamy, a SIMD co-processor shared by multiple CPU cores enabling elastic spatial sharing by dynamically partitioning all the SIMD lanes across workloads based on their phase behaviors so each executes in variable-length SIMD mode, together with an Occamy compiler that supports variable-length vectorization by analyzing phase behaviors and generating vectorized code that works across varying vector lengths) come directly from the abstract, as do the qualitative results (improved SIMD utilization and performance over three representative SIMD architectures at negligible chip-area cost). The abstract does not report specific quantitative figures, and none are invented here. The surrounding explanation of SIMD, vector lanes, fixed versus scalable vector length, compute-bound versus memory-bound behavior, and program phases is standard, widely-known background provided to make the abstract’s claims understandable, and is not specific to this paper. The hardware microarchitecture, the phase-detection and lane-partitioning mechanisms, the compiler internals, and the detailed evaluation are not reproduced here because they were not available.