10.2More Heterogeneity Than Just GPUs

§10.1 solved consistency and coherence inside one accelerator. Now the harder composition problem: expose one global shared memory across multiple multicore devices — CPUs, GPUs, other accelerators — when each device arrives with its own consistency model, enforced by its own coherence protocol. A CPU might guarantee TSO via a consistency-agnostic, SWMR-enforcing directory protocol; a GPU might guarantee scoped RC via a lazy consistency-directed protocol. Bolt them together and two questions follow, both under active research: what does the combined machine promise (§10.2.1), and how do the two protocols physically interoperate (§10.2.2)?

10.2.1 Heterogeneous consistency models

Suppose multicore A enforces SC and multicore B enforces TSO, sharing memory. What is the resulting model? The tempting shortcut — “declare the weaker model the overall model” — fails twice over: if the two models are incomparable there is no weaker one, and even when one subsumes the other, discarding the stronger device’s guarantees is wasteful.

The right semantics is intuitive (though, the book notes, not yet formalized): a compound consistency model compound consistency model The semantics of a heterogeneous shared-memory machine: memory operations from each component device satisfy that device's ordering rules (not simply “the weaker model”). defined in Chapter 10 — open in glossary , in which memory operations from each component satisfy the memory ordering rules of that component. Operationally, picture §3.6’s switch, heterogeneous edition (book Fig 10.2):

Figure 10.2 (recreated): the SC cores’ loads and stores go straight to the switch, in program order; the TSO cores’ stores drain through FIFO store buffers, and their loads may bypass (§4.4’s machinery). Each thread obeys its own core’s rules.

Dekker’s algorithm shows the compound model biting (book Table 10.20 — T1 on an SC core, T2 on a TSO core):

The fix respects the asymmetry (book Table 10.21): a FENCE fence Instruction forcing all program-order-earlier memory operations into memory order before all later ones (a.k.a. memory barrier). defined in Chapter 4 — open in glossary between St2 and Ld2 on the TSO side only restores St2 → Ld2 and forbids (0,0). T1 needs nothing — SC already orders St1 → Ld1. One device’s threads follow one rulebook, the other’s another, and the FENCEs go exactly where a rulebook is weak.

Programming with heterogeneous consistency models

Two details about HRF are worth keeping. First, its relationship to §10.1.4’s scoped RC model is exactly the DRF-to-RC relationship one level up: HRF is a language-level contract with no semantics for racy programs, while scoped RC is a virtual-ISA-level model that gives even racy code defined behavior. Second, HRF comes in two strengths: HRF-direct, where releases and acquires synchronize only if their scopes match exactly, and HRF-indirect, which adds transitive synchronization across different scopes — if T1 syncs with T2 at CTA scope and T2 syncs with T3 at GPU scope, T1 has synchronized with T3. HRF-direct is simpler for hardware and painfully restrictive for programmers; scoped RC (like NVIDIA’s PTX model) follows HRF-indirect.

10.2.2 Heterogeneous coherence protocols

Now the plumbing. Each device has a local coherence protocol keeping its own L1s coherent. Stitching them into one machine means the composed system must satisfy the compound model — every device’s operations obeying that device’s rules — while the protocols speak to each other through something both can understand (book Fig 10.3):

Figure 10.3 (recreated): hierarchical heterogeneous coherence. The approach extends §9.1.6’s hierarchical coherence hierarchical coherence protocol Separate intra-chip and inter-chip coherence protocols composed hierarchically: requests satisfiable on-chip never touch the outer protocol; each chip appears as one node (with coarse state) to the outer directory. Lets commodity chips use simple non-scalable designs. Examples: Sun Wildfire, AlphaServer GS320, Stanford DASH; Marty/Hill virtual hierarchies do it dynamically on flat multicores. defined in Chapter 9 — open in glossary from homogeneous to heterogeneous composition — the amber shims are the new ingredient.

The recipe: a local controller tries to satisfy each request inside its device; anything it can’t complete locally (say, a GetM while the other device has sharers) is forwarded to the global coherence controller, which forwards it to the other device’s local controller, collects the response, and routes it back. Making that work takes two things:

1. a global coherence interface rich enough to carry every kind of request the devices’ local protocols may need to exchange; and
2. per-device shims shim Per-device translator between a local coherence protocol and the global coherence controller's interface (e.g., a snooping device's shim interprets a global Inv as a GetM). defined in Chapter 10 — open in glossary — translators between each local protocol and the global interface, choosing which global requests to emit and how to interpret the ones that arrive.

What must that interface look like? Walk through the three possible pairings.

Scenario 1: consistency-agnostic + consistency-agnostic

A directory-protocol CPU joined to a snooping-bus accelerator — both SWMR-enforcing. The global directory keeps a coarse per-device sharer list, and the interface only needs the classic Table 6.4 request vocabulary. The interesting move is the shim’s: a global Inv arriving at a snooping device becomes a local GetM — because on a bus, that is what invalidates sharers.

Scenario 2: consistency-directed + consistency-directed

An LRCC GPU joined to an RCC accelerator through a shared LLC. Here the global LLC is the point of coherence: shims forward GetV getv Read request in which the REQUESTOR takes responsibility for invalidating the block (via self-invalidation), so the directory need not track it — the key interface addition (vs. GetS) for stitching consistency-directed protocols (cf. Spandex). defined in Chapter 10 — open in glossary requests up and write-backs through, and the LRCC ownership machinery works across the device boundary unchanged — the deferred flush just gets triggered by another device’s acquire.

Scenario 3: consistency-agnostic + consistency-directed

The instructive one — a CPU (SC, MSI directory) and a GPU (non-scoped RC, LRCC) joined by a global directory embedded in the shared LLC. First, CPU as producer (book Table 10.22): St1: St data1 = NEW then St2: St flag = SET on the CPU; Acq Ld1 spinning on flag, then Ld2 r2 = data1 on the GPU, with everything initially cached in both devices.

The CPU’s stores upgrade as usual — but watch what the global directory doesn’t do:

Now the roles reverse (book Table 10.23): the GPU produces (St1, then Rel St2), the CPU consumes (Ld1 spin, Ld2). The GPU’s release needs ownership of flag — and this time the CPU’s copies must die, because nothing in a consistency-agnostic CPU ever self-invalidates:

Mitigating the bandwidth bill

For multi-chip CPU-GPU systems the global directory sits across an off-chip coherent interconnect — and GPU workloads, with their high miss rates, hammer it. Coarse-grained coherence tracking coarse-grained coherence tracking Tracking coherence state at region/page granularity so accesses to private or read-only regions skip the global directory; relieves the off-chip interconnect in multi-chip CPU-GPU systems. defined in Chapter 10 — open in glossary relieves the pressure: track coherence state at, say, page granularity inside each device; when a GPU miss touches a page known to be GPU-private or read-only, skip the global directory entirely and fetch straight from memory over a high-bandwidth bus.

A low-complexity alternative: selective GPU caching

Hierarchical coherence needs shims in both devices — awkward when CPU and GPU come from different vendors. The minimal-invasion option: selective GPU caching selective gpu caching Low-complexity CPU-GPU coherence: the GPU never caches data mapped to CPU memory nor GPU-memory data currently cached by the CPU (enforced via a coarse-grained remote directory). Trivially coherent; request coalescing and CPU-side caching offset the penalty. defined in Chapter 10 — open in glossary . The GPU simply never caches (1) any data mapped to CPU memory, nor (2) any GPU-memory data currently cached by the CPU — the latter tracked by a coarse-grained remote directory in the GPU (a region enters it when the CPU touches a GPU-memory block; if the GPU had the line cached, it flushes it). Coherence holds trivially: no shared cacheable data, no coherence problem. The naive cost — every CPU-cached location must be fetched from the CPU — is offset by GPU request coalescing (merge concurrent requests to the same line) and CPU-side caching of GPU remote requests.