8.9Discussion and the Future of Directory Protocols

book pp. 189–190 · ~2 min read

  • directory dominance
  • scalable coherence
  • partitioned coherence

Directory protocols have come to dominate the market — and not only where you’d expect.

Why they won at small scale

Even in systems snooping could serve, directories are now more common — because they permit point-to-point links in the interconnect. No totally ordered broadcast medium, no bus electrical limits. Coherent HyperTransport and QPI exist for exactly this reason.

Why they won at large scale

For hundreds or thousands of nodes, directories are the only viable option. The numerous snooping optimizations of §7.6 can mitigate snooping’s bottlenecks — fundamentally, none can eliminate them.

Given that scalability, the authors expect directory protocols to keep their crown for the foreseeable future.

The heretical footnote

One hedge: future highly scalable systems may simply not be coherent — or not coherent everywhere. Two sketches:

  • Partitioned coherence: coherent subsystems, with no coherence maintained across subsystems.
  • The supercomputer precedent: Cray machines have shipped either with no coherence at all (T3E) or with coherence that restricts which data may be cached (X1).

Chapter 10’s heterogeneous systems — where GPUs and accelerators bring their own ideas about coherence — return to exactly this tension.

Check yourself

1.Why do directory protocols dominate even SMALL-scale systems, where snooping's bottlenecks barely bite?

2.For systems scaling to hundreds or thousands of nodes, what does the book say about the snooping-vs-directory choice?

3.What alternative future does the book sketch for HIGHLY scalable systems?

3 questions
Chapter 8 references
  1. D. Abts, S. Scott, and D. J. Lilja. So many states, so little time: Verifying memory coherence in the Cray X1. IPDPS, 2003.
  2. A. Agarwal, R. Simoni, M. Horowitz, and J. Hennessy. An evaluation of directory schemes for cache coherence. ISCA, 1988.
  3. J. K. Archibald and J.-L. Baer. An economical solution to the cache coherence problem. ISCA, 1984.
  4. J.-L. Baer and W.-H. Wang. On the inclusion properties for multi-level cache hierarchies. ISCA, 1988.
  5. P. Conway and B. Hughes. The AMD Opteron northbridge architecture. IEEE Micro, 27(2), 2007.
  6. P. Conway, N. Kalyanasundharam, G. Donley, K. Lepak, and B. Hughes. Cache hierarchy and memory subsystem of the AMD Opteron processor. IEEE Micro, 30(2), 2010.
  7. A. Gupta, W.-D. Weber, and T. Mowry. Reducing memory and traffic requirements for scalable directory-based cache coherence schemes. ICPP, 1990.
  8. M. D. Hill, J. R. Larus, S. K. Reinhardt, and D. A. Wood. Cooperative shared memory: Software and hardware for scalable multiprocessors. ACM TOCS, 11(4), 1993.
  9. Intel Corporation. An Introduction to the Intel QuickPath Interconnect, 2009.
  10. J. Laudon and D. Lenoski. The SGI Origin: A ccNUMA highly scalable server. ISCA, 1997.
  11. D. Lenoski et al. The Stanford DASH multiprocessor. IEEE Computer, 25(3), 1992.
  12. R. A. Maddox, G. Singh, and R. J. Safranek. Weaving High Performance Multiprocessor Fabric. Intel Press, 2009.
  13. M. R. Marty and M. D. Hill. Virtual hierarchies to support server consolidation. ISCA, 2007.
  14. S. L. Scott. Synchronization and communication in the Cray T3E multiprocessor. ASPLOS, 1996.