A Dialogue on Distribution

Part III 💬 Dialogue OSTEP pp. 603–604 · ~5 min read

The professor reaches for the peach one last time, cannot make it carry the subject, and abandons it a line after the student agrees to go along with it.

Why this matters

Watch what replaces it. Having asked distance, quantity and rot of one fruit and been told it is not working, the professor says “forget about the peaches” and lists three plain failures instead — lost messages, dead machines, corrupt disks. That list is the syllabus, and the chapters ahead take the first two in order.

The final piece opens as the others did — with a peach, and a professor about to regret trusting one.

Professor:And thus we reach our final little piece in the world of operating systems: distributed systems. Since we can’t cover much here, we’ll sneak in a little intro here in the section on persistence, and focus mostly on distributed file systems. Hope that is OK!
Student:Sounds OK. But what is a distributed system exactly, oh glorious and all-knowing professor?
Professor:Well, I bet you know how this is going to go…
Student:There’s a peach?
Professor:Exactly! But this time, it’s far away from you, and may take some time to get the peach. And there are a lot of them! Even worse, sometimes a peach becomes rotten. But you want to make sure that when anybody bites into a peach, they will get a mouthful of deliciousness.
Student:This peach analogy is working less and less for me.
Professor:Come on! It’s the last one, just go with it.
Student:Fine.
Professor:So anyhow, forget about the peaches. Building distributed systems is hard, because things fail all the time. Messages get lost, machines go down, disks corrupt data. It’s like the whole world is working against you!

Three ways the world works against you — and only the third one we’ve already met (silent corruption, back in data integrity):

messages get lostM1M2✗the reply never arrivesmachines go downM3✗the whole node is gonedisks corrupt dataM4bad blockup, but its data is wrong
The perils of distribution: a lost message (the network drops it), a downed machine (the whole node vanishes), and a corrupt disk (it’s up, but its bits are wrong).
Student:But I use distributed systems all the time, right?
Professor:Yes! You do. And… ?
Student:Well, it seems like they mostly work. After all, when I send a search request to Google, it usually comes back in a snap, with some great results! Same thing when I use Facebook, Amazon, and so forth.
Professor:Yes, it is amazing. And that’s despite all of those failures taking place! Those companies build a huge amount of machinery into their systems so as to ensure that even though some machines have failed, the entire system stays up and running. They use a lot of techniques to do this: replication, retry, and various other tricks people have developed over time to detect and recover from failures.
Student:Sounds interesting. Time to learn something for real?
Professor:It does seem so. Let’s get to work! But first things first… (bites into peach he has been holding, which unfortunately is rotten)

Next: Distributed Systems

Now the concrete machinery. First, communication over an unreliable network — why packets vanish, and how to build a reliable layer on top of an unreliable one — then remote procedure call (RPC) to make a call to another machine look almost like a local one. From there, two real distributed file systems: Sun’s NFS and Andrew’s AFS.

If you remember only three things

  • The piece admits to being a fragment before it starts. It is sneaked into the persistence section, aimed mostly at distributed file systems, and offered with “hope that is OK” — a piece that asks permission to be short before it has begun.

  • The student’s evidence is their own week. Searches that return instantly are offered as proof that these systems work, and the professor’s answer is that they work despite constant failure, which is the whole reason there is machinery to study.

  • Failure stops being all-or-nothing. Part of the system can be gone while the rest keeps answering, so there is now a middle state between working and broken — and everything the professor names, replication and retry alike, exists to keep you from noticing it.

Check yourself: the distributed-systems setup

1.According to the professor, what is the fundamental reason distributed systems are hard to build?

2.Given that failures happen constantly, how do real systems (Google, Facebook, Amazon) manage to 'mostly work'?

3.Within distributed systems, what does the book say it will focus on?

4.How does this final 'peach' differ from the earlier ones, capturing what's new about distribution?

4 questions