Summary Dialogue on Persistence

Part III 💬 Dialogue OSTEP pp. 601–602 · ~6 min read

The one dialogue where the student guesses that the professor is a real person rather than a device of the book, and gets no denial.

Why this matters

Both of them are constructs written by the people being asked about, and saying so costs the summary nothing. What it buys is an admission the other dialogues never make: the voice teaching you has a stake in the material, and it gets caught here promising to write a chapter the reader has already read.

The persistence arc closes the way the others did — the professor, the student, and one last look back before the network arrives.

Student:Wow, file systems seem interesting(!), and yet complicated.
Professor:That’s why my spouse and I do our research in this space.
Student:Hold on. Are you one of the professors who wrote this book? I thought we were both just fake constructs, used to summarize some main points, and perhaps add a little levity in the study of operating systems.
Professor:Uh… er… maybe. And none of your business! And who did you think was writing these things? (sighs) Anyhow, let’s get on with it: what did you learn?
Student:Well, I think I got one of the main points, which is that it is much harder to manage data for a long time (persistently) than it is to manage data that isn’t persistent (like the stuff in memory). After all, if your machine crashes, memory contents disappear! But the stuff in the file system needs to live forever.
Professor:Well, as my friend Kevin Hultquist used to say, “Forever is a long time”; while he was talking about plastic golf tees, it’s especially true for the garbage that is found in most file systems.
Student:Well, you know what I mean! For a long time at least. And even simple things, such as updating a persistent storage device, are complicated, because you have to care what happens if you crash. Recovery, something I had never even thought of when we were virtualizing memory, is now a big deal!
Professor:Too true. Updates to persistent storage have always been, and remain, a fun and challenging problem.
Student:I also learned about cool things like disk scheduling, and about data protection techniques like RAID and even checksums . That stuff is cool.
Professor:I like those topics too. Though, if you really get into it, they can get a little mathematical. Check out some of the latest on erasure codes if you want your brain to hurt.
Student:I’ll get right on that.
Professor:(frowns) I think you’re being sarcastic. Well, what else did you like?
Student:And I also liked all the thought that has gone into building technology-aware systems, like FFS and LFS. Neat stuff! Being disk aware seems cool. But will it matter anymore, with Flash and all the newest, latest technologies?
Professor:Good question! And a reminder to get working on that Flash chapter… (scribbles note down to self) … But yes, even with Flash, all of this stuff is still relevant, amazingly. For example, Flash Translation Layers (FTLs) use log-structuring internally, to improve performance and reliability of Flash-based SSDs. And thinking about locality is always useful. So while the technology may be changing, many of the ideas we have studied will continue to be useful, for a while at least.
Student:That’s good. I just spent all this time learning it, and I didn’t want it to all be for no reason!
Professor:Professors wouldn’t do that to you, would they?

Everything the persistence part earned, gathered in one place — click each row for where it was learned:

The student's mental model of persistence — eight load-bearing beliefs, and where each was earned
the belief
1Persistent is far harder than volatile
2Talking to a device is its own discipline
3The disk is a mechanical beast
4Redundancy across disks survives failure
5Files and directories are an abstraction over blocks
6Locality is worth engineering for
7Crashes mid-update must be survivable
8New media, old ideas — and devices lie
Dotted-underlined cells have explanations — click one.

Next: Distributed Systems

We’ve made data survive a single machine — its crashes, its failing disks, its lying devices. The last stretch stretches storage across machines: how computers cooperate over an unreliable network they can’t trust, building reliable messaging and remote procedure calls on top of it, and sharing files through NFS and AFS. First, a short dialogue to set the stage.

If you remember only three things

  • Recovery is an obligation this part invented. A crash in Part I took the manager and the managed away together, so nothing survived to be inconsistent; here the data outlasts the failure and somebody must go find what was half-done.

  • One door is pointed at and left shut. Erasure codes are named, credited with hurting your brain, and dropped — an admission that the RAID chapter stopped well short of the mathematics of redundancy.

  • The technology changed and the ideas did not have to. Flash arrived with different physics and got managed by the log-structured design from the chapter before it — the one concrete instance behind the professor’s claim that these ideas outlast their hardware.

Check yourself: the persistence part in review

1.The student's headline takeaway: why is managing data persistently much harder than managing volatile (in-memory) data?

2.The professor notes that even brand-new Flash technology reuses an old idea. Which one?

3.Both RAID and checksums are 'data protection', but they guard against different things. What do checksums add that RAID's redundancy alone does not?

4.What's the professor's larger point about 'technology-aware' designs like FFS and LFS as hardware keeps changing?

4 questions