The persistence arc closes the way the others did — the professor, the student, and one last look back before the network arrives.
Student:Wow, file systems seem interesting(!), and yet complicated.
Professor:That’s why my spouse and I do our research in this space.
Student:Hold on. Are you one of the professors who wrote this book? I thought we were both just fake constructs, used to summarize some main points, and perhaps add a little levity in the study of operating systems.
Professor:Uh… er… maybe. And none of your business! And who did you think was writing these things? (sighs) Anyhow, let’s get on with it: what did you learn?
Student:Well, I think I got one of the main points, which is that it is much harder to manage data for a long time (persistently) than it is to manage data that isn’t persistent (like the stuff in memory). After all, if your machine crashes, memory contents disappear! But the stuff in the file system needs to live forever.
Professor:Well, as my friend Kevin Hultquist used to say, “Forever is a long time”; while he was talking about plastic golf tees, it’s especially true for the garbage that is found in most file systems.
Student:Well, you know what I mean! For a long time at least. And even simple things, such as updating a persistent storage device, are complicated, because you have to care what happens if you crash. Recovery, something I had never even thought of when we were virtualizing memory, is now a big deal!
Professor:Too true. Updates to persistent storage have always been, and remain, a fun and challenging problem.
Student:I also learned about cool things like disk scheduling, and about data protection techniques like RAID raid Redundant Array of Inexpensive (or Independent) Disks: a technique that ties several disks into one logical disk to gain capacity, performance (parallelism), and/or reliability (redundancy). It presents the same block interface as a single disk, so it works transparently — you can swap a disk for a RAID without changing any software above it.
defined in ch. 38 — open in glossary
and even checksums checksum A small summary (typically 4–8 bytes) produced by a function computed over a chunk of data (say a 4 KB block). Stored alongside the data and recomputed when the data is read: if the stored and computed checksums differ, the data has been corrupted. The primary mechanism modern storage systems use to preserve data integrity. Common functions include XOR, addition, the Fletcher checksum, and the CRC.
defined in ch. 45 — open in glossary
. That stuff is cool.
Professor:I like those topics too. Though, if you really get into it, they can get a little mathematical. Check out some of the latest on erasure codes if you want your brain to hurt.
Student:I’ll get right on that.
Professor:(frowns) I think you’re being sarcastic. Well, what else did you like?
Student:And I also liked all the thought that has gone into building technology-aware systems, like FFS and LFS. Neat stuff! Being disk aware seems cool. But will it matter anymore, with Flash and all the newest, latest technologies?
Professor:Good question! And a reminder to get working on that Flash chapter… (scribbles note down to self) … But yes, even with Flash, all of this stuff is still relevant, amazingly. For example, Flash Translation Layers (FTLs) flash translation layer The FTL: the SSD's control logic that presents a normal logical block interface while translating client reads/writes into low-level flash read/erase/program operations. It aims for high performance (low write amplification, chip parallelism) and reliability (wear leveling), typically via a log-structured design + a mapping table.
defined in ch. 44 — open in glossary
use log-structuring log-structured file system LFS (Rosenblum & Ousterhout): a file system that turns ALL writes into sequential writes. It buffers updates (data AND metadata) in memory, then writes them to an unused part of the disk as one big segment — never overwriting in place (copy-on-write). Optimizes write performance (most traffic is writes; sequential ≫ random) and sidesteps RAID's small-write problem.
defined in ch. 43 — open in glossary
internally, to improve performance and reliability of Flash-based SSDs. And thinking about locality is always useful. So while the technology may be changing, many of the ideas we have studied will continue to be useful, for a while at least.
Student:That’s good. I just spent all this time learning it, and I didn’t want it to all be for no reason!
Professor:Professors wouldn’t do that to you, would they?
Everything the persistence part earned, gathered in one place — click each row for where it was learned:
| the belief | |
|---|---|
| 1 | Persistent is far harder than volatile |
| 2 | Talking to a device is its own discipline |
| 3 | The disk is a mechanical beast |
| 4 | Redundancy across disks survives failure |
| 5 | Files and directories are an abstraction over blocks |
| 6 | Locality is worth engineering for |
| 7 | Crashes mid-update must be survivable |
| 8 | New media, old ideas — and devices lie |
Next: Distributed Systems
We’ve made data survive a single machine — its crashes, its failing disks, its lying devices. The last stretch stretches storage across machines: how computers cooperate over an unreliable network they can’t trust, building reliable messaging and remote procedure calls on top of it, and sharing files through NFS and AFS. First, a short dialogue to set the stage.Check yourself: the persistence part in review
1.The student's headline takeaway: why is managing data persistently much harder than managing volatile (in-memory) data?
2.The professor notes that even brand-new Flash technology reuses an old idea. Which one?
3.Both RAID and checksums are 'data protection', but they guard against different things. What do checksums add that RAID's redundancy alone does not?
4.What's the professor's larger point about 'technology-aware' designs like FFS and LFS as hardware keeps changing?
4 questions