The Gaps We Carry

Forty-two parables on how systems fail

by Baisil Geevarghese

Two full buckets, and the rope for the one that isn't there.

This is a book about the ways systems fail, and it is told as a story. The same village, a year on. The well works. There is a second well past the ridge now, a mill that takes orders, and a man who spent thirty years building these things before he came home. The same four people keep coming back to ask him questions, and the questions have changed. They are no longer about how to build a thing. They are about the morning it stopped working and nobody could say why. Across forty-two chapters they work out why a copy of an answer goes stale without telling anyone, what happens when the thing you depend on turns slow rather than broken, why the step everybody believes is the slow one usually is not, what a system costs to keep long after the day it was built, and why so many faults are found by a customer rather than by anything you own. WHAT IS IN EVERY CHAPTER First the story. Four people get it wrong before they get it right, and the wrong answer is almost always the one you would have given. Then the Master puts the real name to what just happened. Then it is explained plainly: what it is, why the trouble is built into the shape of it rather than caused by anyone being careless, what it costs, why the obvious fixes fail, and when you should not do any of it. Then one real incident, followed from the first symptom to the fix, including the wrong turns and how long they took. Nobody in those accounts behaves stupidly. That is the point of including them. Then a table of what each answer actually costs, because a strategy offered without its price is not advice. Then a short list of things to go and look at in your own system this week. Every item is something you can come back from with an answer. WHAT IT COVERS Caching and the copies that go quietly out of date. Finding things that have moved. Timeouts, retries and the convoy they build. Circuit breakers that never trip because slow was never counted as failure. Race conditions, locking and transactions. Logs, metrics, tracing and the alarm nobody believes any more. Measuring where the time really goes. Percentiles and the morning the average hides. What a system costs to own. Deploying, rolling back and the switch you can actually reach at two in the morning. Data modelling, search, backups, migration and what should not be kept at all. Rate limiting and backpressure. Breaking changes, boundaries and proving what happened. WHO IT IS FOR If you work with software, or near it, or on top of something somebody else built and left, this is the part nobody explained. It is more technical than the first book. It is not more advanced. There is no code in it, no mathematics beyond arithmetic a baker does in her head, no named theorems or algorithms, and no tool, company or product is named anywhere. The things that change are not the things worth teaching. Eighty-four diagrams. They are drawings of mechanisms, not decoration. You do not need to have read Beginnings at the Well. You will occasionally feel that somebody has started a sentence in the middle, and that is because they have.