We shipped a graceful shutdown, tested it under real load, and got a perfect score. Then we found out the shutdown code hadn't saved a single request.
What happens when you replace a running app?
Your app runs as a copy on a server somewhere. Ship new code and that copy has to be replaced — something stops the old one and starts the new one.
The complication is that the old copy is busy. At any instant it is partway through answering real requests: someone loading a feed, someone posting a photo.
Why it doesn't stop politely on its own
To stop it, the system sends it a signal meaning please shut down — SIGTERM. Think of it as a tap on the shoulder.
By default, the app doesn't handle the tap. It just stops. Immediately.
Every request it was holding dies halfway. The person on their phone doesn't get an error page from your app — they get nothing at all, because your app is already gone. The connection simply drops.
And this happens on every deploy. Which quietly makes your own releases the most common source of errors in your system.
Broken deploys don't just cost you the broken requests. They cost you the ability to ship.
Because the fix people reach for is deploying less often. Weekly instead of daily. That makes each release bigger, riskier and harder to debug when it goes wrong — so the next one is scarier still, and you ship even less.
So what is a drain test?
Draining is the polite version of stopping. The copy stops accepting new requests, finishes the ones it is already holding, and only then exits. The word is a sink: close the tap, let the basin empty, walk away.
A drain test checks that this actually happens. You can't read it off a config file. Four separate systems each own one of the timers involved — the load balancer, the app, the thing running the container, and the client — and the behaviour is correct only if all four agree. Any one of them being wrong breaks it silently.
So you don't reason about it. You cause it: send steady traffic, kill the copy mid-flight, and count what broke.
Why reading the code isn't enough
Before any of this, someone had written the shutdown logic. You could read it and agree with it.
It would never have run.
The container started the app through a package manager, so the package manager was the program the operating system actually talked to and the app was its child. The tap on the shoulder went to the parent — which doesn't pass signals down to its children.
The app never heard it. It was killed outright 0.2 seconds later, on every deploy, for months.
Those 0.2 seconds are the tell. A drain that works takes as long as the drain window — twenty seconds, in our case. A container that vanishes instantly hasn't drained. It has been shot.
The entire bug, and the entire fix, is one line of a startup script.
# before: the package manager is process 1, and it does not pass # the shutdown signal down to the app it started exec yarn start:prod # after: the app itself is process 1 exec $(node -p "require('./package.json').scripts['start:prod']")
The second version hardcodes nothing. It reads the very command the package manager would have read, and runs it itself — so there is still one place that says how the app starts, and the app is now the process the system is talking to.
That bug is invisible in the shutdown code, because it isn't in the shutdown code. Every line of the shutdown logic was correct. It would pass review; it would pass its tests. The defect was that nothing ever reached it — and that fact lives in a different file, one nobody thinks of as shutdown-related.
So you don't find it by reading. You find it by stopping the app and watching what the container actually does.
The test passed
Fixed, deployed, and tested properly: steady traffic at twenty requests a second, straight through a real rolling deploy.
5,600 requests. Zero errors. Every one answered normally.
Perfect. And almost completely uninformative.
Isn't zero the whole answer? It is the outcome we wanted, measured under the load we cared about. It is also the only thing we learned.
Zero errors doesn't tell you what saved you
A passing test gives you the outcome. It does not tell you which mechanism produced it.
Those sound like the same thing and they are not. We had a number that said safe. We had no idea which part of the system was making it safe — and there were two candidates.
What a counter can and can't say
That zero came from a counter, and it is worth being exact about what a counter is.
Once a minute, the machine in front of the app writes down two numbers: how many requests it handled, and how many of those came back as errors. For the minute our deploy landed in, it wrote 1,201 and 0.
Those two numbers are the complete record of that minute. They are not a summary of something more detailed held underneath — there is nothing underneath. Each request was added to the total and forgotten as it happened.
Which means the counter cannot answer the one question we actually had: which copy of the app served that request? It isn't withholding it. It never knew. A running total has no room for which, or who, or when.
That is the trade, and it is a deliberate one. A counter costs the same whether you serve a thousand requests a minute or ten million, because it is always just a number going up. Keeping the detail instead means writing down a line for every single request, forever.
A counter tells you whether you're okay. It can't tell you why.
The load balancer keeps its own diary
There is a thing sitting in front of your app that every request passes through — a load balancer. Think of it as a receptionist: requests arrive at her desk and she decides which copy of the app gets each one.
She can be told to keep a diary. Every request that crosses the desk, written down: who asked, which copy she handed it to, whether that copy answered, how long it took.
That is an access log. Turn it on and you stop having totals and start having receipts.
Here is what it showed. For the first two minutes and twenty seconds every request went to the old copy. Then for 13.8 seconds both copies were serving at once. Then everything went to the new one. Nothing was ever cut off, because there was always somebody to hand the request to.
The thirty-two seconds
Then the diary showed something nobody was looking for.
The old copy served its final request. The shutdown signal arrived thirty-two seconds later.
Thirty-two seconds of sitting there with nothing to do. So the drain — the careful logic that waits for in-flight work to finish — waited for requests that had completed half a minute earlier.
It protected nothing. What actually kept those 5,600 requests safe was something else: the scheduler had quietly stopped routing traffic to the old copy and waited, well before it told the app anything at all.
“You can't read it off a config file.” Neither, it turns out, can you read off which of your safety mechanisms is the one doing the work.
So was the drain pointless?
No — and this is the part worth being careful about.
The drain is redundant on the path we tested. During an ordinary deploy the scheduler empties the copy first, so by the time the drain starts there is nothing left for it to do.
It is the only thing that exists on every other path. A machine reclaimed by the cloud provider, a host that dies, a container killed for using too much memory — none of those deregister anything first. There is no warning, and the drain is the entire safety net.
Redundancy on the path you tested isn't redundancy. It is coverage of the paths you didn't test that happens to overlap with the one you did — and the path that's easy to trigger deliberately is almost always the gentlest one you have.
The one-paragraph version
Replacing a running copy kills whatever it was holding, so you teach it to drain: stop taking new work, finish what's open, then exit. Four systems own a timer each, so you can't verify it by reading — you cause it under load and count the damage. But counting gives you an outcome, never a reason. The reason lives in the load balancer's record of every request, and ours said the drain had sat idle for thirty-two seconds while something else did the work. The number was right and the story we would have told about it was wrong.