← Learnings

A Circuit Breaker Alone Won't Save Your Database

2 min·resilience, redis, backend

A common misconception is that a circuit breaker protects your database. It doesn't. It only protects your application from repeatedly calling a dependency that is already known to be unhealthy.

What happens when Redis goes down?

Without a breaker, every single request pays the full cost of finding out.

WITHOUT A CIRCUIT BREAKER request Redis WAITS OUT THE FULL 500ms TIMEOUT, EVERY TIME database REACHED ANYWAY — THE WAIT BOUGHT NOTHING
FIG 1 · THE TIMEOUT IS PAID IN FULL, PER REQUEST

Each request waits out the timeout, spends a network call to learn nothing, and reaches the database anyway. The wait buys you nothing at all — the request was always going to end up at the database, and it arrives half a second late.

What the breaker actually changes

After Redis fails repeatedly, the circuit opens, and requests stop being sent to a dependency already known to be down.

AFTER THE CIRCUIT OPENS request circuit breaker ANSWERS IMMEDIATELY — REDIS IS NOT CALLED fallback WHATEVER YOU CHOSE IT TO BE
FIG 2 · REDIS IS NO LONGER ASKED

Notice what moved and what didn't. The circuit breaker doesn't decide the fallback. It decides exactly one thing:

Should I even try Redis?

So what is the fallback?

That's your decision, not the breaker's, and in a production system it is usually a ladder rather than a single destination.

WHAT THE FALLBACK ACTUALLY IS circuit open? no call Redis normally local cache hit? yes return it rate limiter allows? no return 503 database ONLY WHAT THE LIMITER LET THROUGH
FIG 3 · EVERY RUNG IS CHEAPER THAN THE ONE BELOW IT

Each step is a cheaper way of answering than the step under it, and the database is the last resort rather than the default.

Why a DB rate limiter?

Because of the arithmetic underneath the cache.

WHAT REDIS WAS ABSORBING normal traffic 100K RPS served by Redis 99K RPS THE CACHE IS DOING ALMOST ALL OF IT the database can take 1K RPS WHAT IT WAS SIZED FOR Redis dies and all 100K arrives at a database built for 1K.
FIG 4 · THE CACHE WAS ABSORBING 99% OF IT

Your database was never sized for the traffic Redis was absorbing. If every request falls through to it, the fallback path finishes the job the Redis outage started. So the fallback has to be capped.

FALLBACK PATHGO
if circuitBreaker.IsOpen() {
    if localCache.Has(key) {
        return localCache.Get(key)
    }
    if !dbRateLimiter.Allow() {
        return 503
    }
    return db.Get(key)
}
FIG 5 · THE CAP IS THREE LINES, AND IT IS THE WHOLE POINT

Only a limited number of requests are allowed to reach the database. The rest fail fast, which keeps the system alive and degraded rather than dead.

Where does the limiter go?

Not in the API gateway. The gateway only sees GET /users/123. It has no idea whether your service is about to query Redis, query Postgres, call Elasticsearch, read from Kafka, or touch no database at all.

The service itself knows when it's about to hit the database, so that is where a dependency-specific limiter belongs. I've written up how that limiter is built and where exactly it sits in Database Rate Limiting: The Missing Piece After a Circuit Breaker.

A circuit breaker and a rate limiter solve different problems. The breaker stops you calling a dependency that is already failing; the limiter protects the database from being overwhelmed when the fallbacks arrive. In production they almost always work together.