Every Row Has a Home
Give every row a home region so its vote stays local, then see why a new key still has to be checked in every region
★Who votes when the copies span three continents?
“Does Iowa have to agree to Priya’s new reading?”
Part I ended on a question: can the copies stay near their users without every write crossing the ocean? The app now runs in three regions, Iowa, Mumbai and Europe, and with a copy in each, any majority (2 of 3) includes a region across an ocean. So when Priya, a user in Mumbai, adds a reading (one row in the readings table), her write waits on a vote across the ocean, and this part asks where a row’s vote can live so that it doesn’t.
01What if every row had a home, and only the home voted?
Give each row a home region, near the person who writes it: make the table
REGIONAL BY ROW and each row, not the whole table, gets its own home, kept in a hidden crdb_region column. By default its lead (the copy every write goes through, Part I’s panel 02) and all its voting copies live there, so only the home votes and a Mumbai row’s write needs no vote from Iowa (the first supporting block below shows how that survives failures). CockroachDB puts the region in front of the key, so each range, a slice of the table, holds one region’s rows while the server still just sees priya:r1.SUPPORTING — HOW CAN THE VOTE STAY IN ONE REGION AND STILL SURVIVE FAILURES?▾
What’s a zone?
“What’s a zone?”
A region is one cloud location: Google’s us-central1 is in Iowa, asia-south1 in Mumbai. Inside a region sit several zones, which Google builds so that a power, cooling or network outage in one is unlikely to hit the others, joined by fast links so traffic between them stays quick. CockroachDB runs as a set of nodes, the machines that hold the data, and here each zone holds one.
Why one copy per zone?
“Does it matter where the copies sit?”
Part I’s rule: a range, one slice of a table, keeps working while 2 of its 3 copies are alive. Put all three copies in one zone and one outage takes all three; put one in each zone and losing a zone still leaves 2 of 3. You don’t place them yourself: each node is started with its zone as a setting, and CockroachDB spreads a range’s copies across as many different zones as it can.
What should the database survive: a zone, or a whole region?
| SURVIVE ZONE FAILURE | SURVIVE REGION FAILURE | |
|---|---|---|
| the vote | local | slower |
| reads | same | same |
| lose a whole region | data voting there stops | keeps working |
SWITCH WITH:
ALTER DATABASE … SURVIVE REGION FAILURE;
You give each database a survival goal: the failure it must keep working through. The default survives losing a zone, because every copy that votes sits in one region, one per zone, so the vote never leaves that region. Surviving a whole region takes 5 copies spread 2+2+1, so losing any one region still leaves at least 3 of 5, a majority (Part I, panel 02), and every write then waits on at least the nearest other region.
Why at least three regions?
“Can we start with two regions?”
Surviving a whole region, the second goal above, keeps 5 copies, and a range works while 3 of the 5 are alive. With two regions, Iowa and Mumbai, one of them holds at least 3 copies, so losing it leaves 2 and the range stops; adding Europe to split them 2+2+1 means losing any one leaves at least 3. So, whichever goal you pick, a multi-region CockroachDB Advanced cluster (one of CockroachDB Cloud’s plans) needs at least three regions, and at least 3 nodes in each, so data whose 3 votes sit in one region (the default goal above) survives losing a node there.
SUPPORTING — WHERE DOES AN ORDINARY TABLE VOTE?▾
So which region holds an ordinary table’s vote?
A table’s home region is where the leads of its ranges sit (the lead is the copy every write goes through, Part I’s panel 02), and by default every table is homed in the database’s primary region, the first region you add to it with
ALTER DATABASE … PRIMARY REGION. Make Iowa the primary region and the whole readings table is homed there, its lead and all its voting copies too (the default goal, in the block above). So Priya’s write from Mumbai still crosses the ocean, the cost the ★ panel opened with, which is why panel 01 gives each row its own home instead of giving the whole table one.02Who picks a row’s home?
“Do we have to pick a home for each user?”
WHICH REGION SERVES YOU? THE LOAD BALANCER
WHICH REGION IS THE ROW’S HOME? THE GATEWAY
SAME ANSWER BY DEFAULT
No: if you don’t say, a new row’s home is the gateway region (call it the gateway default), the region of the CockroachDB node your server’s connection landed on. For Priya that is Mumbai, since the load balancer lands her on the Mumbai server and that server uses Mumbai’s nodes (the first supporting block below shows how), so by default the home is per row, not per user, and her readings from a US trip are homed in Iowa. You can also set the home yourself on insert or change it later with an update, and keeping one home per user is possible too (Part III, “Should each user have one home region?”).
SUPPORTING — HOW DOES PRIYA’S REQUEST REACH MUMBAI’S NODES?▾
How does Priya’s request reach Mumbai’s nodes?
“Does the load balancer pick the database too?”
A global load balancer gives users one address and sends each to the closest healthy server, since you run a server in each region (a DNS geolocation policy gets a similar result by giving each region’s users a different address). It only handles the user-to-server hop; which database the server talks to is the server’s own setting, and the Mumbai server is simply given the address of the database’s Mumbai nodes. That’s the shape CockroachDB’s own docs describe, each server pointed at the database nodes in its region, so the node a user’s request reaches is normally in the user’s own region.
SUPPORTING — WHAT IF A BACKGROUND WORKER DOES THE INSERT?▾
Who’s the gateway when a worker does the insert?
The gateway default looks at the connection that ran the insert, not at the person the row is about. If a background worker in Iowa, connected to Iowa’s nodes, takes Priya’s reading off a queue and writes it, the row’s home is Iowa, and every later write to that row from Mumbai crosses the ocean. Point the worker at the nodes of the region it writes for, or set
crdb_region on the insert yourself.03What do the other regions hold?
Every other region keeps a read-only copy that never votes: it follows the lead’s log, the record of the writes the lead accepted (Part I, “What happens on a write and a read, step by step?”), so it has almost no effect on write speed. In the docs’ three-region example that makes 5 copies of each range, 3 voting at home and 1 read-only in each other region, unlike the goal of surviving a whole region, where all 5 vote. Only the home votes, remember; these copies just listen.
04So are Priya’s writes fast now?
KEY UNIQUE ACROSS THE WHOLE TABLE → CHECK EVERY REGION
STORED AS mumbai/priya:r9 · priya:r9 MUST BE UNIQUE
Updates to Priya’s Mumbai-homed rows now stay in Mumbai, but the write this part opened with, adding a new reading, still waits on the ocean: its key,
priya:r9, must be unique across the whole table, and a row with that key could be homed in any region, like her US-trip readings in Iowa (panel 02). So before the insert commits, CockroachDB asks Iowa and Europe whether priya:r9 is taken, and the insert waits on both answers. A home decides who votes, not where a key might already exist, so how can Priya’s new reading be inserted without first asking Iowa and Europe?