The Server Moved, the Data Didn’t
Why moving the server to Mumbai made the database slower, how a copy of the data there fixes reads, and why keeping the copies in step makes far-away writes slow
★Why did moving the server closer make the database slower?
1 CROSSING, ≥130 MS → 5 CROSSINGS, ≥650 MS
“Why is it slower now?”
Traffic from India grows, so you move the server to Mumbai and the phone-to-server hop shrinks to almost nothing. But the database stayed in Iowa, and the server talks to it once per query: a request that makes, say, five queries one after another now pays five ocean round trips where it used to pay one, and light in fiber alone needs about 130 ms for each. Your users got closer to the server; the server got further from its data.
01Why not put a copy of the database in Mumbai too?
“Same table, two places?”
Then do exactly that: keep a copy of the database in Mumbai as well as in Iowa, so the Mumbai server reads the Mumbai copy and reads stay local. The readings table, one row per measurement a user’s phone sends, now lives in two places, and a request that reads five times no longer pays five ocean round trips. The catch to come: two copies are really one table only if they stay in step on every write.
02How do the copies stay in step?
THE LEAD (C) SENDS THE WRITE TO A AND B AT ONCE
B ANSWERS FIRST: C + B IS 2 OF 3
Before stretching the copies across an ocean, see how CockroachDB keeps them in step in one place, Iowa: several machines called nodes act as one database, and each slice of a table, a range, is kept as 3 copies on 3 different nodes. One copy is the lead: every write goes through it, it sends the write to the other two, and the write is done once a majority has it (2 of 3, the lead counting itself). That agreement among the copies is the vote; the supporting blocks below show how a table is split and copied, the write step by step, and why three.
SUPPORTING — HOW DOES COCKROACHDB SPLIT AND COPY A TABLE?▾
What does CockroachDB look like inside Iowa?
“One table, three machines. Who holds what?”
CockroachDB doesn’t run on one machine: it runs on several, called nodes, and together they act as one database. Your server can connect to any node and query any table; here it picked A, which hands the work to wherever the data sits and carries the answer back. So how does one table live on three machines?
What is a range, and who decides how many?
CockroachDB sorts a table by its primary key and cuts it into slices called ranges; with a key like (user, reading), one user’s readings sit next to each other. You don’t have to pick how many: a new table starts as one range and splits when a range reaches 512 MiB (the default maximum size of a range) or gets too busy. You choose how many nodes; CockroachDB decides how many ranges.
How many copies, on how many nodes?
Each range is kept as 3 copies on 3 different nodes by default, so a node can fail without losing it (the “Why three copies?” block below counts how many failures). As the table grows to 5 ranges and you add nodes, each range still has 3 copies; they just spread out, so each node holds only some of the ranges. More nodes means more room, not more copies.
SUPPORTING — WHAT HAPPENS ON A WRITE AND A READ, STEP BY STEP?▾
How does a write get agreed?
1 · SERVER → A
2 · A FORWARDS TO C
(A IS NOT THE LEAD)
(A IS NOT THE LEAD)
3 · C WRITES TO ITS LOG AND SENDS TO A AND B AT ONCE
4 · B ANSWERS FIRST
(THAT MAKES 2 OF 3)
(THAT MAKES 2 OF 3)
5 · C → A
6 · A → SERVER
One copy of each range is the lead (CockroachDB calls it the leaseholder), and every write goes through it. The lead saves the write in its log, its own record of the writes it accepted, and sends it to the others; once a majority has it (2 of 3, the lead counting itself) the write is committed, and that agreement among the copies is the vote. Whichever node your server happened to connect to plans the query, hands the write to the lead and carries the answer back.
Does a read need a vote?
A normal read goes to the lead alone, with no vote: every write goes through the lead, so the lead always has the latest. And the lead is a role, not a place; CockroachDB can hand it to another copy to balance load or to replace a dead node, without moving any data.
SUPPORTING — WHY THREE COPIES?▾
Why three copies?
| COPIES | MAJORITY | FAILURES SURVIVED |
|---|---|---|
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
FAILURES SURVIVED = (COPIES − 1) ÷ 2, ROUNDED DOWN
A range keeps working while a majority of its copies can reach each other, so 5 copies survive 2 failures, and 3 is the fewest copies that survive even one (4 survives no more). Lose 2 of 3 and that range stops, unavailable but never wrong, because CockroachDB would rather refuse than return inconsistent results. Cut the network in two and only the side holding a majority can write, so the halves can never drift apart.
03So what happens when the copies span two continents?
“Who has to agree now?”
Spread the copies over both continents and the majority, or the lead that gathers it, is always across the ocean from some writers: for them, like the Mumbai server here, every write waits a round trip, an AGREE? out and an AGREED back. Reads got closer, but writes got slower. So can the copies stay near their users without every write crossing the ocean?