One writer, many readers — and how strict we want to be about it

1 Master–slave (leader–follower) replication

That teacher is the master (modern docs say leader or primary). The kids are the slaves (modern term: followers, replicas, or secondaries). We'll use "master/slave" because that's still the most common interview phrasing, but know the kinder, clearer words too — they mean the same thing.

Why copy data at all? (recall Session 7)

In Session 7 we said replication buys us three things. Master–slave gives all three:

• Read scaling — the headline benefit today. Most apps read far more than they write (think: 100 page-views per 1 comment posted). Spread reads across many slaves and you serve more traffic.
• High availability — if the master dies, a slave can be promoted to take over (Topic 3).
• Geography & backups — keep a slave in another data centre for disaster recovery, or one nearby your users for low-latency reads.

How a write propagates

When the master accepts a write, it doesn't just change its own data and forget about it. It records the change in an ordered log and ships that log to every slave. The log has different names in different systems but the same job:

The key word is ordered: every slave applies the changes in the same sequence the master did. Apply them out of order and the copies would disagree. So a slave is really just a machine replaying the master's diary, line by line, forever.

The topology, drawn out

Your app talks to the cluster through a layer that decides where each query goes. Writes go to the one master; reads fan out to the slaves:

Read/write splitting — how reads actually scale

The trick that scales reads is read/write splitting: the application (or a proxy in front of the database) sends every INSERT/UPDATE/DELETE to the master, and load-balances every SELECT across the pool of slaves.

# db_router.py — toy read/write splitter
master   = connect("db-master:5432")
replicas = [connect("db-replica-1:5432"),
            connect("db-replica-2:5432"),
            connect("db-replica-3:5432")]

def run(sql):
    if sql.strip().upper().startswith(("INSERT", "UPDATE", "DELETE")):
        return master.execute(sql)        # writes → the one master
    else:
        replica = random.choice(replicas)  # reads → spread across slaves
        return replica.execute(sql)

2 Basic setup & the replication factor

The replication factor, written X, is the number of total copies of the same data:

Why an odd number of servers?

Typically we use an odd number of total servers (the master plus an even number of slaves). This is not mandatory — you can have an even number — but odd gives you a clear majority, which makes tie-breaking trivial when the cluster has to vote (during master election, or during a quorum read/write). With an even number, a split vote can deadlock.

How many copies do you actually need?

• At least X = 2 (1 master + 1 slave). With only 1 copy there's no replication at all, so any disk failure means data loss.
• X = 3 is the typical default (1 master + 2 slaves).
• X = 5 for critical data where you cannot tolerate any data loss. Going above 5 buys almost nothing — it's already astronomically unlikely that 5 servers in the same shard fail at once.
• Very high X (e.g. 100) only when you're using replicas purely to scale read throughput. This is normally done for a cache, not the whole database, and it's usually fine for such a cache to be eventually consistent.

Common operational questions

3 Replication lag & async vs sync replication

Replication lag is the time gap between a write committing on the master and that same write becoming visible on a given slave. The big design decision is when the master considers a write "done" — and that gives us three replication modes.

Asynchronous replication — fast, possibly lossy

In asynchronous (async) replication, the master writes locally, immediately tells the client "done!", and ships the change to slaves afterwards, in the background. The client never waits for the slaves.

• Pro: very low write latency — the client only waits for the master.
• Con: if the master crashes after step 2 but before step 3 reaches any slave, that write is gone. The client was told "saved" but it never propagated. This is data loss.

Synchronous replication — safe, slow

In synchronous (sync) replication, the master waits for the slave(s) to confirm they've received and stored the change before telling the client "done."

• Pro: no data loss for an acknowledged write — at least one other node has it.
• Con: the client waits for the slowest slave on every write. And if a synchronous slave goes down or gets slow, writes stall — you've coupled the master's availability to the slave's. Pure full-sync (wait for all slaves) is rare for exactly this reason.

Semi-synchronous — the practical middle

Semi-synchronous replication is the popular compromise: the master waits for at least one slave to confirm, then acks the client. The rest catch up asynchronously.

This means at least one backup copy always exists (so a single master crash loses nothing), but you don't pay the cost of waiting for every replica. MySQL's semi-sync and PostgreSQL's synchronous_commit with one sync standby both work this way. As you'll see in Topic 4, "wait for at least K of N" is exactly a quorum — the same idea generalised.

The read-your-own-writes problem

Here's the classic bug async replication causes. A user updates their profile photo (write → master), then the page reloads and reads their profile (read → a slave that hasn't caught up yet). They see their old photo and think the save failed. They saved successfully — they just can't see their own write.

Common fixes — you'll be expected to name at least one:

• Read from the master for data the user themselves just changed (e.g. read their own profile from the master for the next 30 seconds after an edit).
• Sticky / monotonic reads — pin a given user's reads to one specific replica so they at least don't bounce backward in time between replicas.
• Wait for the replica to catch up — remember the log position of the user's write, and only let a replica serve them once it has applied up to that position.

4 Failover & promotion

Because only the master accepts writes, the master is a single point of failure for writes. If it dies, reads can still be served by slaves, but no new writes can happen until a new master exists. Failover is the process of detecting the dead master and promoting a slave to replace it. Promotion is the act of turning a read-only slave into the new writable master.

The failover sequence

1. Detecting the death

The cluster uses heartbeats: nodes ping each other on a timer. If the master misses several heartbeats in a row (say, no reply for 10 seconds), it's presumed dead. Note "presumed" — we can't actually tell a crashed master apart from a master that's merely slow or behind a broken network link. That uncertainty is the root of all the danger below.

2. Electing the new master

You don't pick a slave at random. You want the slave that is most caught up — the one whose replicated log position is furthest along — because any writes it's missing are lost forever once it becomes master. Many systems use a consensus algorithm such as Raft or Paxos to elect a leader safely (we treat these as black boxes here; their whole job is to make sure the cluster agrees on exactly one leader).

3. Promotion

The chosen slave flips from read-only to read-write. The other slaves are told to start replicating from the new master instead of the dead one.

4. Reconfiguration

Clients and the router must learn the new master's address. This is often done with a stable hostname or a virtual IP that gets re-pointed, or via a service-discovery layer, so apps don't need to hard-code which box is master.

Split-brain — the nightmare

Split-brain happens when a network partition (recall the "P" in CAP from Session 7) makes part of the cluster think the master is dead and promote a new one — while the old master is actually alive and still taking writes on the other side of the partition. Now you have two masters, each accepting conflicting writes, with no idea the other exists.

How systems prevent split-brain:

• Quorum / majority rule — a node may only act as master if it can talk to a majority of nodes. In a partition, only one side can hold the majority, so the minority side refuses writes. (This is why clusters favour an odd number of nodes — 3, 5, 7 — so a majority always exists.)
• Fencing / STONITH ("Shoot The Other Node In The Head") — before promoting a new master, forcibly disable the old one (cut its power or network) so it physically cannot keep serving writes.
• Leases / tokens — a master holds a time-limited lease; if it can't renew it (because it's partitioned away), it must stop accepting writes on its own.

A word on multi-master

So far there's exactly one writer. Multi-master (also multi-leader) replication intentionally allows several nodes to accept writes at once — handy for writing locally in multiple data centres. The price is that the same row can be edited in two places simultaneously, so you must have a conflict-resolution strategy (last-write-wins by timestamp, version vectors, CRDTs, or app-defined merges). Master–slave sidesteps all of that by having one writer; multi-master trades that simplicity for write availability and locality. Leaderless systems (Dynamo-style) push this even further — which leads us straight to quorums.

5 The four consistency configurations

Master–slave is not one fixed thing — by choosing how many places a read touches (R) and how many places a write touches (W), you pick a point on the consistency spectrum. Below are the four classic configurations, from least to most consistent. In every one, reads use R = 1 (one random slave, chosen fresh each time via round-robin); what changes is the writes.

Config A — No consistency (data loss possible)

R = 1, W = 1. Reads go to any 1 random slave; writes go only to the master.

Writes going only to the master doesn't mean slaves have no data — they sync up periodically, so the master always has the latest data and the slaves usually lag a bit.

• Can there be data loss? Yes. If the master's HDD fails with unsynced changes, those changes are lost forever.
• Can stale reads happen? Yes — slaves sync only periodically. But here we don't even care about stale reads, because data loss is worse and this config already allows it.

Config B — Eventual consistency

R = 1, W = 2. Use this when eventual consistency is good enough. Reads go to any 1 random slave; writes go to the master + 1 random slave, atomically (random slave chosen round-robin).

• How do we write to two places at once? Using Two-Phase Commit (2PC): acquire distributed locks so the write to (master + 1 slave) is atomic. If it doesn't succeed in both places, we roll back the partial write and return an error; the client can retry.
• Can there be data loss? No — the data is in at least 2 places. (If both those HDDs fail before any other slave syncs, sure — but that's astronomically rare; HDDs fail maybe once every 2–3 years. Want fewer losses? Write to more places.)
• Can stale reads happen? Yes. The latest write went to Master + Slave A, but a read went to Slave B which hasn't synced yet → stale read.

Config C — Immediate consistency: Write-Everywhere

R = 1, W = X. Use this when you need immediate consistency and writes are rare (read-heavy system). Reads go to any 1 random slave; every write goes to all X nodes (master + all slaves) atomically.

• Can there be data loss? No — every write is replicated everywhere; all copies stay in sync.
• Can stale reads happen? No — all copies are always in sync.

Config D — Immediate consistency: Quorum

R = W = (X+1)/2. Use this when you need immediate consistency and writes are frequent (write-heavy or balanced system). The idea: improve writes at the cost of reads. Write-everywhere has lightning reads but glacial writes; trade a little off each and both become decent.

• Can there be data loss? No — we write to a majority, and a majority is at least 2.
• Can stale reads happen? No — you always read the latest data, because a read-majority and a write-majority must overlap in at least 1 server, and that server has the latest write.

Since different replicas can hold different values, every write stamps a timestamp; on read, we return the value with the latest timestamp.

6 Tunable consistency & quorums

Instead of the rigid "one master, slaves are read-only" model, many distributed databases let you treat all replicas more equally and tune consistency per query using three numbers:

Why the overlap is guaranteed (the intuition)

Picture N=3 replicas as three boxes. A write lands in W of them; a read inspects R of them. By the pigeonhole principle, if W + R is bigger than the total N, the two groups can't be completely separate — at least one box must be in both. That shared box holds the freshest value, and the reader picks the most recent (each value carries a version/timestamp), so it always sees the latest write.

Unlocking the full spectrum

Here's the deeper insight from the four configs above: for immediate consistency, R and W don't have to be equal. All that matters is that the read set and write set overlap. Three knobs control everything:

The four configurations of Topic 5 are just special points in this space:

Because we can choose any R and W, we get tremendous power:

• If R is low → reads are faster and more available (touch fewer servers).
• If W is low → writes are faster and more available.
• If R + W > X → the system is immediately consistent.

As R + W increases: availability drops and latency rises (more servers must be alive for requests to succeed), while consistency rises (more likely to find an overlap holding the latest write). So we can not only choose our consistency level — we can also tune the system for read-heavy or write-heavy operation. We're no longer stuck picking just "availability" or "consistency"; we can land on any point on the spectrum.

Tuning the dials for different goals

The beauty is you can slide W and R to favour reads, writes, or safety — sometimes per query:

How real systems expose this

This model comes from Amazon's 2007 Dynamo paper and lives on in Apache Cassandra, Amazon DynamoDB, Riak, and others. They're leaderless — no single master; you talk to a coordinator node that fans the request out to N replicas and waits for W or R of them.

-- Keyspace pins N = 3 copies per row
CREATE KEYSPACE shop WITH replication = {
  'class': 'SimpleStrategy', 'replication_factor': 3 };

-- A write that needs W = 2 (QUORUM of 3 = 2)
INSERT INTO shop.orders (id, total) VALUES (42, 99)
  USING CONSISTENCY QUORUM;

-- A read that also needs R = 2 → W+R = 4 > 3 → strong
SELECT total FROM shop.orders WHERE id = 42
  USING CONSISTENCY QUORUM;

How do you write to more than one place atomically?

Whenever a config requires writing to several servers at once (W = 2, quorum, or write-everywhere), those writes must be atomic — all-or-nothing. The standard mechanism is Two-Phase Commit (2PC): a prepare phase where every participant locks and votes "ready," followed by a commit phase where the coordinator tells everyone to commit (or, if anyone voted no, to abort and roll back). If it can't succeed everywhere, the partial write is rolled back and the client retries.

★ Putting it all together

Four topics, one storyline: how to keep copies of your data, and how strict to be about them.

Quick self-check

📚 References & Further Reading

Class material

• 📄 Original class notes / handout (Google Doc) — open the shared class material for this session.
• Class handout: "[SST-2028] Master Slave".

Papers, docs & deep dives

• PostgreSQL — High Availability, Load Balancing & Replication — real-world leader/follower replication, sync vs async, and failover.
• Amazon Dynamo (SOSP 2007) — quorum reads/writes (N, R, W) behind tunable consistency.
• Designing Data-Intensive Applications (Kleppmann) — Replication chapter — replication lag, read-your-writes, and promotion in depth.