Explain how Raft works and why a majority quorum prevents split brain.

Raft elects a single leader per term. Followers who stop hearing heartbeats become candidates, increment the term and request votes; a node votes at most once per term, so at most one candidate can win a majority. The leader accepts all writes, appends them to its log, replicates to followers, and commits an entry once a majority have stored it. Split brain is impossible because two disjoint majorities of the same set cannot exist - any two majorities of N nodes must share at least one member, and that member's single vote per term prevents a second leader. 1 · Leader election: Each follower runs a randomised election timeout (150–300 ms). If no heartbeat arrives it becomes a candidate, increments the term and requests votes. Randomised timeouts are what stop every node becoming a candidate simultaneously and splitting the vote forever.. 2 · Winning requires a majority: A candidate needs votes from more than half the cluster. Each node grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own - which guarantees no committed entry is ever lost.. 3 · Log replication: All client writes go to the leader, which appends to its log and sends AppendEntries to followers. Once a majority have persisted an entry, the leader marks it committed and applies it to the state machine.. 4 · Safety on partition: A leader that loses contact with the majority cannot commit anything - it simply stops making progress. When it rejoins and sees a higher term, it steps down and truncates any uncommitted entries. No divergence, no data loss. N2 + 1 + N2 + 1 > N Cluster size | Majority | Failures tolerated | Note. 3 | 2 | 1 | The usual minimum. 5 | 3 | 2 | Standard for important clusters. 7 | 4 | 3 | Diminishing returns; slower commits. 4 | 3 | 1 |

Implement a distributed lock with Redis. Why is the naive version unsafe, and what fixes it?

The naive version is `SET key value NX PX ttl` plus a delete on release, and it is unsafe for two reasons. First, releasing must verify ownership atomically - otherwise a client whose lock expired can delete a lock now held by someone else, which requires a Lua script comparing the token. Second, and more fundamentally, a TTL-based lock cannot guarantee mutual exclusion: a client can be paused by GC or a network delay, its lock expires, another client acquires it, and then the first client wakes up and acts believing it still holds the lock. The only real fix is a monotonically increasing fencing token that the protected resource itself checks and rejects if stale. The fundamental problem no TTL can solve: Client A acquires the lock with a 10-second TTL. A 12-second stop-the-world GC pause hits. The lock expires. Client B acquires it and starts working. Client A wakes up, still believing it holds the lock, and writes. Two clients in the critical section, and neither did anything wrong. No amount of TTL tuning fixes this, because you cannot bound a process pause, a network delay or a clock jump. Fencing tokens: the only real fix How a fencing token prevents the stale writer: Client A - token 33 - Acquires the lock and receives token 33, then pauses for a long GC. Its lock silently expires while it is stopped.. Client B - token 34 - Acquires the now-free lock and receives token 34. Writes successfully, so storage records 34 as the highest token seen.. Lock service - monotonic counter - Every successful acquisition returns a strictly increasing token. This is the piece the naive Redis lock lacks entirely.. Storage - rejects token ≤ 34 - Records the highest token it has accepted. When A finally wakes and writes with token 33, the write is rejected - correctness is enforce

Consensus, Raft and Distributed Locks Explained

Expertraftconsensusleader electionAsked at Google, CockroachLabs

30-second answer

Raft elects a single leader per term. Followers who stop hearing heartbeats become candidates, increment the term and request votes; a node votes at most once per term, so at most one candidate can win a majority. The leader accepts all writes, appends them to its log, replicates to followers, and commits an entry once a majority have stored it. Split brain is impossible because two disjoint majorities of the same set cannot exist - any two majorities of N nodes must share at least one member, and that member's single vote per term prevents a second leader.

1 · Leader election
Each follower runs a randomised election timeout (150–300 ms). If no heartbeat arrives it becomes a candidate, increments the term and requests votes. Randomised timeouts are what stop every node becoming a candidate simultaneously and splitting the vote forever.
2 · Winning requires a majority
A candidate needs votes from more than half the cluster. Each node grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own - which guarantees no committed entry is ever lost.
3 · Log replication
All client writes go to the leader, which appends to its log and sends AppendEntries to followers. Once a majority have persisted an entry, the leader marks it committed and applies it to the state machine.
4 · Safety on partition
A leader that loses contact with the majority cannot commit anything - it simply stops making progress. When it rejoins and sees a higher term, it steps down and truncates any uncommitted entries. No divergence, no data loss.

⌊ \frac{N}{2} ⌋ + 1 + ⌊ \frac{N}{2} ⌋ + 1 > N

Two majorities of the same N-node set must overlap in at least one node. That node votes once per term, so two leaders in one term are impossible.

Cluster size	Majority	Failures tolerated	Note
3	2	1	The usual minimum
5	3	2	Standard for important clusters
7	4	3	Diminishing returns; slower commits
4	3	1	Worse than 3 - same tolerance, more latency
2	2	0	Pointless: either node failing halts the cluster

Always use an odd number. Even sizes add a node without adding fault tolerance, while making every commit wait on more acknowledgements.

Where you actually meet consensus

System	Algorithm	Used for
etcd	Raft	Kubernetes control plane, service discovery, config
ZooKeeper	ZAB	Kafka (historically), HBase, leader election
Consul	Raft	Service discovery, health, KV config
Kafka (KRaft)	Raft	Metadata and controller election
Spanner	Paxos	Per-shard replication groups
CockroachDB / TiDB	Raft	Per-range replication

Say these points

Raft: randomised election timeouts, one vote per node per term, majority to win.
Two majorities of the same set must overlap, which makes split brain impossible.
Always odd cluster sizes; 4 nodes tolerate no more failures than 3.
The minority side becomes unavailable, never inconsistent.
Use consensus for rarely-changing metadata, never on the per-request data path.

Avoid these mistakes

Even-sized clusters.
Assuming a leader that has lost quorum knows it immediately - it does not, until it tries to commit.
Running consensus across regions and paying WAN latency on every write.
Confusing 'a majority acknowledged' with 'all replicas have applied it'.

Expect these follow-ups

What happens if two nodes both believe they are leader in different terms?
How does Raft handle a follower whose log has diverged?
Why does a leader need to commit a no-op entry when it starts a new term?

Expertdistributed lockredisfencingAsked at Amazon, Stripe

30-second answer

The naive version is `SET key value NX PX ttl` plus a delete on release, and it is unsafe for two reasons. First, releasing must verify ownership atomically - otherwise a client whose lock expired can delete a lock now held by someone else, which requires a Lua script comparing the token. Second, and more fundamentally, a TTL-based lock cannot guarantee mutual exclusion: a client can be paused by GC or a network delay, its lock expires, another client acquires it, and then the first client wakes up and acts believing it still holds the lock. The only real fix is a monotonically increasing fencing token that the protected resource itself checks and rejects if stale.

pythonAcquire and release - the atomic release is mandatory

import uuid, redis

def acquire(r, key, ttl_ms=10_000):
    token = str(uuid.uuid4())                       # unique per acquisition
    ok = r.set(key, token, nx=True, px=ttl_ms)      # atomic: set-if-absent + expiry
    return token if ok else None

# ❌ Unsafe release: your lock may have expired and been re-acquired by someone
#    else between your check and your delete. You would delete THEIR lock.
#    if r.get(key) == token: r.delete(key)

# ✅ Atomic compare-and-delete in a single Lua script.
RELEASE = """
if redis.call('get', KEYS[1]) == ARGV[1] then
  return redis.call('del', KEYS[1])
else
  return 0
end
"""
def release(r, key, token):
    return r.eval(RELEASE, 1, key, token)

Fencing tokens: the only real fix

How a fencing token prevents the stale writer

The lock service issues an ever-increasing token; the storage layer rejects anything below the highest it has seen.

ClientServiceDatastoreTap a box for detail

Flow

Client ALock service(acquire → 33)
Client BLock service(acquire → 34)
Client BStorage(write(34) ✓)
Client AStorage(write(33) ✗ rejected)

When you actually need a distributed lock

Usually you do not. Prefer a database uniqueness constraint, an atomic UPDATE … WHERE version = ?, or a single-partition consumer that serialises work by design.
Efficiency locks (avoid duplicate expensive work, e.g. two workers regenerating the same report) tolerate occasional double execution. Redis is fine here.
Correctness locks (mutual exclusion actually required) need consensus-backed leases plus fencing tokens - etcd, ZooKeeper - or a design that removes the need for a lock.
Redlock, the multi-Redis variant, is contested: it still relies on bounded clock drift and process pauses, so it does not solve the fundamental problem. Knowing the debate exists is itself a good signal.

Say these points

`SET NX PX` to acquire; release must be an atomic compare-and-delete via Lua.
TTL locks cannot guarantee mutual exclusion - pauses are unbounded.
Fencing tokens enforced at the resource are the only real fix.
Distinguish efficiency locks (best-effort is fine) from correctness locks.
Prefer database constraints or partitioned consumers over distributed locks.

Avoid these mistakes

Releasing with GET-then-DEL instead of an atomic script.
Believing a longer TTL makes the lock safe.
Using a Redis lock where correctness genuinely depends on mutual exclusion.
Forgetting that the lock holder must renew if the work outlasts the TTL.

Expect these follow-ups

How would you implement a lease that renews while work is in progress?
Design this so no lock is needed at all.
What are the arguments against Redlock?

Consensus, Leader Election and Distributed Locks

Explain how Raft works and why a majority quorum prevents split brain.

Where you actually meet consensus

Implement a distributed lock with Redis. Why is the naive version unsafe, and what fixes it?

Fencing tokens: the only real fix

How a fencing token prevents the stale writer

When you actually need a distributed lock

Check your understanding

Hands-on challenge

Build a leader-elected job scheduler

Curriculum