Load Balancing Algorithms and Health Checking
Which algorithm to pick and why it matters more than you think. Plus the operational half nobody prepares for: health checks, connection draining and retry amplification.
Covers: Round robin, least connections, weighted, power of two choices, hashing; health checks; draining; outlier ejection
Choosing a load-balancing algorithm looks like a trivia question. It is not: the wrong choice concentrates load on your slowest instance, and the classic least connections algorithm has a failure mode where a broken server that errors instantly attracts all the traffic.
30-second answer
Round robin is simple and fine when every server and every request is roughly identical. Least connections adapts to uneven request cost and is the usual default for heterogeneous workloads. Weighted variants handle servers of different sizes. Consistent hashing routes the same key to the same server, which matters for cache locality and sticky routing. In practice the best general-purpose choice is power of two choices: sample two backends at random and pick the less loaded one - nearly the quality of least connections with none of its coordination cost or herd behaviour.
| Algorithm | How | Good when | Fails when |
|---|---|---|---|
| Round robin | Next server in rotation | Uniform servers and uniform requests | Request costs vary - slow server gets equal share |
| Weighted round robin | Proportional to configured weight | Mixed instance sizes | Weights go stale as conditions change |
| Least connections | Fewest in-flight requests | Variable request duration | A server failing fast has zero connections and attracts everything |
| Least response time | Lowest latency × connections | Latency-sensitive services | Needs accurate measurement; can oscillate |
| Power of two choices | Sample 2 at random, pick the lighter | Almost always - the pragmatic default | Slightly worse than perfect least-connections in theory |
| Consistent hashing | hash(key) → server on a ring | Cache locality, session affinity, sharded state | Uneven key distribution creates hot backends |
| Random | Uniform random | Very large fleets, minimal overhead | No load awareness at all |
Health checks: the part that decides whether any of this works
| Check type | Detects | Misses |
|---|---|---|
| TCP connect | Process is dead | Deadlock, exhausted pools, broken dependencies |
HTTP 200 on /health | Process can serve HTTP | Dependency failure, if the check does not test it |
| Deep check (verifies dependencies) | Real ability to serve | Risks cascading - see below |
| Passive / outlier ejection | Actual production failures | Nothing; it observes real traffic |
@app.get("/livez") # "should I be restarted?"
def livez():
return 200 # deliberately trivial: only process health
@app.get("/readyz") # "should I receive traffic?"
def readyz():
if db_pool.available() == 0: return 503, "db pool exhausted"
if not cache.ping(timeout=0.1): return 200, "degraded: cache down" # NOT 503
if shutting_down: return 503, "draining"
return 200
# The critical judgement: a *shared* dependency being down must NOT make every
# instance report unready. If Redis dies and all 50 instances return 503, the
# load balancer has no healthy backends and takes down a service that could
# still have served from the database. Degrade, do not de-register.Say these points
- Power of two choices is the best general default: near-optimal, no coordination.
- Least connections is dangerous alone - a fast-failing backend attracts all traffic.
- Outlier ejection on error rate is what protects you from that.
- Separate liveness from readiness; never let a shared dependency mark the whole fleet unready.
- Drain connections before terminating, or expect 502s on every deploy.
Avoid these mistakes
- TCP-only health checks that cannot detect a deadlocked process.
- Deep health checks that cascade a dependency outage into a total outage.
- Least connections without error-rate-based ejection.
- Terminating instances without a drain period.
Expect these follow-ups
- Your health check calls the database. The database has a brief hiccup. What happens?
- How do retries interact with load balancing during a partial outage?
- How would you load balance gRPC, where one connection carries many requests?
30-second answer
Retry amplification. When a service slows, callers time out and retry, which multiplies the load on an already-struggling service. With three layers each retrying three times, one user request becomes 27 backend requests exactly when the backend can handle fewest. The service gets slower, more retries fire, and the system collapses. Fixes are retry budgets rather than fixed retry counts, exponential backoff with jitter, circuit breakers, load shedding at the server, and retrying at only one layer of the stack.
L layers, each retrying r times. Three layers × 2 retries each = 27× load. Retries compound multiplicatively, not additively.
How one slow service becomes an outage
Follow the arrows. Each layer independently doing 'the sensible thing' produces a system-wide failure.
Flow
- 1 user requestGateway
- GatewayService A(×3)
- Service AService B(×9)
- Service BDatabase(×27)
The five fixes, in order of impact
1 · Retry budgets, not retry counts
Cap retries as a fraction of total traffic - e.g. retries may not exceed 10% of requests. Under a broad outage the budget is exhausted immediately and retries stop, which is exactly the behaviour you want. A fixed count of 3 does the opposite: it scales retries up precisely when the system is worst.2 · Retry at one layer only
Decide where retries happen - usually the edge or the client - and make every other layer fail fast. This turns multiplicative amplification into additive.3 · Exponential backoff with full jitter
sleep = random(0, min(cap, base × 2^attempt)). Backoff spreads load over time; jitter desynchronises clients. Backoff without jitter just creates a synchronised retry wave on a slower cadence.4 · Circuit breakers
After N consecutive failures, stop calling for a cooldown and fail immediately. This gives the struggling dependency room to recover instead of holding it underwater.5 · Load shedding at the server
When the inbound queue exceeds a threshold, reject new requests instantly with 503. Rejecting in 1 ms is far better than timing out in 30 s: fast rejection lets callers move on, while slow timeouts hold resources on both sides.
class RetryBudget:
"""Retries may not exceed 'ratio' of total requests over a sliding window.
Under a wide outage the budget empties and retries stop automatically."""
def __init__(self, ratio=0.1, window=10.0):
self.ratio, self.window = ratio, window
self.requests, self.retries = Sliding(window), Sliding(window)
def allow_retry(self):
return self.retries.count() < self.ratio * max(self.requests.count(), 1)
def call_with_retry(fn, budget, max_attempts=3, base=0.05, cap=2.0):
budget.requests.add()
for attempt in range(max_attempts):
try:
return fn()
except Retryable:
if attempt == max_attempts - 1 or not budget.allow_retry():
raise # give up rather than pile on
budget.retries.add()
time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))Say these points
- Retries multiply across layers: 3 layers × 3 attempts = 27× load.
- Retry budgets stop retrying during a broad outage; fixed counts make it worse.
- Retry at one layer, back off exponentially, and always add full jitter.
- Circuit breakers give a failing dependency room to recover.
- Timeouts must shrink with depth - propagate a deadline instead of hard-coding.
Avoid these mistakes
- Retries at every layer of the stack.
- Backoff without jitter, producing synchronised retry waves.
- Deeper timeouts longer than shallower ones.
- Retrying non-idempotent operations.
- Queueing under overload instead of shedding load.
Expect these follow-ups
- How do you choose the timeout for each layer of a five-hop call chain?
- What is the difference between load shedding and rate limiting?
- How does a circuit breaker decide when to close again?
Showing 2 of 2 questions for load-balancing-algorithms.
Check your understanding
4 questions · no sign-up, nothing stored
Hands-on challenge
Build it - this is what you talk about in a deep-dive round.
Make a call chain survive a slow dependency
Take a three-service chain and make it degrade gracefully instead of collapsing when the deepest service slows to 5-second responses.
Requirements
- Assign a timeout to every hop such that deadlines shrink with depth, and justify each number.
- Implement a retry budget and demonstrate it stops retrying during a broad outage.
- Add a circuit breaker with defined open, half-open and closed transitions.
- Add server-side load shedding and show fast 503s instead of slow timeouts under overload.
- Load-test the chain with the dependency artificially slowed and plot amplification with and without your fixes.
Stretch goals
- Add outlier ejection and show a fast-failing backend being removed automatically.
- Propagate a deadline through all three hops and show the deepest call being abandoned early.