Designing for Failure: Bulkheads, Degradation and Chaos
Everything fails eventually. The discipline is deciding in advance how it fails - which features degrade, which stay up, and how much data you are willing to lose.
Covers: Failure modes, bulkhead isolation, graceful degradation, blast radius, chaos engineering, RTO and RPO, disaster recovery
Junior designs assume the happy path and add error handling afterwards. Senior designs start from the failure list and work backwards. In an interview, raising failure modes before you are asked is one of the clearest levelling signals available.
30-second answer
Isolate resources with bulkheads so one slow dependency cannot consume the whole thread or connection pool, and design explicit graceful degradation so the feature that depends on it disappears rather than the page. The classic failure is a shared thread pool: one dependency slows from 50 ms to 5 s, every thread ends up blocked waiting on it, and requests that never touch that dependency start timing out too. Separate pools per dependency, combined with circuit breakers and a defined degraded mode per feature, contain the blast radius.
The bulkhead
The name comes from ships: compartments sealed off so a hull breach floods one section rather than sinking the vessel. In software it means each dependency gets its own bounded resource pool, so exhausting one cannot starve the others.
# ❌ One pool for everything. The recommendation service slows to 5 s;
# all 100 threads end up parked on it, and login requests time out too.
executor = ThreadPoolExecutor(max_workers=100)
# ✅ Bounded, separate pools sized by importance.
pools = {
"auth": ThreadPoolExecutor(max_workers=40), # critical
"orders": ThreadPoolExecutor(max_workers=40), # critical
"recommendations": ThreadPoolExecutor(max_workers=10), # optional
"analytics": ThreadPoolExecutor(max_workers=10), # fire and forget
}
def call(dep, fn, timeout, fallback):
try:
return pools[dep].submit(fn).result(timeout=timeout)
except (TimeoutError, Exception):
return fallback() # degrade this feature, not the request
# Recommendations can now consume at most 10 threads. When it stalls, that
# feature disappears from the page and everything else is unaffected.| Isolation level | Isolates | Cost |
|---|---|---|
| Separate thread/connection pools | Resource exhaustion between dependencies | Some idle capacity per pool |
| Separate processes | Memory leaks, crashes | More overhead |
| Separate instances/clusters | Bad deploys, noisy neighbours | Infrastructure cost |
| Separate availability zones | Zone-level failure | Cross-zone traffic charges |
| Cell-based architecture | Everything - a bug affects one cell | Significant operational complexity |
Graceful degradation: decide the order things fall over
For every feature, define what happens when its dependency is unavailable. Write it down before the incident, because during the incident nobody will agree.
| Feature | Dependency | Degraded behaviour |
|---|---|---|
| Product page | Recommendation service | Hide the carousel; page renders normally |
| Product page | Inventory service | Show 'checking availability'; allow add-to-cart, verify at checkout |
| Product page | Pricing service | Fail - never show a wrong price |
| Checkout | Fraud scoring | Accept below a value threshold, queue for review above it |
| Feed | Personalisation | Serve the popular/chronological fallback |
| Search | Search cluster | Fall back to a database prefix query with a banner |
The interesting rows are the ones that fail closed. Being able to say 'this one degrades, this one must not' is the whole skill.
Chaos engineering
- Start in staging, then production during business hours. Failures at 3 a.m. with a skeleton crew are how you learn nothing.
- Form a hypothesis first. 'Killing one of three replicas should cause zero user-visible errors.' If the outcome surprises you, that is the finding.
- Define abort criteria before you begin, and make stopping the experiment a single command.
- Start small: kill one instance, then a zone, then inject latency, then partition the network. Latency injection tends to find more bugs than instance termination, because timeouts are usually less well tested than restarts.
RTO and RPO
| Strategy | RTO | RPO | Relative cost |
|---|---|---|---|
| Backup and restore | Hours | Hours (since last backup) | Lowest |
| Pilot light (minimal standby) | Tens of minutes | Minutes | Low |
| Warm standby | Minutes | Seconds | Medium |
| Active-active multi-region | Near zero | Near zero | Highest |
RTO = how long until you are back. RPO = how much data you can lose. Both are business decisions before they are engineering ones.
Say these points
- Bulkheads: bounded per-dependency pools so one slow dependency cannot starve the rest.
- Define degraded behaviour per feature in advance, including which must fail closed.
- Fail-open versus fail-closed is a business risk decision - say so.
- Chaos experiments need a hypothesis, business hours and abort criteria.
- RTO/RPO drive the DR strategy; an unrestored backup is unverified.
Avoid these mistakes
- One shared thread pool for all outbound calls.
- No defined degraded mode, so every dependency is implicitly critical.
- Silently choosing fail-open for a security control.
- Chaos testing at night when nobody can respond.
- Backups that have never been restored.
Expect these follow-ups
- Your recommendation service adds 4 s of latency. Trace exactly what happens without bulkheads.
- How would you decide fail-open versus fail-closed for a rate limiter?
- Design a cell-based architecture for a multi-tenant SaaS product.
Showing 1 of 1 questions for designing-for-failure.
Check your understanding
3 questions · no sign-up, nothing stored