Scalability, Availability and Reliability Explained
The three properties every interviewer probes, made precise. How availability maths actually composes, why reliability is not the same thing, and how to spot a single point of failure in any diagram.
Covers: Nines and error budgets, SLA vs SLO vs SLI, MTBF and MTTR, redundancy, single points of failure, fault tolerance
Candidates use scalable, available and reliable interchangeably. They are three different properties with three different failure modes, and an interviewer who hears them conflated will start probing. This lesson makes each precise and shows how they compose - because the arithmetic of composing availability is genuinely counter-intuitive.
30-second answer
Scalability is the ability to handle growth by adding resources, ideally with cost rising linearly rather than exponentially. Availability is the fraction of time the system responds successfully. Reliability is the probability it behaves correctly over a period - an available system that returns wrong answers is not reliable. A system can be highly available and unreliable (always up, silently corrupting data), or reliable but unavailable (perfectly correct when running, down four hours a week).
| Property | Question it answers | Measured by | Bought with |
|---|---|---|---|
| Scalability | Can we handle 10× the load by adding machines? | Cost and latency curves as load grows | Statelessness, partitioning, caching, async work |
| Availability | Is it up right now? | Successful requests ÷ total requests | Redundancy, failover, health checks, graceful degradation |
| Reliability | Does it do the right thing, consistently? | MTBF, error rate, data-correctness incidents | Testing, idempotency, checksums, durable writes, backups |
| Fault tolerance | Does it keep working when a part breaks? | Behaviour under injected failure | Replication, retries, circuit breakers, bulkheads |
Why the distinction matters in an answer
- A cache that serves stale data during a database outage is improving availability at the cost of reliability - and that is often the right call for a feed, and the wrong call for a balance.
- A system that scales by adding servers but shares one database primary is scalable on the web tier and not scalable overall. Scalability is a property of the bottleneck, not the average.
- Retries improve availability for transient faults and destroy reliability if the operation is not idempotent - you have just charged the card twice.
Amdahl's Law: the ceiling on parallel speedup
p is the parallelisable fraction, N the number of workers. As N → ∞, speedup approaches 1/(1−p).
If 5% of your request path is inherently serial - a lock, a single-primary write, a global counter - then no matter how many servers you add, you cannot go more than 20× faster. Finding and shrinking that serial fraction is what real scaling work looks like.
Say these points
- Scalable = handles growth with proportional cost; available = up; reliable = correct.
- Scalability is determined by your worst bottleneck, not your average component.
- Retries trade reliability for availability unless operations are idempotent.
- Amdahl's Law: the serial fraction sets a hard ceiling on parallel speedup.
Avoid these mistakes
- Using the three words interchangeably.
- Claiming a design scales while every request still touches one primary.
- Adding retries without idempotency keys.
- Ignoring that coordination overhead can make added capacity net-negative.
Expect these follow-ups
- Give an example of a system that is highly available but unreliable.
- What is the serial fraction in a typical read path, and how do you shrink it?
- How would you demonstrate that a system is actually scalable rather than just large?
30-second answer
Dependencies in series multiply: 0.999³ ≈ 0.997, so about 99.7%, which is roughly 2.2 hours of downtime a month rather than 43 minutes. Each dependency you add makes things worse. You improve it either by adding redundancy within each service - two independent replicas at 99.9% give 99.9999% for that component - or by removing the dependency from the critical path entirely with caching, defaults, or asynchronous processing.
Series: multiply
| Dependencies in series (each 99.9%) | Combined | Downtime/month |
|---|---|---|
| 1 | 99.9% | 43.8 min |
| 3 | 99.70% | 2.2 hours |
| 10 | 99.00% | 7.3 hours |
| 30 | 97.04% | 21.6 hours |
Parallel: multiply the failures instead
Two independent replicas at 99.9% each: failure probability 0.001 × 0.001 = 10⁻⁶, so availability is 99.9999%. Three replicas give nine nines on paper. On paper is doing a lot of work in that sentence - see the warning below.
Three ways to actually improve it
Remove the dependency from the critical path
The most effective fix and the one candidates mention least. Does the page really need the recommendation service, or can it render with a cached default? A dependency that can fail without failing the request contributes ~0% to your downtime.Add independent redundancy
Multiple instances across multiple availability zones, with health checks and automatic failover. Emphasise independent: different racks, different zones, ideally staggered deploys.Degrade gracefully
Serve stale cache when the source is down. Return the feed without the like counts. Queue the write and confirm later. Partial success beats a 500 for almost every consumer product.
SLA vs SLO vs SLI, and the error budget
| Term | What it is | Audience |
|---|---|---|
| SLI | The measurement: % of requests under 200 ms | Engineers |
| SLO | The internal target: 99.9% of requests under 200 ms | The team |
| SLA | The contract, with penalties: 99.5% or you get credits | Customers, legal |
SLA is always looser than SLO - you leave yourself margin before money is at stake.
The error budget is the inverse of the SLO: a 99.9% target permits 43 minutes of failure a month, and that budget is a resource to spend. Under budget? Ship faster, take risks. Blown the budget? Freeze features and fix reliability. It converts an argument about caution into an arithmetic question, which is why it is such a durable idea.
Say these points
- Series dependencies multiply availability; parallel redundancy multiplies failure probability.
- Three 99.9% dependencies in series ≈ 99.7% - microservices tax availability by default.
- Redundancy only compounds when failures are genuinely independent.
- Removing a dependency from the critical path beats making it more reliable.
- SLI measures, SLO targets, SLA contracts; the error budget makes the trade-off explicit.
Avoid these mistakes
- Assuming replicas fail independently when they share a rack, region or deploy pipeline.
- Adding services to the critical path without recomputing availability.
- Setting an SLA tighter than your SLO.
- Treating every dependency as mandatory when most could degrade gracefully.
Expect these follow-ups
- How do you make a dependency optional without silently serving wrong data?
- Your SLO is 99.9% but you keep exhausting the error budget in week one. What now?
- Why does real-world availability plateau around four nines even with heavy redundancy?
30-second answer
Walk the request path and ask of every component: if exactly this one thing dies, does the system stop? Anything that answers yes is a single point of failure. The classic ones are the database primary, the load balancer itself, a single availability zone, the DNS provider, a shared config or secrets service, and the deployment pipeline. You eliminate them with redundancy plus automatic failover - and you verify the failover actually works, because untested failover is not redundancy.
The audit: five questions per component
- If this instance dies right now, what breaks?
- Is there a second one, and is it in a different failure domain (rack, zone, region, provider)?
- Does failover happen automatically, and how long does it take?
- What is the state of in-flight requests during failover?
- When was this failover last actually tested?
| Common SPOF | Why it hides | Standard mitigation |
|---|---|---|
| Database primary | Everyone remembers replicas, few discuss promotion | Automated failover with a consensus-based promoter; accept a few seconds of write unavailability |
| Load balancer | It is drawn as the thing that provides redundancy | Redundant LBs behind an anycast IP or DNS failover; managed LBs are already multi-AZ |
| Single availability zone | The diagram has three servers - in one zone | Spread across ≥3 AZs; size each to absorb the others' traffic |
| DNS / certificate expiry | Not a server, so not in the diagram | Multiple DNS providers, automated cert renewal with alerting well before expiry |
| Shared config / feature-flag service | It is 'just config' | Cache last-known-good locally; fail open to the cached value, never to an exception |
| Deploy pipeline | It is not in the request path - until a bad push is | Canary and staged rollout, automated rollback on SLO breach |
Before and after: eliminating the SPOFs
Tap the nodes. The left column is a design with four single points of failure; the right shows the standard redundancy applied to each.
Flow
- Single load balancerApp servers, one AZ
- App servers, one AZSingle DB primary
- Redundant LBApp tier × 3 AZ
- App tier × 3 AZPrimary + sync standby
MTBF, MTTR, and where to spend
There are two ways to raise availability: fail less often (raise MTBF) or recover faster (lower MTTR). Beyond a point, MTBF is very expensive to improve - hardware and networks fail, full stop. MTTR is usually the cheaper lever: better alerting, automated rollback, one-click failover, runbooks. A system that fails weekly but recovers in 20 seconds beats one that fails yearly and takes six hours.
Say these points
- Walk the request path and ask 'if only this dies, does the system stop?'
- Redundancy must cross failure domains - three instances in one zone is one failure domain.
- Non-server SPOFs (DNS, certs, config, deploy pipeline) cause many real outages.
- Availability = MTBF / (MTBF + MTTR); lowering MTTR is usually the cheaper lever.
- Untested failover should be assumed broken.
Avoid these mistakes
- Calling a design redundant when everything lives in one availability zone.
- Forgetting the load balancer and DNS in the SPOF audit.
- Ignoring what happens to in-flight requests during a failover.
- Never testing failover, then discovering the standby was misconfigured during an incident.
Expect these follow-ups
- Your standby has been replicating for a year but has never been promoted. What could be wrong with it?
- How do you make a config service fail open safely?
- What is an acceptable RTO and RPO for a payments system, and how does that change the design?
Showing 3 of 3 questions for scalability-availability-and-reliability.
Check your understanding
4 questions · no sign-up, nothing stored
Hands-on challenge
Build it - this is what you talk about in a deep-dive round.
Run a SPOF audit and set an error budget
Take any architecture diagram you have (or the one from lesson 1) and produce a reliability review a senior engineer would sign off.
Requirements
- List every component and mark each as SPOF / redundant-same-domain / redundant-cross-domain.
- Compute end-to-end availability assuming each component's individual availability, treating the critical path as a series chain.
- Identify the two dependencies you could remove from the critical path, and say how.
- Define one SLI, one SLO and the resulting monthly error budget in minutes.
- Write the failover test you would run as a game day, including the abort criteria.
Stretch goals
- Recompute availability after your two critical-path removals and quantify the improvement.
- Identify a correlated-failure risk that the independence assumption hides.