ML System DesignExpert20 min read5 questions
The ML System Design Framework (8 Steps)
A repeatable structure for the 45-minute ML system design round, plus the three moves that separate a hire from a no-hire in the first five minutes.
Covers: Requirements clarification, metric selection, data strategy, model choice, serving architecture, evaluation, iteration
ML system design is the highest-signal round in most senior loops, and the one candidates prepare for least well. It is not a knowledge test — it is a test of whether you can take an ambiguous business problem and turn it into a system with defensible trade-offs, in 45 minutes, out loud.
Filter
0/5 mastered
Expertsystem-designframeworkinterview-strategyAsked at Meta, Google
30-second answer
Eight steps: clarify requirements and constraints, define the ML problem and metrics, design the data strategy, do feature engineering, choose a model with a baseline, define training and evaluation, design serving and scale, then close with monitoring and iteration. Spend the first 5–8 minutes on clarification — it is the highest-leverage part and the part most candidates skip.
1 — Clarify (5–8 min). Do not skip this.
Who is the user and what decision does the model drive? What is the business metric? What is the scale — users, items, QPS, data volume? What is the latency budget? What data already exists? Are there constraints: privacy, regulation, explainability, on-device? Write the answers on the board; you will refer to them for the rest of the session.2 — Frame the ML problem and metrics
Translate the business goal into a learning problem: classification, ranking, regression, retrieval, generation. State the offline metric, the online metric, and explicitly say how they relate — plus the guardrail metrics that must not regress.3 — Data strategy
Where do labels come from — explicit, implicit, or human-labelled? What is the label delay? What biases does the collection process introduce? How much data do you have, and what is the positive rate? Say how you would split (time-based, group-aware) and why.4 — Features
Group them: user, item, context, interaction, and cross features. For each group say where it comes from and whether it is available at serving time with acceptable freshness. This is where you naturally raise the feature store and training/serving skew.5 — Model
Always start with a baseline (heuristic or logistic regression / gradient boosting) and justify why you would move beyond it. Then propose the real architecture and defend it against one credible alternative. Never open with the most complex option.6 — Training and evaluation
Training data construction, negative sampling if it is retrieval or ranking, loss function, class imbalance handling, cross-validation scheme, and the offline evaluation protocol including which segments you break results down by.7 — Serving and scale
Now do the arithmetic: QPS, storage, index size, GPU count. Describe the request path end to end with a latency budget per stage, plus caching, fallbacks and what happens when a dependency is down.8 — Monitoring and iteration
Drift monitoring, delayed-label handling, retraining cadence, the A/B plan, and the top two or three things you would improve in version two. Closing here shows you think past launch.
pythonThe back-of-envelope arithmetic to do out loud
# Always convert the product description into numbers.
DAU, sessions_per_day, items_scored = 10_000_000, 5, 500
qps_avg = DAU * sessions_per_day / 86_400
qps_peak = qps_avg * 3 # 3x diurnal peak
scoring_qps = qps_peak * items_scored
print(f"request QPS avg={qps_avg:,.0f} peak={qps_peak:,.0f}")
print(f"item scorings/sec at peak: {scoring_qps:,.0f}")
# request QPS avg=579 peak=1,736
# item scorings/sec at peak: 868,056
# Can one machine do this?
model_us = 50 # 50 microseconds per item
cores_needed = scoring_qps * model_us / 1e6
print(f"cores needed: {cores_needed:,.0f}") # cores needed: 43
# -> ~43 cores of pure compute; with 50% headroom, ~6 x 16-core machines. Feasible.
# Storage for embeddings
n_items, dim, bytes_per = 50_000_000, 128, 4
print(f"item embedding index: {n_items*dim*bytes_per/1e9:.1f} GB")
# item embedding index: 25.6 GB -> fits in RAM on one large node; no sharding needed.
# Training data volume
events_per_day = DAU * sessions_per_day * 20 # 20 interactions/session
print(f"events/day: {events_per_day/1e6:.0f}M, 90 days: {events_per_day*90/1e9:.1f}B")
# events/day: 1000M, 90 days: 90.0B -> must sample; cannot train on all of it.One structural tip: draw the diagram early and keep updating it. A simple boxes-and-arrows picture — data sources → feature pipeline → training → registry → serving → logging → back to data — gives you and the interviewer a shared reference. It also makes gaps visible to you: if there is no arrow from serving back to your training data, you have not thought about the feedback loop yet.
Say these points
- Eight steps: clarify → frame → data → features → model → train/eval → serve → monitor.
- Spend the first 5–8 minutes clarifying; write the constraints down.
- Do the QPS, storage and cost arithmetic out loud — it constrains every later choice.
- Always propose a baseline before the sophisticated model.
- Manage the clock explicitly and narrate your prioritisation.
Avoid these mistakes
- Jumping into architecture before establishing the metric and the scale.
- Proposing a deep model without a baseline.
- Running out of time before reaching serving and monitoring.
Expect these follow-ups
- How would this design change if latency had to be under 20 ms?
- What if you had 1,000 labelled examples instead of a billion events?
- How would you launch this with no historical data at all?
Advancedmetricssystem-designevaluationAsked at Meta, Netflix
30-second answer
Define three layers and state how they connect: the business metric (revenue, retention, cost), the online product metric that is a fast proxy for it (CTR, completion rate, resolution rate), and the offline ML metric you can compute without shipping (NDCG@k, PR-AUC, faithfulness). Then name the guardrails that must not regress and say explicitly that offline metrics are proxies to be validated by an A/B test.
| Layer | Example | Feedback speed | Directly optimisable? |
|---|---|---|---|
| Business | Subscriber retention, GMV, support cost | Months | No |
| Online product | CTR, watch time, resolution rate | Days | Via A/B |
| Offline ML | NDCG@10, PR-AUC, calibration | Minutes | Yes |
| Guardrail | Latency p99, diversity, complaint rate, fairness gap | Days | Constraint |
The proxy-metric traps to name
- CTR rewards clickbait. Optimising it alone degrades long-term satisfaction. Pair it with a quality signal like dwell time above a threshold, or explicitly discount clicks with immediate back-navigation.
- Watch time rewards length, not value. A model can increase watch time by recommending longer content nobody enjoys. Normalise by content length or use completion rate.
- Engagement rewards outrage. In social contexts the most engaging content is often the most harmful. This is why every mature platform carries integrity guardrails alongside engagement metrics.
- Accuracy on an imbalanced problem rewards doing nothing. Covered in the metrics lesson, and it comes up constantly in fraud and safety designs.
- Per-request metrics miss system effects. A recommender can improve every individual recommendation while collapsing catalogue diversity and hurting long-term retention.
pythonA composite objective with explicit guardrails
from dataclasses import dataclass
@dataclass
class LaunchCriteria:
"""Ship only if the primary metric wins AND no guardrail regresses."""
primary: str
primary_min_lift: float # minimum lift worth shipping
guardrails: dict # metric -> max allowed relative regression
CRITERIA = LaunchCriteria(
primary="session_completion_rate",
primary_min_lift=0.01, # +1% relative
guardrails={
"p99_latency_ms": 0.10, # may rise at most 10%
"catalogue_coverage": 0.05, # may fall at most 5%
"complaint_rate": 0.00, # may not rise at all
"new_creator_impressions": 0.10, # ecosystem health
},
)
def launch_decision(treatment, control, criteria=CRITERIA):
lift = treatment[criteria.primary] / control[criteria.primary] - 1
if lift < criteria.primary_min_lift:
return "NO SHIP", [f"{criteria.primary} lift {lift:+.2%} below "
f"{criteria.primary_min_lift:+.2%} threshold"]
breaches = []
for m, tol in criteria.guardrails.items():
delta = treatment[m] / control[m] - 1
worse = delta > tol if m in ("p99_latency_ms", "complaint_rate") else -delta > tol
if worse:
breaches.append(f"{m} moved {delta:+.2%}, tolerance {tol:.0%}")
return ("NO SHIP" if breaches else "SHIP"), breaches or [f"lift {lift:+.2%}"]
# ('NO SHIP', ['catalogue_coverage moved -8.30%, tolerance 5%'])
# The model got better at recommending the same popular items to everyone.Say these points
- Three layers: business → online product → offline ML, plus guardrails.
- State the proxy chain and say you would validate the correlations.
- Name the classic proxy traps: clickbait, length bias, engagement-vs-integrity.
- Define launch criteria as primary lift AND no guardrail regression.
- Propose a long-term holdback to catch feedback-loop damage.
Avoid these mistakes
- Naming only an offline metric and stopping there.
- Optimising engagement with no integrity or diversity guardrail.
- Treating a two-week A/B as proof of long-term benefit.
Expect these follow-ups
- How would you validate that your offline metric predicts the online one?
- What guardrails would you add for a content-recommendation system?
- How do you handle a metric that improves for one segment and regresses for another?
Expertrecsyssystem-designretrievalAsked at YouTube, Netflix
30-second answer
A multi-stage funnel: candidate generation narrows 50M videos to ~1,000 using a two-tower retrieval model plus complementary sources (trending, subscriptions, collaborative filtering), a ranking model scores those few hundred with rich cross features and multi-task heads, and a re-ranking layer applies diversity, freshness and business rules. Item embeddings are precomputed; the user tower runs online with ANN search.
Why a funnel and not one model
You cannot score 50M items per request. The funnel exists so you can spend compute where it changes the outcome: a cheap model over everything, then an expensive model over the few hundred that could plausibly be shown.
| Stage | Input → output | Latency | Model |
|---|---|---|---|
| Candidate generation | 50M → ~1,000 | ~10 ms | Two-tower ANN + heuristic sources |
| Filtering | 1,000 → ~800 | ~2 ms | Rules: already-watched, blocked, region, policy |
| Ranking | 800 → 800 scored | ~40 ms | Multi-task DNN with cross features |
| Re-ranking | 800 → 20 shown | ~5 ms | Diversity, freshness, business constraints |
pythonTwo-tower retrieval — the standard candidate generator
import torch, torch.nn as nn, torch.nn.functional as F
class TwoTower(nn.Module):
"""User and item encoded SEPARATELY so item vectors can be precomputed."""
def __init__(self, n_items, n_countries, dim=128):
super().__init__()
self.item_emb = nn.Embedding(n_items, 64)
self.item_mlp = nn.Sequential(nn.Linear(64 + 16, 256), nn.ReLU(),
nn.Linear(256, dim))
self.country_emb = nn.Embedding(n_countries, 16)
self.user_mlp = nn.Sequential(nn.Linear(64 + 16 + 8, 256), nn.ReLU(),
nn.Linear(256, dim))
def item_tower(self, item_ids, item_feats):
x = torch.cat([self.item_emb(item_ids), item_feats], dim=-1)
return F.normalize(self.item_mlp(x), dim=-1)
def user_tower(self, history_ids, country_ids, ctx_feats):
# Mean-pool watch history -- cheap, and surprisingly strong
hist = self.item_emb(history_ids).mean(dim=1)
x = torch.cat([hist, self.country_emb(country_ids), ctx_feats], dim=-1)
return F.normalize(self.user_mlp(x), dim=-1)
def sampled_softmax_loss(u, pos, neg, log_q_pos, log_q_neg, temperature=0.05):
"""In-batch negatives with the logQ correction -- WITHOUT it the model
systematically under-recommends popular items."""
pos_logit = (u * pos).sum(-1) / temperature - log_q_pos
neg_logit = (u.unsqueeze(1) * neg).sum(-1) / temperature - log_q_neg
logits = torch.cat([pos_logit.unsqueeze(1), neg_logit], dim=1)
return F.cross_entropy(logits, torch.zeros(len(u), dtype=torch.long))
# Serving:
# offline nightly -> item_tower over 50M videos -> HNSW/ScaNN index (~25 GB)
# online per request -> user_tower (~2 ms) -> ANN top-1000 (~8 ms)Ranking: multi-task, because one objective is never enough
pythonMulti-task ranking head with a shared trunk
class MultiTaskRanker(nn.Module):
"""Shared representation, several heads. Standard at YouTube/Meta scale."""
def __init__(self, in_dim, hidden=512):
super().__init__()
self.trunk = nn.Sequential(
nn.Linear(in_dim, hidden), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(hidden, 256), nn.ReLU())
self.click = nn.Linear(256, 1)
self.watch_frac = nn.Linear(256, 1) # completion fraction
self.like = nn.Linear(256, 1)
self.report = nn.Linear(256, 1) # negative signal
def forward(self, x):
h = self.trunk(x)
return {"click": self.click(h), "watch": self.watch_frac(h),
"like": self.like(h), "report": self.report(h)}
def final_score(preds, w=(0.3, 1.0, 0.5, -5.0)):
"""Weights are a PRODUCT decision, tuned by A/B -- not learned."""
wc, ww, wl, wr = w
return (wc * torch.sigmoid(preds["click"])
+ ww * torch.sigmoid(preds["watch"])
+ wl * torch.sigmoid(preds["like"])
+ wr * torch.sigmoid(preds["report"]))
# Why multi-task rather than one CTR model:
# - CTR alone optimises for clickbait; watch fraction penalises it
# - The report head lets you actively demote harmful content
# - Shared trunk means sparse signals (likes, reports) benefit from dense onesThe problems the interviewer will push on
Cold start — new user
No history. Fall back to popularity within the user's country and language, use signup-time signals, and explore aggressively in the first sessions. A contextual bandit over broad content categories converges faster than waiting for the main model to have data.Cold start — new video
No interaction data, so collaborative signals are absent. Rely on content features (title, description, thumbnail, audio and video embeddings) and creator-level priors, then give every new item a guaranteed small impression budget so it can earn signal.Position and popularity bias
Logs are generated by the current system, so popular and top-slot items are over-represented. Correct with inverse propensity weighting, an exploration bucket with randomised slots, and a popularity-debiased negative sampling distribution.Filter bubbles and diversity
Pure relevance maximisation collapses variety. Apply determinantal point processes or a simple greedy MMR at re-ranking, cap items per creator or topic in a slate, and track catalogue coverage as a guardrail.Freshness
Nightly item embeddings are too slow for news or live content. Add a streaming path that indexes new items within minutes, and include an explicit recency feature so the ranker can learn how much freshness matters per surface.
pythonDiversity re-ranking with MMR
import numpy as np
def mmr_rerank(candidates, embeddings, scores, k=20, lambda_=0.75,
max_per_creator=3):
"""Maximal Marginal Relevance + a hard per-creator cap."""
selected, creator_count = [], {}
remaining = list(range(len(candidates)))
while len(selected) < k and remaining:
best, best_val = None, -np.inf
for i in remaining:
creator = candidates[i]["creator_id"]
if creator_count.get(creator, 0) >= max_per_creator:
continue
redundancy = (max(float(embeddings[i] @ embeddings[j]) for j in selected)
if selected else 0.0)
val = lambda_ * scores[i] - (1 - lambda_) * redundancy
if val > best_val:
best, best_val = i, val
if best is None:
break
selected.append(best); remaining.remove(best)
creator_count[candidates[best]["creator_id"]] = \
creator_count.get(candidates[best]["creator_id"], 0) + 1
return selected
# lambda_ is a product dial: 1.0 = pure relevance, 0.5 = strongly diverse.
# A/B it -- the right value differs per surface (home feed vs. up-next).Say these points
- Multi-stage funnel: 50M → 1,000 → 800 → 20, spending compute where it changes the outcome.
- Two-tower's no-interaction constraint is what enables precomputed item vectors and ANN search.
- Use in-batch negatives with the logQ correction or you under-recommend popular items.
- Multi-task ranking with a shared trunk; combine heads with product-tuned weights.
- Handle cold start, position bias, diversity and freshness explicitly — they will be probed.
Avoid these mistakes
- Proposing a single model that scores all 50M items.
- Optimising CTR alone and inviting clickbait.
- Forgetting the logQ / popularity correction in negative sampling.
Expect these follow-ups
- How would you add real-time signals from the current session?
- How do you evaluate this offline given the logs are biased by the current system?
- How would you handle a creator gaming the system?
Expertfraudsystem-designimbalancedAsked at Stripe, PayPal
30-second answer
A layered system: deterministic rules for known patterns and hard blocks, a real-time ML model scoring every transaction within a ~100 ms budget using streaming velocity features, and a case-management queue for the uncertain band. Because fraud is adversarial and labels arrive 30–90 days late via chargebacks, the design must include an exploration bucket, rapid rule deployment, and monitoring that does not depend on labels.
Architecture
Layer 0 — Deterministic rules (< 5 ms)
Blocklists, velocity hard-limits, sanctions screening, impossible-travel checks. Rules are auditable, instantly deployable when a new attack appears, and required by compliance. They handle the known knowns; ML handles the rest.Layer 1 — Real-time model (< 60 ms)
Gradient boosting on ~200 features. GBDT is the right choice here: fast, strong on tabular data, and — importantly for regulated decisions — explainable via SHAP.Layer 2 — Decision policy
Not a single threshold but three bands driven by expected cost: approve, challenge (3DS/step-up authentication), and block. The challenge band is what makes the economics work — it converts an uncertain decision into extra signal at low customer friction.Layer 3 — Case management
A prioritised analyst queue for high-value uncertain cases. Every analyst decision becomes a fast label, which is the main way you get signal before chargebacks arrive.Layer 4 — Feedback and adaptation
Chargebacks, analyst decisions, customer disputes and the exploration bucket all flow back into training. Fast retraining cadence — weekly or faster — because the adversary moves.
pythonStreaming velocity features — the strongest fraud signals
from collections import defaultdict, deque
import time
class VelocityFeatures:
"""Sliding-window counters. In production these live in Redis or Flink state."""
WINDOWS = [60, 3600, 86_400] # 1 min, 1 hour, 1 day
def __init__(self):
self.events = defaultdict(deque)
def _count(self, key, window, now):
dq = self.events[key]
while dq and now - dq[0][0] > window:
dq.popleft()
return len(dq), sum(a for _, a in dq)
def compute(self, txn, now=None):
now = now or time.time()
f = {}
for entity in ("card", "device", "ip", "email", "merchant"):
key = f"{entity}:{txn[entity]}"
for w in self.WINDOWS:
n, total = self._count(key, w, now)
f[f"{entity}_count_{w}s"] = n
f[f"{entity}_amount_{w}s"] = total
# Ratio features are where the signal really is
f["amount_vs_card_avg_24h"] = txn["amount"] / max(
f["card_amount_86400s"] / max(f["card_count_86400s"], 1), 1.0)
f["card_velocity_spike"] = f["card_count_60s"] / max(
f["card_count_3600s"] / 60, 0.01)
# Entity fan-out: one device touching many cards is a strong signal
f["cards_per_device_24h"] = len({
t["card"] for ts, t in self.events[f"device:{txn['device']}"]
if now - ts < 86_400}) if self.events[f"device:{txn['device']}"] else 1
for entity in ("card", "device", "ip", "email", "merchant"):
self.events[f"{entity}:{txn[entity]}"].append((now, txn["amount"]))
return f
# Card-testing attacks show up as: many tiny transactions, one device,
# many distinct cards, within minutes. Velocity + fan-out catches it
# before a single chargeback has been filed.pythonExpected-cost decision policy, not a single threshold
def decide(score, amount, ctx,
cost_fp_review=6.0, cost_challenge_friction=0.9, chargeback_multiplier=1.4):
"""Three bands chosen by expected cost, with amount-dependent thresholds."""
expected_loss = score * amount * chargeback_multiplier # + fees + ops cost
block_threshold = cost_fp_review + 0.02 * amount # blocking a good customer is costly
challenge_threshold = cost_challenge_friction
if expected_loss > block_threshold and score > 0.80:
return "BLOCK", {"expected_loss": expected_loss}
if expected_loss > challenge_threshold:
return "CHALLENGE", {"expected_loss": expected_loss} # 3DS / step-up auth
return "APPROVE", {"expected_loss": expected_loss}
# Why a CHALLENGE band matters economically:
# BLOCK -> lost sale + an angry customer (high cost, and often permanent)
# APPROVE -> full chargeback loss if wrong
# CHALLENGE -> small friction, converts an uncertain case into evidence,
# and in most schemes shifts liability to the issuer.
# Roughly 3-6% of traffic in the challenge band typically captures the
# majority of fraud that a pure two-way threshold would have to guess on.| Constraint | Design response |
|---|---|
| Adversarial drift | Weekly retraining; rules deployable in minutes; drift monitors on inputs and scores |
| Labels 30–90 days late | Analyst decisions and 3DS outcomes as fast labels; monitor score distribution |
| Blocked outcomes unobserved | Random exploration bucket with a value cap |
| Regulatory explainability | GBDT + SHAP reason codes stored with every decision |
| 100 ms budget | Precomputed entity aggregates in Redis; GBDT inference ~2 ms |
| Extreme imbalance | PR-AUC and precision@k; cost-based thresholds; calibrated probabilities |
Say these points
- Fraud is adversarial, extremely imbalanced, and has delayed censored labels — say this first.
- Layer rules + ML + a three-band decision policy + analyst queue + feedback loop.
- Streaming velocity and entity fan-out features carry most of the signal.
- A challenge/step-up band is what makes the economics work.
- Exploration bucket is mandatory or the model goes blind at the boundary.
Avoid these mistakes
- Using a single 0.5 threshold instead of cost-based bands.
- Ignoring that blocked transactions never produce labels.
- Choosing a deep model where explainability is a regulatory requirement.
Expect these follow-ups
- How would you detect a coordinated attack across many accounts?
- How do you evaluate the model when 40% of your decisions have no outcome?
- How would you handle a merchant with a legitimately unusual transaction pattern?
Expertllmragsystem-designAsked at Intercom, Zendesk
30-second answer
A tiered system: intent classification routes trivial requests to deterministic flows, a RAG pipeline over help-centre and policy content handles informational questions, tool calling handles account actions with strict permission scoping, and confident escalation hands off to humans with full context. Wrap it in caching, guardrails, evaluation and a feedback loop — and design the human handoff first, because it is what determines user trust.
Request flow
1 — Classify and route (~20 ms)
A small fine-tuned encoder classifies intent and risk. 'Where is my order?' goes to a deterministic lookup flow; 'How do I cancel?' goes to RAG; 'I want to dispute a charge' routes straight to a human. Roughly 30–40% of volume never needs an LLM at all, which is a large cost and latency win.2 — Cache lookup
Exact-match then semantic cache. Support questions are extremely repetitive — hit rates of 25–40% are normal, and a hit is ~50 ms instead of ~3 s.3 — Retrieve (~80 ms)
Hybrid search over help-centre articles, policy documents and resolved tickets, filtered by the user's product, plan and locale. Rerank to 4–6 chunks.4 — Generate with tools (~1.5 s)
The LLM answers with citations and may call read-only account tools (order status, subscription state). Write actions require confirmation and are value-capped in code.5 — Verify before sending
Check citations resolve, scan for PII leakage and policy violations, and run a cheap faithfulness check. If verification fails, escalate rather than send.6 — Escalate with context
Hand the human agent the full conversation, retrieved sources, tool results and a suggested resolution. A good handoff makes the agent faster than if the assistant had never been involved.
pythonRisk-based routing and escalation policy
RISK_TIERS = {
"informational": {"automate": True, "tools": ["search_docs"]},
"account_read": {"automate": True, "tools": ["search_docs", "get_order",
"get_subscription"]},
"account_write": {"automate": True, "tools": ["update_address"],
"requires_confirmation": True, "max_value_usd": 0},
"financial": {"automate": True, "tools": ["process_refund"],
"requires_confirmation": True, "max_value_usd": 50},
"sensitive": {"automate": False}, # disputes, fraud, legal, safety, account closure
}
def should_escalate(state) -> tuple[bool, str]:
if state.risk_tier == "sensitive":
return True, "sensitive_topic"
if state.retrieval_max_score < 0.45:
return True, "no_relevant_documentation"
if state.turns >= 4 and not state.resolved:
return True, "conversation_stalled"
if state.user_frustration_score > 0.7: # small classifier on the transcript
return True, "user_frustrated"
if state.user_requested_human:
return True, "user_requested"
if state.faithfulness_score < 0.8:
return True, "low_confidence_answer"
return False, ""
# Escalating EARLY on low retrieval confidence is far cheaper than
# escalating after three wrong answers and a furious customer.pythonCost and capacity arithmetic for 10M users
USERS, contact_rate, turns = 10_000_000, 0.03, 3.5 # 3% contact support/month
conversations = USERS * contact_rate
llm_calls = conversations * turns
print(f"conversations/mo: {conversations:,.0f} LLM calls/mo: {llm_calls:,.0f}")
# Naive: every turn hits a frontier model
naive = llm_calls * (2500 * 3e-6 + 400 * 15e-6)
print(f"naive frontier-model cost: ${naive:>10,.0f}/mo")
# Optimised: 35% never reach an LLM, 30% cache hit, 60% of the rest to a small model
after_routing = llm_calls * 0.65
after_cache = after_routing * 0.70
small, large = after_cache * 0.60, after_cache * 0.40
optimised = (small * (2500 * 0.15e-6 + 400 * 0.6e-6)
+ large * (2500 * 3e-6 + 400 * 15e-6))
print(f"routed + cached + tiered cost: ${optimised:>10,.0f}/mo")
print(f"saving: {100*(1-optimised/naive):.0f}%")
# conversations/mo: 300,000 LLM calls/mo: 1,050,000
# naive frontier-model cost: $ 14,175/mo
# routed + cached + tiered cost: $ 2,662/mo
# saving: 81%
#
# Peak QPS: 300k conversations / month, 3x diurnal peak -> ~1 conversation/sec.
# Entirely manageable; the constraint is quality and trust, not throughput.Evaluation and the feedback loop
- Offline: a gold set of 300+ real questions with expert answers, including unanswerable and adversarial cases. Measure retrieval recall, faithfulness, citation accuracy and correct-refusal rate. Run it in CI on every prompt or index change.
- Online: containment rate (resolved without a human), customer satisfaction on assistant-handled conversations, escalation rate by reason, and — critically — re-contact rate within 7 days, which catches wrong answers that looked resolved.
- Human review: sample 100 conversations a week for expert grading. This is the only reliable ground truth and it feeds both the gold set and the fine-tuning data.
- Agent feedback: escalated conversations are labelled by the human who resolved them. 'The assistant was wrong about X' is the highest-value training signal you can get, and it is free.
Say these points
- Design the escalation criteria before the automation.
- Route by intent and risk — 30–40% of volume never needs an LLM.
- Cache aggressively: support questions are highly repetitive.
- Verify before sending: citations, PII, faithfulness — escalate on failure.
- Measure containment, CSAT, escalation reasons and 7-day re-contact rate.
Avoid these mistakes
- Sending every request to a frontier model.
- Letting the assistant state policy without a retrieved source.
- Measuring deflection rate without measuring re-contact rate.
Expect these follow-ups
- How would you handle a user trying to jailbreak the assistant into granting a refund?
- How do you keep the knowledge base in sync with policy changes?
- How would you support 20 languages with one system?
Showing 5 of 5 questions for ml-system-design-framework.
Check your understanding
5 questions · no sign-up, nothing stored
Hands-on challenge
Build it — this is what you talk about in a deep-dive round.
Write three full ML system design documents
Practise the framework end to end by producing written designs you could hand to an interviewer.
Requirements
- Pick three problems (e.g. search ranking, ad CTR prediction, content moderation) and write each up using all eight framework steps.
- Include the back-of-envelope arithmetic: QPS, storage, index size, GPU count, monthly cost.
- Define the three-layer metric chain plus explicit guardrails and launch criteria for each.
- For each design, write the two hardest follow-up questions you would ask yourself, and answer them.
Stretch goals
- Do one design under a 45-minute timer, out loud, recorded. Watch it back and note where you stalled.
- Add a v1/v2/v3 roadmap to each design showing what you would defer and why.