MLOps & Production MLAdvanced18 min read5 questions
Monitoring, Drift Detection & Incident Response
Models decay silently. Learn what to monitor when labels arrive months late, how to distinguish drift types, and how to run an ML incident.
Covers: Data drift vs concept drift, PSI and KS tests, delayed labels, retraining triggers, ML incident response
A deployed model is a depreciating asset. The world moves, your inputs change, user behaviour adapts to your own predictions — and unlike a crashed service, a degrading model returns 200s the whole way down.
Filter
0/5 mastered
AdvanceddriftmonitoringdegradationAsked at Stripe, Meta Integrity
30-second answer
Data drift is a change in P(X) — the inputs shift but the input→output relationship holds. Concept drift is a change in P(Y|X) — the same inputs now imply a different outcome. Label drift is a change in P(Y), the base rate. Concept drift is the most dangerous because it is invisible in input monitoring: your features look perfectly normal while the model is systematically wrong.
| Type | What changes | Example | Detectable without labels? |
|---|---|---|---|
| Data / covariate drift | P(X) | A marketing campaign brings younger users | Yes — monitor feature distributions |
| Concept drift | P(Y|X) | Fraudsters change tactics; the same signals now mean something else | No — needs labels or a proxy |
| Label / prior drift | P(Y) | Fraud rate rises from 0.1% to 0.4% | Partially — prediction distribution shifts |
| Upstream data drift | Schema or semantics of a source | A vendor changes an enum's meaning | Yes — schema and cardinality checks |
pythonPSI and KS for input drift, with interpretation thresholds
import numpy as np
from scipy import stats
def population_stability_index(reference, current, bins=10):
"""PSI < 0.1 stable | 0.1-0.25 moderate | > 0.25 significant shift."""
edges = np.percentile(reference, np.linspace(0, 100, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
ref_pct = np.histogram(reference, edges)[0] / len(reference)
cur_pct = np.histogram(current, edges)[0] / len(current)
eps = 1e-6
ref_pct, cur_pct = np.clip(ref_pct, eps, None), np.clip(cur_pct, eps, None)
return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))
def drift_report(reference_df, current_df, features):
rows = []
for f in features:
r, c = reference_df[f].dropna(), current_df[f].dropna()
if np.issubdtype(r.dtype, np.number):
psi = population_stability_index(r, c)
ks, p = stats.ks_2samp(r, c)
severity = ("SIGNIFICANT" if psi > 0.25 else
"moderate" if psi > 0.1 else "stable")
rows.append((f, round(psi, 4), round(float(ks), 4), severity))
else:
# Categorical: chi-square on the shared category set + new-category check
new_cats = set(c.unique()) - set(r.unique())
rows.append((f, None, None,
f"NEW CATEGORIES: {new_cats}" if new_cats else "stable"))
return sorted(rows, key=lambda t: -(t[1] or 0))
# feature PSI KS severity
# merchant_category 0.412 0.198 SIGNIFICANT <- investigate first
# txn_amount 0.187 0.093 moderate
# device_type -- -- NEW CATEGORIES: {'watchos'}
# account_age_days 0.021 0.014 stableDetecting concept drift without labels
- Prediction distribution shift. If inputs are stable but the predicted positive rate moves 30%, something is wrong even without a single label.
- Confidence collapse. A rising fraction of predictions near the decision boundary means the model is increasingly unsure — often the first symptom of concept drift.
- Proxy outcomes. Fraud labels take 60 days, but chargeback initiations, manual review overrides and customer complaints arrive in days. Correlate those with predictions.
- Analyst agreement rate. In any human-in-the-loop system, the rate at which reviewers agree with the model is a fast, free proxy for accuracy.
- A small fast-labelled sample. Pay to label 200 random cases a week with expedited review. It costs little and gives you a real, if noisy, accuracy estimate weeks before the natural labels land.
pythonPrediction-drift monitoring — works with zero labels
from collections import deque
import numpy as np
class PredictionMonitor:
"""Watches the model's OUTPUT distribution. Needs no ground truth."""
def __init__(self, reference_scores, window=10_000):
self.ref_mean = float(np.mean(reference_scores))
self.ref_pct = np.percentile(reference_scores, [10, 50, 90])
self.ref_positive_rate = float(np.mean(reference_scores > 0.5))
self.window = deque(maxlen=window)
def observe(self, score):
self.window.append(float(score))
def check(self):
if len(self.window) < self.window.maxlen // 2:
return None
cur = np.array(self.window)
alerts = []
if abs(cur.mean() - self.ref_mean) / max(self.ref_mean, 1e-9) > 0.20:
alerts.append(f"mean score {self.ref_mean:.3f} -> {cur.mean():.3f}")
pos = float(np.mean(cur > 0.5))
if abs(pos - self.ref_positive_rate) / max(self.ref_positive_rate, 1e-9) > 0.30:
alerts.append(f"positive rate {self.ref_positive_rate:.3f} -> {pos:.3f}")
# Confidence collapse: more predictions bunching near the boundary
uncertain = float(np.mean((cur > 0.4) & (cur < 0.6)))
if uncertain > 0.35:
alerts.append(f"{uncertain:.0%} of predictions in the uncertain band")
return alerts or NoneSay these points
- Data drift = P(X) changes; concept drift = P(Y|X) changes; label drift = P(Y) changes.
- Concept drift is most dangerous because input monitoring stays green.
- PSI measures effect size and is the operational alert; KS p-values fire constantly at scale.
- Without labels, monitor prediction distribution, confidence collapse and proxy outcomes.
- Adversarial domains guarantee concept drift by design.
Avoid these mistakes
- Alerting on KS p-values at high volume and drowning in noise.
- Monitoring only inputs and missing concept drift entirely.
- Comparing against a reference window that itself drifts.
Expect these follow-ups
- How would you detect drift in a text or embedding feature?
- What reference window do you compare against, and how often do you refresh it?
- How do you distinguish drift from a bug in an upstream pipeline?
AdvancedretrainingmonitoringautomationAsked at Uber, Stripe
30-second answer
Scheduled retraining is simple and predictable but wastes compute when nothing changed and reacts too slowly when something did. Trigger-based retrains on a signal — drift above threshold, performance below floor, or enough new labels — which is more efficient but needs reliable signals. Continuous/online learning adapts fastest but is hard to validate and easy to poison. Most production systems use scheduled retraining with a trigger override and a mandatory offline gate before any deploy.
| Strategy | Pros | Cons | Fits |
|---|---|---|---|
| Scheduled (weekly/monthly) | Predictable, simple, easy to reason about | Wasteful or too slow | Most business models |
| Trigger-based | Retrains exactly when needed | Needs trustworthy signals; can thrash | Volatile domains |
| Continuous / online | Adapts within minutes | Hard to validate; vulnerable to poisoning | Recsys, ads, high-velocity |
| Manual | Full human judgement | Does not scale; gets forgotten | Low-stakes or rarely-changing models |
pythonA trigger policy with cooldown and a mandatory quality gate
from dataclasses import dataclass
from datetime import datetime, timedelta
@dataclass
class RetrainPolicy:
max_age_days: int = 30 # always retrain at least monthly
psi_threshold: float = 0.25
performance_floor: float = 0.78 # AUC on whatever labels we have
min_new_labels: int = 5_000
cooldown_days: int = 3 # prevents thrashing on a noisy signal
def should_retrain(state, policy=RetrainPolicy()):
reasons = []
since_last = (datetime.utcnow() - state.last_trained_at).days
if since_last < policy.cooldown_days:
return False, ["cooldown active"]
if since_last >= policy.max_age_days:
reasons.append(f"model is {since_last}d old")
if state.max_feature_psi > policy.psi_threshold:
reasons.append(f"PSI {state.max_feature_psi:.3f} on {state.worst_feature}")
if state.recent_auc is not None and state.recent_auc < policy.performance_floor:
reasons.append(f"AUC {state.recent_auc:.3f} < floor {policy.performance_floor}")
if state.new_labels_since_train > policy.min_new_labels:
reasons.append(f"{state.new_labels_since_train:,} new labels available")
return bool(reasons), reasons
def retrain_pipeline(state):
trigger, reasons = should_retrain(state)
if not trigger:
return "no_action"
candidate = train_model(data_up_to=datetime.utcnow())
# THE GATE: automatic retraining must never mean automatic deployment.
if candidate.holdout_auc < state.champion.holdout_auc - 0.005:
alert(f"Retrained model is WORSE ({candidate.holdout_auc:.4f} vs "
f"{state.champion.holdout_auc:.4f}). Investigate before shipping. "
f"Trigger reasons: {reasons}")
return "candidate_rejected"
return deploy_with_canary(candidate, reasons=reasons)What data to retrain on
- Sliding window — last N months only. Adapts fastest to concept drift but forgets rare patterns and seasonal behaviour.
- Expanding window — all history. Maximum data but old, stale relationships dilute the recent signal.
- Weighted — all history with exponentially decaying sample weights. Usually the best default: recent data dominates without discarding rare events.
- Seasonal-aware — include the same period from previous years so the model has seen Black Friday before this Black Friday.
pythonTime-decayed sample weighting
import numpy as np
def recency_weights(timestamps, half_life_days=90, now=None):
"""Weight = 0.5 ** (age / half_life). 90 days old -> half the weight."""
now = now or timestamps.max()
age_days = (now - timestamps).dt.total_seconds() / 86_400
return np.power(0.5, age_days / half_life_days)
w = recency_weights(train_df["event_ts"], half_life_days=90)
model.fit(X, y, sample_weight=w)
# Measured on a 2-year fraud dataset (evaluated on the most recent month):
# sliding 90d AUC 0.881 (misses rare seasonal patterns)
# expanding all AUC 0.874 (diluted by stale behaviour)
# weighted hl=90d AUC 0.902 <- best of both
# weighted hl=365d AUC 0.889Say these points
- Most systems: scheduled cadence with trigger overrides and a cooldown.
- Triggers: model age, PSI, performance floor, sufficient new labels.
- Automatic retraining must not imply automatic deployment — keep the gate.
- Time-decayed weighting usually beats both sliding and expanding windows.
- Keep a random exploration bucket or the training distribution collapses onto the model's own decisions.
Avoid these mistakes
- Auto-deploying whatever the nightly retrain produces.
- No cooldown, so a noisy drift signal retrains hourly.
- Training only on data the current model allowed through.
Expect these follow-ups
- How would you validate a retrained model when labels are 60 days delayed?
- How do you size the exploration bucket against its cost?
- What would make you roll back to an older model version?
Advancedincident-responsedebuggingoperationsAsked at Stripe, Netflix
30-second answer
Mitigate first, diagnose second. Check whether the impact is user-visible and, if so, roll back or fall back to a safe default immediately — do not debug in production while users are affected. Then work up the stack: is it the model, the features, the upstream data, or a genuine world change? The most common cause by far is an upstream data pipeline change, not the model.
0 — Stop the bleeding (first 5 minutes)
Assess user impact. If predictions are materially wrong and user-facing, roll back to the previous model version or switch to a rules-based fallback. Rollback is cheap and reversible; a prolonged incident is not. Never debug live while users are affected.1 — Establish the timeline
When exactly did it start? A sharp step change points at a deploy or a schema change; a gradual slope points at genuine drift. Correlate the timestamp against the deploy log for the model, the feature pipeline, and every upstream service. This single check resolves most incidents.2 — Model or input?
Re-score a frozen golden batch of inputs with the current model. If the outputs match the recorded expected values, the model is fine and the inputs changed. If they differ, the artefact or the runtime changed. This bisects the problem in one step.3 — Which feature?
Run per-feature drift against the training reference. Look especially for a spike in null rate, a new category, a unit change (dollars → cents), or a constant value where there should be variance. Rank by PSI and start at the top.4 — Trace upstream
Once you have the feature, find its source. Ask the owning team what changed. Common culprits: a vendor API version bump, a schema migration, a timezone change, a currency change, a backfill that rewrote history, or a job that silently failed and left stale values.5 — Fix, verify, and write it up
Fix the root cause, verify on the golden batch, ramp back carefully with a canary, and write a blameless postmortem whose main output is the monitor that would have caught this sooner.
pythonA triage script that bisects the problem in minutes
import numpy as np, json
def triage(model, golden_batch, live_features, training_reference):
findings = {}
# 1. Is the MODEL still the same? Golden-batch regression.
scores = model.predict_proba(golden_batch["X"])[:, 1]
max_delta = float(np.abs(scores - golden_batch["expected_scores"]).max())
findings["model_changed"] = max_delta > 1e-4
findings["golden_max_delta"] = round(max_delta, 6)
if findings["model_changed"]:
findings["verdict"] = ("MODEL ARTEFACT OR RUNTIME CHANGED - check the "
"deployed model hash and library versions")
return findings
# 2. Model is fine -> the inputs changed. Which feature?
suspects = []
for f in live_features.columns:
live, ref = live_features[f], training_reference[f]
null_now, null_ref = live.isna().mean(), ref.isna().mean()
if null_now > null_ref + 0.05:
suspects.append((f, "NULL SPIKE", round(float(null_now), 3)))
continue
if live.nunique() == 1 and ref.nunique() > 1:
suspects.append((f, "CONSTANT VALUE", live.iloc[0]))
continue
if np.issubdtype(live.dtype, np.number):
ratio = live.mean() / (ref.mean() + 1e-9)
if ratio > 50 or ratio < 0.02:
suspects.append((f, "SCALE CHANGE (units?)", round(float(ratio), 2)))
else:
psi = population_stability_index(ref.dropna(), live.dropna())
if psi > 0.25:
suspects.append((f, "DISTRIBUTION SHIFT", round(psi, 3)))
else:
new = set(live.dropna().unique()) - set(ref.dropna().unique())
if new:
suspects.append((f, "NEW CATEGORIES", sorted(new)[:5]))
findings["suspects"] = suspects
findings["verdict"] = ("INPUT DATA CHANGED - trace the top suspect upstream"
if suspects else "NO OBVIOUS INPUT CAUSE - check traffic mix")
return findings
print(json.dumps(triage(model, golden, live_df, train_ref), indent=2, default=str))
# {"model_changed": false, "golden_max_delta": 0.0,
# "suspects": [["txn_amount", "SCALE CHANGE (units?)", 100.4]],
# "verdict": "INPUT DATA CHANGED - trace the top suspect upstream"}
#
# 100x scale change: an upstream service switched from dollars to cents.| Symptom | Most likely cause | First check |
|---|---|---|
| Sharp step change at a timestamp | A deploy or a schema change | Deploy log across all upstream services |
| All predictions near a single value | A feature became constant or all-null | Per-feature null rate and cardinality |
| Scores shifted by ~100× | A unit change (dollars↔cents, s↔ms) | Feature mean ratio vs reference |
| Gradual drift over weeks | Genuine population or concept drift | PSI trend; consider retraining |
| Only one segment affected | A regional pipeline or locale bug | Segment the drift report |
| Latency spiked with the scores | A fallback path is being hit | Error rates on feature-fetch calls |
Say these points
- Mitigate before diagnosing: roll back or fall back if users are affected.
- Golden-batch re-scoring bisects model-vs-input in one step.
- Most ML incidents are upstream data incidents, not model failures.
- Look for null spikes, constant values, unit changes and new categories before subtle drift.
- The postmortem's output should be a new monitor or data contract.
Avoid these mistakes
- Debugging in production while users are affected.
- Assuming the model is at fault before checking the inputs.
- Closing an incident without adding detection.
Expect these follow-ups
- How would you build a golden-batch regression test into CI?
- How do you page on model degradation without alert fatigue?
- Who owns an ML incident when the root cause is another team's pipeline?
AdvancedlatencyoptimizationdistillationAsked at Meta, Pinterest
30-second answer
Profile first to find where the 800 ms goes — it is frequently feature fetching or serialisation rather than the model. Then work through: batching and async I/O, caching, model-level optimisation (quantisation, ONNX/TensorRT export, operator fusion), architectural changes (distillation to a smaller model, a two-stage cascade), and finally hardware. Distillation and cascades usually give the biggest wins.
pythonProfile before optimising — the step people skip
import time
from contextlib import contextmanager
from collections import defaultdict
TIMINGS = defaultdict(list)
@contextmanager
def timed(stage):
t0 = time.perf_counter()
try:
yield
finally:
TIMINGS[stage].append((time.perf_counter() - t0) * 1000)
def predict(request):
with timed("auth"): user = authenticate(request)
with timed("feature_fetch"): raw = feature_store.get(request.entity_id)
with timed("preprocess"): x = pipeline.transform(raw)
with timed("inference"): score = model.predict_proba([x])[0, 1]
with timed("postprocess"): out = format_response(score)
with timed("logging"): log_prediction(request, x, score)
return out
import numpy as np
for stage, ms in TIMINGS.items():
print(f"{stage:16s} p50={np.percentile(ms,50):6.1f} p99={np.percentile(ms,99):7.1f}")
# auth p50= 2.1 p99= 4.0
# feature_fetch p50= 38.0 p99= 610.0 <- 76% of p99. HERE is the problem.
# preprocess p50= 4.2 p99= 9.1
# inference p50= 11.0 p99= 18.3 <- the model is NOT the bottleneck
# postprocess p50= 1.0 p99= 2.2
# logging p50= 9.0 p99= 140.0 <- SYNCHRONOUS logging. Make it async.| Technique | Typical speedup | Accuracy cost | Effort |
|---|---|---|---|
| Async / parallel feature fetch | 2–5× on I/O-bound paths | None | Low |
| Move logging off the request path | Removes a tail spike | None | Low |
| Caching (features or predictions) | 10–100× on hits | Staleness | Low |
| Dynamic batching | 3–10× throughput | Adds a few ms latency | Medium |
| ONNX / TensorRT export | 2–5× | None (fp32) / minimal | Medium |
| Quantisation (int8) | 2–4× | 0.5–2% | Medium |
| Distillation to a smaller model | 5–50× | 1–5% | High |
| Cascade (cheap model first) | 5–20× average | Tunable | Medium |
pythonThe cascade pattern — usually the biggest architectural win
class CascadeClassifier:
"""A cheap model handles the easy majority; the expensive one sees the rest."""
def __init__(self, fast_model, slow_model, low=0.05, high=0.95):
self.fast, self.slow = fast_model, slow_model
self.low, self.high = low, high
self.stats = {"fast_only": 0, "escalated": 0}
def predict(self, x):
p = self.fast.predict_proba([x])[0, 1] # ~2 ms gradient boosting
if p < self.low or p > self.high: # confident -> done
self.stats["fast_only"] += 1
return p
self.stats["escalated"] += 1
return self.slow.predict_proba([x])[0, 1] # ~180 ms transformer
def report(self):
n = sum(self.stats.values())
esc = self.stats["escalated"] / n
avg_ms = 2 + esc * 180
return {"escalation_rate": round(esc, 3), "avg_latency_ms": round(avg_ms, 1)}
# {'escalation_rate': 0.118, 'avg_latency_ms': 23.2}
#
# 88% of traffic never touches the expensive model. Average latency drops from
# 180ms to 23ms and cost drops ~85%, while accuracy on the escalated 12% is
# unchanged. Tune the thresholds on validation data to hit your accuracy floor.pythonDistillation: match the teacher at a fraction of the cost
import torch, torch.nn.functional as F
def distillation_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7):
"""Soft targets carry the teacher's uncertainty structure -- the 'dark knowledge'."""
soft = F.kl_div(
F.log_softmax(student_logits / T, dim=-1),
F.softmax(teacher_logits / T, dim=-1),
reduction="batchmean") * (T ** 2) # T^2 keeps gradient scale stable
hard = F.cross_entropy(student_logits, labels)
return alpha * soft + (1 - alpha) * hard
# Typical result on a text classification task:
# teacher: 340M params, 180 ms, F1 0.912
# student: 22M params, 11 ms, F1 0.897 <- 16x faster, -1.5 F1
#
# Distilling on UNLABELLED production data is the key trick: the teacher
# labels millions of real examples for free, so the student trains on the
# true serving distribution rather than a small curated set.Say these points
- Profile every stage first — feature I/O and logging often dominate the model.
- Move logging and non-essential work off the request path.
- Cascades give large average-latency and cost wins with tunable accuracy loss.
- Distillation on unlabelled production data matches the real serving distribution.
- Measure p99 under realistic concurrency; the tail is queueing, not compute.
Avoid these mistakes
- Optimising the model when the bottleneck is I/O.
- Benchmarking latency on an idle machine with batch size 1.
- Synchronous logging or metrics on the request path.
Expect these follow-ups
- How would you choose cascade thresholds to hit a specific accuracy floor?
- When does GPU inference beat CPU for a small model?
- How does dynamic batching trade latency against throughput?
AdvancedcostinfrastructureefficiencyAsked at Netflix, Pinterest
30-second answer
Attribute costs first — training, inference, storage, data processing — because teams are usually wrong about the split. Then: right-size instances and use spot for training, cache aggressively, use cascades and smaller distilled models for inference, autoscale to real traffic instead of peak, delete unused features and stale artefacts, and kill models nobody uses. The last one is usually the single biggest saving and nobody looks for it.
pythonCost attribution before optimisation
def attribute(monthly):
total = sum(monthly.values())
for k, v in sorted(monthly.items(), key=lambda kv: -kv[1]):
print(f"{k:26s} ${v:>9,.0f} {v/total:>6.1%}")
print(f"{'TOTAL':26s} ${total:>9,.0f}")
attribute({
"inference_gpu": 88_000,
"training_gpu": 42_000,
"feature_pipeline": 31_000,
"vector_db": 18_000,
"data_storage": 9_000,
"experiment_tracking": 7_000,
"monitoring": 5_000,
})
# inference_gpu $ 88,000 44.0% <- always-on capacity
# training_gpu $ 42,000 21.0%
# feature_pipeline $ 31,000 15.5%
# vector_db $ 18,000 9.0%
# data_storage $ 9,000 4.5%
# experiment_tracking $ 7,000 3.5%
# monitoring $ 5,000 2.5%
#
# Now go one level deeper on inference: which MODEL, and what UTILISATION?| Lever | Typical saving | Quality risk | Effort |
|---|---|---|---|
| Decommission unused models | 10–30% | None | Low — audit call volume |
| Spot/preemptible for training | 60–80% of training | None (with checkpointing) | Low |
| Right-size instances | 20–40% | None | Low |
| Autoscale to traffic, not peak | 30–50% of inference | None if warm-up is handled | Medium |
| Cascade + distillation | 50–85% of inference | 1–3% | High |
| Cache predictions/features | 20–60% | Staleness | Medium |
| Prune unused features | 10–30% of pipeline | None | Medium |
| Lifecycle policies on artefacts | 50%+ of storage | None | Low |
pythonUtilisation-driven right-sizing
def gpu_efficiency(model_stats):
"""Low utilisation means you are paying for idle silicon."""
for m in model_stats:
util = m["avg_gpu_util"]
rps_per_gpu = m["rps"] / m["n_gpus"]
monthly = m["n_gpus"] * m["hourly_cost"] * 24 * 30
verdict = ("CONSOLIDATE or move to CPU" if util < 0.20 else
"reduce replicas" if util < 0.45 else
"healthy" if util < 0.80 else "add capacity")
print(f"{m['name']:22s} util={util:5.1%} rps/gpu={rps_per_gpu:6.1f} "
f"${monthly:>8,.0f}/mo -> {verdict}")
gpu_efficiency([
{"name":"ranker-v3", "avg_gpu_util":0.72,"rps":840,"n_gpus":6,"hourly_cost":3.06},
{"name":"embed-service", "avg_gpu_util":0.14,"rps": 22,"n_gpus":4,"hourly_cost":3.06},
{"name":"llm-assistant", "avg_gpu_util":0.38,"rps": 11,"n_gpus":8,"hourly_cost":4.10},
])
# ranker-v3 util=72.0% rps/gpu= 140.0 $ 13,219/mo -> healthy
# embed-service util=14.0% rps/gpu= 5.5 $ 8,813/mo -> CONSOLIDATE or move to CPU
# llm-assistant util=38.0% rps/gpu= 1.4 $ 23,616/mo -> reduce replicas
#
# ~$20k/month of the $88k inference spend is idle capacity.Two more levers worth naming: batch what does not need to be online (many 'real-time' predictions are consumed hours later and can be precomputed at a fraction of the cost), and negotiate committed-use discounts once your baseline utilisation is stable — 30–50% off on-demand pricing for a one-year commitment is standard and requires no engineering at all.
Say these points
- Attribute costs before optimising — intuitions about the split are usually wrong.
- Audit deployed models by call volume; unused endpoints are free savings.
- Spot instances with checkpointing cut training cost 60–80%.
- GPU utilisation below ~20% means consolidate or move to CPU.
- Report cost and quality together, and gate cuts behind the same canary as a model change.
Avoid these mistakes
- Optimising the largest line item without checking utilisation first.
- Provisioning for peak traffic 24/7.
- Cutting cost without a quality guardrail.
Expect these follow-ups
- How do you use spot instances safely for a 3-day training run?
- How would you decide between self-hosting and a managed inference API?
- How do you attribute shared platform cost back to individual models?
Showing 5 of 5 questions for model-monitoring-and-drift-detection.
Check your understanding
5 questions · no sign-up, nothing stored
Hands-on challenge
Build it — this is what you talk about in a deep-dive round.
Build an ML monitoring and incident-response toolkit
Instrument a deployed model with the observability an on-call engineer would actually need.
Requirements
- Per-feature PSI and KS drift reporting against a versioned reference window, with severity classification.
- Label-free monitoring: prediction distribution shift, confidence collapse, and a proxy-outcome correlation.
- A retraining policy with triggers, cooldown, and a mandatory offline gate before promotion.
- A triage script that bisects model-vs-input using a golden batch and ranks suspect features.
Stretch goals
- Implement a cascade and report the escalation rate, average latency and accuracy delta.
- Simulate three incident types (unit change, null spike, gradual drift) and verify your monitors catch each.