MLOps & Production MLAdvanced19 min read5 questions
Model Deployment: Serving Patterns, Feature Stores & Rollouts
The gap between a notebook and a service. Serving patterns, the feature-store problem they solve, and how to ship a model without breaking production.
Covers: Batch vs online vs streaming, training/serving skew, feature stores, shadow and canary deploys, rollback
The single most common reason a good model fails in production is not the model — it is the seam between training and serving. Different code paths, different data freshness, different feature definitions. Interviewers for ML engineering roles probe this hard because it is where the real work is.
Filter
0/5 mastered
IntermediateservingarchitecturelatencyAsked at Amazon, Uber
30-second answer
Batch precomputes predictions on a schedule and serves them from a key-value store — cheapest and simplest, but predictions are stale and you can only score entities you know about. Online computes on request — fresh and works for unseen inputs, but you must meet a latency SLA and have every feature available at request time. Streaming sits between: predictions update on events. Choose by how quickly the input changes and how tight the latency budget is.
| Batch | Online | Streaming | |
|---|---|---|---|
| Latency | Hours to days stale | 10–500 ms | Seconds |
| Cost | Lowest — one big job | Highest — always-on capacity | Medium |
| Handles unseen inputs | No | Yes | Yes |
| Feature freshness | As of the last run | Real-time | Event-time |
| Failure blast radius | Small — retry the job | Large — user-facing | Medium |
| Typical use | Weekly churn scores, nightly recommendations | Fraud check, search ranking, chat | Live personalisation, anomaly alerts |
pythonThe latency budget that decides your architecture
LATENCY_BUDGET_MS = 200 # from product: p99 for the whole request
budget = {
"network in/out": 20,
"auth + routing": 10,
"feature fetch (online)": 40, # <- the usual surprise
"feature fetch (batch)": 15,
"preprocessing": 10,
"model forward": ?, # what is left
"postprocess + log": 10,
}
fixed = 20 + 10 + 40 + 15 + 10 + 10
print(f"model inference budget: {LATENCY_BUDGET_MS - fixed} ms")
# model inference budget: 95 ms
#
# 95 ms at p99 rules out: a 7B LLM (300ms+), a 200-model ensemble,
# and any design needing 3 sequential feature-store round trips.
# It comfortably allows: gradient boosting (~2ms), a small transformer
# encoder on GPU (~15ms), or a 4-layer MLP (~1ms).
#
# Notice feature fetching is often LARGER than inference. Optimising the
# model when the bottleneck is I/O is the classic junior mistake.The hybrid pattern most systems actually use
Real systems rarely pick one. The standard architecture is precompute the expensive part, compute the cheap part online:
- Two-tower recommenders: item embeddings are precomputed nightly (expensive); the user tower runs online (cheap) and does an ANN lookup. You get freshness where it matters and amortise the cost where it does not.
- Candidate generation batch, ranking online: narrow millions of items to a few hundred offline, rank those with a heavy model in real time.
- Cached predictions with online fallback: serve the batch score if the entity is known and its features have not changed materially; compute online otherwise. Covers the cold-start case without paying online cost for every request.
- Precomputed features, online model: the most common split of all — features are materialised by a streaming/batch pipeline, and only the model forward pass happens per request.
Say these points
- Choose from the product decision: when is the prediction consumed and how fast does the input change?
- Write down the latency budget — feature fetching is often larger than inference.
- Most real systems are hybrid: precompute the expensive part, compute the fresh part online.
- Batch failures are silent; always serve and monitor a prediction-freshness timestamp.
Avoid these mistakes
- Building online serving for a prediction consumed weekly.
- Optimising model latency when the bottleneck is feature I/O.
- No freshness monitoring on batch predictions.
Expect these follow-ups
- How would you handle a cold-start entity in a batch-serving system?
- What changes when the model needs a GPU?
- How do you version predictions so you can attribute a bad outcome to a model version?
Advancedtraining-serving-skewfeature-storeproductionAsked at Uber, Airbnb
30-second answer
Skew is any difference between how a feature is computed in training and at inference: different code, different data sources, different time windows, or different handling of missing values. The structural fixes are a single shared feature-transformation library used by both paths, a feature store with point-in-time correct training data and identical online serving, and continuous monitoring that re-scores logged serving features offline and compares distributions.
| Skew type | Example | Detection |
|---|---|---|
| Implementation skew | Training uses pandas, serving reimplements it in Java | Log serving features; recompute offline; assert equality |
| Temporal skew | Training uses a 30-day window that includes the label period | Point-in-time joins; forward-only holdout evaluation |
| Source skew | Training reads the warehouse, serving reads a cache with different values | Compare feature distributions between the two paths |
| Freshness skew | Training sees complete data; serving sees a feature 6 hours stale | Log feature age at serving; simulate that staleness in training |
| Null-handling skew | Training imputes the median; serving sends 0 | Explicit null contract; monitor null rates on both sides |
| Distribution skew | Trained on 2024 data; serving on a changed population | Drift monitoring (PSI/KS) per feature |
pythonOne transformation, two call sites — the structural fix
from dataclasses import dataclass
@dataclass(frozen=True)
class FeatureSpec:
"""Defined ONCE. Both training and serving import this module."""
name: str
version: str
dtype: str
default: float | None # explicit null contract
valid_range: tuple[float, float] | None = None
class FeaturePipeline:
SPECS = [
FeatureSpec("txn_count_30d", "v3", "int", default=0),
FeatureSpec("avg_amount_30d", "v3", "float", default=0.0, valid_range=(0, 1e6)),
FeatureSpec("days_since_last", "v2", "int", default=9999),
FeatureSpec("merchant_risk", "v1", "float", default=0.5, valid_range=(0, 1)),
]
def transform(self, raw: dict) -> dict:
"""The ONLY place feature logic lives. No reimplementation anywhere."""
out = {}
for spec in self.SPECS:
v = raw.get(spec.name)
if v is None:
v = spec.default
if spec.valid_range:
lo, hi = spec.valid_range
v = min(max(v, lo), hi) # identical clipping in both paths
out[spec.name] = v
return out
@property
def signature(self) -> str:
"""Baked into the model artefact; serving refuses to load on mismatch."""
import hashlib, json
return hashlib.sha256(json.dumps(
[(s.name, s.version, s.dtype, s.default) for s in self.SPECS],
sort_keys=True).encode()).hexdigest()[:12]
# --- training ---
pipeline = FeaturePipeline()
X = [pipeline.transform(r) for r in training_rows]
model.metadata["feature_signature"] = pipeline.signature
# --- serving --- (same class, same code path)
def predict(request):
features = pipeline.transform(fetch_raw(request.entity_id))
assert pipeline.signature == model.metadata["feature_signature"], \
"FEATURE SCHEMA MISMATCH - refusing to serve"
log_features(request.id, features) # <- essential for skew detection
return model.predict([features])[0]pythonContinuous skew detection on sampled traffic
import numpy as np
from scipy import stats
def detect_skew(logged_serving_features, feature_names, threshold=0.05):
"""Re-derive features offline from the same entities and compare."""
offline = training_pipeline.transform_batch(
[f["entity_id"] for f in logged_serving_features])
issues = []
for name in feature_names:
online_vals = np.array([f[name] for f in logged_serving_features], float)
offline_vals = np.array([f[name] for f in offline], float)
# 1. Exact mismatch rate -- should be ~0 for deterministic features
mismatch = float(np.mean(~np.isclose(online_vals, offline_vals, rtol=1e-6)))
# 2. Distribution shift, in case values are recomputed rather than copied
ks, p = stats.ks_2samp(online_vals, offline_vals)
if mismatch > threshold or p < 0.001:
issues.append({"feature": name, "mismatch_rate": round(mismatch, 4),
"ks": round(float(ks), 4),
"online_mean": round(float(online_vals.mean()), 4),
"offline_mean": round(float(offline_vals.mean()), 4)})
return issues
# [{'feature': 'merchant_risk', 'mismatch_rate': 0.312, 'ks': 0.184,
# 'online_mean': 0.5, 'offline_mean': 0.271}]
#
# Read that as: 31% of requests got the DEFAULT 0.5 online because the
# merchant-risk service was timing out and the client swallowed the error.
# The model was trained believing this feature was always populated.Why feature stores exist
A feature store is the productised answer to this problem. Its two defining capabilities are point-in-time correct training-set generation (ask 'what did we know about entity X at timestamp T?' and get an answer that never includes future data) and a low-latency online store serving the identical values. Definitions are written once and materialised to both the offline and online stores from the same computation, which removes implementation skew by construction.
Say these points
- Skew has six flavours: implementation, temporal, source, freshness, null-handling, distribution.
- One shared transformation module used by both paths is the structural fix.
- Bake a feature-schema signature into the model artefact and refuse to serve on mismatch.
- Log serving features and re-score offline — this one practice catches most skew.
- Feature stores productise point-in-time correctness; adopt when scale justifies it.
Avoid these mistakes
- Reimplementing feature logic in the serving language.
- Not logging the features actually used at inference time.
- Assuming a feature is always populated at serving because it always was in training.
Expect these follow-ups
- How do you build point-in-time correct training data from event logs?
- How do you handle a feature that is 6 hours stale at serving but fresh in training?
- What is the migration path when a feature definition must change?
AdvanceddeploymentcanaryshadowAsked at Netflix, Stripe
30-second answer
Shadow first: run the new model on real traffic without serving its output, and compare score distributions and latency against the incumbent. Then canary a small percentage with automated guardrail monitoring, ramp progressively, and keep the old version warm for instant rollback. Ship the model behind a flag so the rollback is a config change, not a redeploy.
1 — Offline gate
The new model must beat the incumbent on a frozen holdout, with the same CV scheme, plus checks on calibration, per-segment performance, and fairness metrics if applicable. Regressions on any protected segment block the release even if the aggregate improves.2 — Shadow deployment
Serve production traffic to both models; return only the incumbent's output. Compare score distributions (KS test), p99 latency, error rate, and disagreement rate. This catches skew, missing features and performance problems with zero user risk. Run it for at least one full traffic cycle — a week if there is weekly seasonality.3 — Canary
Route 1% → 5% → 25% → 50% → 100%, holding each stage long enough to observe the primary metric. Automate the guardrails: error rate, latency p99, and business metric all have kill thresholds that trigger an automatic rollback without a human in the loop.4 — A/B for the business metric
Canary answers 'is it broken?'. Only a properly sized A/B answers 'is it better?'. Run it long enough to detect the MDE you care about, with pre-registered success criteria.5 — Keep a holdback
Leave 1–5% of traffic permanently on the old model for weeks. It is your long-horizon control for feedback-loop effects that a two-week test cannot see.
pythonShadow comparison and automated canary guardrails
import numpy as np
from scipy import stats
def shadow_report(champion_scores, challenger_scores, champ_ms, chall_ms):
ks, p = stats.ks_2samp(champion_scores, challenger_scores)
disagree = float(np.mean((champion_scores > 0.5) != (challenger_scores > 0.5)))
return {
"score_ks": round(float(ks), 4),
"score_ks_p": float(p),
"mean_shift": round(float(challenger_scores.mean() - champion_scores.mean()), 4),
"decision_disagreement": round(disagree, 4),
"p99_latency_champ": round(float(np.percentile(champ_ms, 99)), 1),
"p99_latency_chall": round(float(np.percentile(chall_ms, 99)), 1),
}
# {'score_ks': 0.041, 'score_ks_p': 0.21, 'mean_shift': 0.003,
# 'decision_disagreement': 0.038, 'p99_latency_champ': 42.1,
# 'p99_latency_chall': 118.7}
#
# Scores agree closely (good) but p99 latency TRIPLED. That is a blocker
# the offline evaluation could never have surfaced.
GUARDRAILS = {
"error_rate": {"max": 0.005, "vs_baseline": 1.5},
"p99_latency_ms": {"max": 200, "vs_baseline": 1.2},
"conversion_rate": {"min_vs_baseline": 0.98}, # allow 2% dip, no more
"prediction_mean": {"vs_baseline": 1.3}, # catches a broken feature
}
def evaluate_canary(metrics, baseline):
breaches = []
for name, rule in GUARDRAILS.items():
v, b = metrics[name], baseline[name]
if "max" in rule and v > rule["max"]:
breaches.append(f"{name}={v} exceeds hard max {rule['max']}")
if "vs_baseline" in rule and b and v > b * rule["vs_baseline"]:
breaches.append(f"{name}={v} is {v/b:.2f}x baseline {b}")
if "min_vs_baseline" in rule and b and v < b * rule["min_vs_baseline"]:
breaches.append(f"{name}={v} is only {v/b:.2f}x baseline {b}")
return breaches
if breaches := evaluate_canary(current, baseline):
rollback(reason=breaches) # automatic, no human required| Technique | Answers | Risk | Traffic needed |
|---|---|---|---|
| Shadow | Does it break? Is it slow? | Zero | Any |
| Canary | Does it break under real use? | Low, bounded | Small |
| A/B test | Is it better? | Medium | Large (powered) |
| Interleaving (ranking) | Is it better? | Medium | 10–100× less than A/B |
| Blue-green | Instant cutover with instant revert | Medium — all-or-nothing | Any |
| Multi-armed bandit | Which variant, adaptively? | Low | Medium |
Say these points
- Shadow → canary → A/B → permanent holdback, in that order.
- Shadow catches latency and skew problems offline evaluation cannot.
- Automate guardrail kill thresholds so rollback needs no human.
- Rollback must be a config flag against a warm artefact, not a redeploy.
- Model rollback does not undo downstream writes; gate mutating effects.
Avoid these mistakes
- Going straight from offline metrics to 100% traffic.
- Canarying without automated rollback triggers.
- Deploying a model before the feature pipeline it depends on.
Expect these follow-ups
- How long should a shadow run before you trust it?
- How do you roll out a model that changes downstream data?
- What would you monitor in the first hour after a full ramp?
Advancedtestingci-cdqualityAsked at Google, Stripe
30-second answer
Regular software has deterministic expected outputs; ML code has statistical ones. So you test at four levels: data tests (schema, ranges, null rates, distribution), code tests (deterministic unit tests on transformations, shape and dtype assertions), model tests (behavioural — invariance, directional expectation, minimum functionality — plus performance thresholds on a frozen set), and infrastructure tests (serialisation round-trip, latency, training/serving parity).
| Level | What you assert | Runs when |
|---|---|---|
| Data | Schema, types, ranges, null rate, cardinality, distribution vs a reference | Every pipeline run |
| Code | Transformations are deterministic and correct on hand-built cases | Every commit |
| Model — behavioural | Invariance, directional expectation, minimum functionality | Every training run |
| Model — performance | Metric ≥ threshold on a frozen set; no per-segment regression | Every training run |
| Infrastructure | Serialise/deserialise identity, latency budget, train/serve parity | Every deploy |
pythonBehavioural tests — the ML-specific level (CheckList style)
import pytest, numpy as np
# --- 1. INVARIANCE: perturbations that must NOT change the prediction ---
def test_invariant_to_name(model, template):
base = model.predict_proba([template.format(name="Alice")])[0, 1]
for name in ["Bob", "Priya", "Kwame", "Yuki", "Mohammed"]:
p = model.predict_proba([template.format(name=name)])[0, 1]
assert abs(p - base) < 0.05, f"prediction moved {p-base:+.3f} for {name}"
def test_invariant_to_irrelevant_feature(model, sample):
base = model.predict_proba([sample])[0, 1]
perturbed = {**sample, "signup_source": "google"} # not causally relevant
assert abs(model.predict_proba([perturbed])[0, 1] - base) < 0.02
# --- 2. DIRECTIONAL: changes that MUST move the prediction a known way ---
@pytest.mark.parametrize("feature,delta,direction", [
("n_failed_payments", +3, "up"), # more failures -> higher churn risk
("days_since_last_login", +60, "up"),
("n_products_used", +2, "down"), # more engagement -> lower risk
])
def test_directional_expectation(model, sample, feature, delta, direction):
base = model.predict_proba([sample])[0, 1]
moved = model.predict_proba([{**sample, feature: sample[feature] + delta}])[0, 1]
if direction == "up":
assert moved > base + 0.01, f"{feature} +{delta} did not increase risk"
else:
assert moved < base - 0.01, f"{feature} +{delta} did not decrease risk"
# --- 3. MINIMUM FUNCTIONALITY: cases the model must never get wrong ---
def test_obvious_cases(model):
obvious_churner = {"days_since_last_login": 400, "n_failed_payments": 5,
"n_products_used": 0, "support_tickets_30d": 8}
assert model.predict_proba([obvious_churner])[0, 1] > 0.8
obvious_retained = {"days_since_last_login": 0, "n_failed_payments": 0,
"n_products_used": 6, "support_tickets_30d": 0}
assert model.predict_proba([obvious_retained])[0, 1] < 0.2
# --- 4. NO SEGMENT REGRESSION: aggregate gains can hide segment losses ---
def test_no_segment_regression(new_model, old_model, eval_df, tol=0.02):
for segment, group in eval_df.groupby("region"):
if len(group) < 200:
continue
new_auc = roc_auc(group.y, new_model.predict_proba(group.X)[:, 1])
old_auc = roc_auc(group.y, old_model.predict_proba(group.X)[:, 1])
assert new_auc >= old_auc - tol, \
f"{segment}: AUC {old_auc:.3f} -> {new_auc:.3f}"pythonData validation as a pipeline gate
from dataclasses import dataclass
@dataclass
class ColumnContract:
name: str
dtype: str
nullable: bool = False
min_value: float | None = None
max_value: float | None = None
max_null_rate: float = 0.0
expected_cardinality: tuple[int, int] | None = None
CONTRACT = [
ColumnContract("amount", "float", min_value=0, max_value=1e6, max_null_rate=0.001),
ColumnContract("country", "str", expected_cardinality=(50, 250)),
ColumnContract("age", "int", min_value=13, max_value=120, max_null_rate=0.05),
]
def validate(df, contract=CONTRACT, reference=None) -> list[str]:
errs = []
for c in contract:
if c.name not in df.columns:
errs.append(f"MISSING COLUMN {c.name}"); continue
col = df[c.name]
null_rate = col.isna().mean()
if null_rate > c.max_null_rate:
errs.append(f"{c.name}: null rate {null_rate:.3f} > {c.max_null_rate}")
if c.min_value is not None and col.min() < c.min_value:
errs.append(f"{c.name}: min {col.min()} < {c.min_value}")
if c.max_value is not None and col.max() > c.max_value:
errs.append(f"{c.name}: max {col.max()} > {c.max_value}")
if c.expected_cardinality:
lo, hi = c.expected_cardinality
if not lo <= col.nunique() <= hi:
errs.append(f"{c.name}: cardinality {col.nunique()} outside [{lo},{hi}]")
if reference is not None and c.name in reference:
psi = population_stability_index(reference[c.name], col)
if psi > 0.25:
errs.append(f"{c.name}: PSI {psi:.3f} -> major distribution shift")
return errs
if errors := validate(new_batch, reference=training_reference):
raise DataQualityError("\n".join(errors)) # FAIL the pipeline, do not trainSay these points
- ML tests assert properties, not exact outputs.
- Four levels: data contracts, deterministic code tests, behavioural model tests, infrastructure tests.
- Behavioural: invariance, directional expectation, minimum functionality.
- Test per-segment performance — aggregate gains hide segment regressions.
- Data contracts must fail the pipeline, not log a warning.
Avoid these mistakes
- Only asserting an aggregate metric threshold.
- No test that the serialised model reproduces in-memory predictions.
- Treating data-quality violations as warnings.
Expect these follow-ups
- How do you test a model whose training run is non-deterministic?
- How would you write a test for a fairness requirement?
- What belongs in CI versus a nightly job, given training takes 4 hours?
Intermediatereproducibilityexperiment-trackingversioningAsked at Databricks, regulated fintech
30-second answer
Five things must be versioned together: code (git SHA), data (a snapshot or immutable query with a timestamp), environment (pinned dependencies and a container digest), configuration (all hyperparameters and feature definitions), and random seeds. Experiment tracking should record all five automatically at training time, plus the resulting metrics and artefacts, because nobody reconstructs this retrospectively.
| Artefact | How to version | Common failure |
|---|---|---|
| Code | Git SHA, and refuse to train on a dirty tree | Trained from an uncommitted notebook |
| Data | Snapshot, or a query + as-of timestamp (Delta/Iceberg time travel) | 'SELECT * FROM users' — the table has changed |
| Environment | Lockfile + container image digest | pip install -r requirements.txt resolves differently later |
| Config | The full resolved config, serialised with the run | Hyperparameters typed into a notebook cell |
| Seeds | Set and record every RNG seed | Different init changes results by 1–2% |
| Feature definitions | Versioned transformation module | Someone 'improved' a feature since |
pythonCapture everything automatically at train time
import hashlib, json, os, platform, random, subprocess, sys
import numpy as np, torch
def set_all_seeds(seed: int):
random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False # NOTE: costs throughput
os.environ["PYTHONHASHSEED"] = str(seed)
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" # deterministic cuBLAS
def capture_provenance(config, data_query, seed) -> dict:
dirty = bool(subprocess.check_output(["git", "status", "--porcelain"]).strip())
if dirty:
raise RuntimeError("Refusing to train from a dirty working tree.")
return {
"git_sha": subprocess.check_output(["git","rev-parse","HEAD"]).decode().strip(),
"data": {
"query": data_query,
"as_of": config["data_as_of"], # point-in-time snapshot
"row_count": config["expected_rows"],
"checksum": hashlib.sha256(data_query.encode()).hexdigest()[:16],
},
"environment": {
"python": sys.version.split()[0],
"platform": platform.platform(),
"packages": subprocess.check_output([sys.executable,"-m","pip","freeze"]
).decode().splitlines(),
"container_digest": os.environ.get("IMAGE_DIGEST", "unknown"),
"cuda": torch.version.cuda,
},
"config": config,
"seed": seed,
"feature_signature": FeaturePipeline().signature,
}
set_all_seeds(42)
prov = capture_provenance(config, DATA_QUERY, seed=42)
import mlflow
with mlflow.start_run() as run:
mlflow.log_params(flatten(config))
mlflow.log_dict(prov, "provenance.json")
model = train(...)
mlflow.log_metrics(evaluate(model))
mlflow.log_artifact("provenance.json")
mlflow.sklearn.log_model(model, "model",
registered_model_name="churn-classifier")The organisational point worth adding: reproducibility must be automatic. Any process that relies on a person remembering to record the data snapshot will fail within a month. Wire provenance capture into the training entrypoint so it is impossible to train without it — and make a dirty git tree a hard error, as in the code above.
Say these points
- Version code, data, environment, config and seeds — all five, together.
- Data is the hard part; use snapshots or table-format time travel.
- Capture provenance automatically in the training entrypoint, never manually.
- Refuse to train from a dirty git tree.
- Aim for statistical reproducibility; bit-exactness costs throughput.
Avoid these mistakes
- Training from a notebook with uncommitted changes.
- Recording the data query but not the as-of timestamp.
- Assuming a requirements.txt reproduces the environment months later.
Expect these follow-ups
- How do you make distributed training reproducible?
- How would you reproduce a model when a user's data was deleted for GDPR?
- What goes in a model card and why does it matter for audits?
Showing 5 of 5 questions for ml-deployment-and-serving-patterns.
Check your understanding
5 questions · no sign-up, nothing stored
Hands-on challenge
Build it — this is what you talk about in a deep-dive round.
Ship a model with a real deployment pipeline
Take any trained model and build the surrounding machinery a production team would require.
Requirements
- A shared feature-transformation module used by both training and a FastAPI serving endpoint, with a schema signature check.
- Data contracts that fail the pipeline on violation, including a PSI drift check against a reference.
- A behavioural test suite: invariance, directional expectation, minimum functionality and per-segment regression.
- Automatic provenance capture (git SHA, data as-of, environment, config, seeds) logged with every run.
Stretch goals
- Implement shadow mode: score both models per request, log both, and produce the comparison report.
- Add canary guardrails with automatic rollback and demonstrate a triggered rollback.