Linear & Logistic Regression Deep Dive
Interviewers still open with linear models because they expose whether you understand optimisation, probability and diagnostics — or just call .fit().
Covers: Linear regression assumptions, logistic regression & odds, L1/L2 regularization, gradient descent variants, multicollinearity
Linear and logistic regression remain the most-asked models in ML interviews, at every level. Not because companies deploy them everywhere (many do), but because they are the smallest models where you can be asked about the loss function, the optimiser, the probabilistic interpretation, regularisation and diagnostics — all in one question.
30-second answer
Linearity in the parameters, independent errors, homoscedasticity, normally distributed errors, and no perfect multicollinearity. In practice linearity and independence matter most; normality only matters for small-sample inference (the CLT rescues large samples), and heteroscedasticity biases your standard errors rather than your coefficients — fixable with robust standard errors.
| Assumption | Check | Consequence if violated | Fix |
|---|---|---|---|
| Linearity in parameters | Residuals vs fitted plot; partial dependence | Biased coefficients — the model is simply wrong | Add polynomial/spline/interaction terms, or switch model class |
| Independent errors | Durbin–Watson; ACF plot of residuals | Standard errors far too small → false significance | Time-series models, clustered/robust SEs, mixed effects |
| Homoscedasticity | Residuals vs fitted (funnel shape); Breusch–Pagan | Coefficients still unbiased, SEs wrong | Robust (HC3) SEs, log-transform y, weighted least squares |
| Normal errors | Q-Q plot of residuals | Only matters for small-sample inference | Usually ignore at n > ~100; bootstrap if worried |
| No perfect multicollinearity | VIF, condition number | Unstable, uninterpretable coefficients | Drop/combine features, ridge, PCA |
import numpy as np
import statsmodels.api as sm
from statsmodels.stats.diagnostic import het_breuschpagan
from statsmodels.stats.outliers_influence import variance_inflation_factor
def diagnose(X, y):
Xc = sm.add_constant(X)
m = sm.OLS(y, Xc).fit()
resid, fitted = m.resid, m.fittedvalues
# Heteroscedasticity
lm, lm_p, _, _ = het_breuschpagan(resid, Xc)
print(f"Breusch-Pagan p = {lm_p:.4f}"
f" -> {'HETEROSCEDASTIC' if lm_p < 0.05 else 'ok'}")
# Multicollinearity
vifs = [variance_inflation_factor(Xc.values if hasattr(Xc,'values') else Xc, i)
for i in range(1, Xc.shape[1])]
for i, v in enumerate(vifs):
flag = "HIGH" if v > 10 else ("moderate" if v > 5 else "ok")
print(f" VIF x{i}: {v:6.2f} {flag}")
# Robust standard errors when homoscedasticity fails
if lm_p < 0.05:
print("\nRefitting with HC3 robust standard errors:")
print(m.get_robustcov_results(cov_type="HC3").summary2().tables[1])
return m
# Breusch-Pagan p = 0.0001 -> HETEROSCEDASTIC
# VIF x0: 1.03 ok
# VIF x1: 14.72 HIGH <- correlated with x2, coefficients unstable
# VIF x2: 14.66 HIGHMulticollinearity: what it does and does not break
This is the most misunderstood item on the list. Multicollinearity does not hurt predictive accuracy — the fitted values are fine. It hurts coefficient interpretation: when two features are 95% correlated, the fit can put +50 on one and −48 on the other and be numerically almost as good as +2 and 0. Your coefficients become unstable across resamples, standard errors inflate, and any causal story you tell from them is fiction.
R²_j is from regressing feature j on all the others. VIF > 10 is the conventional alarm; > 5 deserves a look.
Say these points
- Five assumptions, but linearity and independence are the ones that break predictions.
- Heteroscedasticity breaks standard errors, not coefficients — use HC3.
- Normality of errors is largely irrelevant at large n thanks to the CLT.
- Multicollinearity destroys interpretability, not accuracy. VIF > 10 is the flag.
- Always ask whether the goal is prediction or inference — it changes everything.
Avoid these mistakes
- Claiming linear regression assumes the *features* are normally distributed. It assumes the *errors* are.
- Dropping correlated features in a pure prediction problem for no benefit.
- Reporting p-values from an OLS fit on autocorrelated time-series residuals.
Expect these follow-ups
- How would you model a clearly non-linear relationship while keeping a linear model?
- What does 'linear in the parameters' allow that 'linear in the features' does not?
- When would you use quantile regression instead of OLS?
30-second answer
MSE combined with a sigmoid is non-convex in the weights, so gradient descent can stall in local minima, and its gradient contains a σ'(z) factor that vanishes exactly when the model is confidently wrong. Cross-entropy is convex for logistic regression and its gradient simplifies beautifully to (p − y)x — the sigmoid derivative cancels, so confident mistakes produce large gradients.
The derivation (know this by heart)
Let , . The key identity is .
What goes wrong with MSE
That extra factor survives. When or it goes to zero, so a confidently wrong prediction produces an almost zero gradient. The model gets stuck being confidently wrong — the worst possible failure mode. On top of that, MSE ∘ sigmoid is non-convex in , so you lose the global-optimum guarantee that makes logistic regression attractive in the first place.
import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))
print(f"{'z':>6} {'p':>7} {'y':>3} {'|CE grad|':>10} {'|MSE grad|':>11}")
for z in (-8, -4, -1, 0, 1, 4, 8):
p = sig(z); y = 1.0 # truth is 1; negative z = confidently WRONG
ce = abs(p - y)
mse = abs((p - y) * p * (1 - p))
print(f"{z:>6} {p:>7.4f} {y:>3.0f} {ce:>10.4f} {mse:>11.6f}")
# z p y |CE grad| |MSE grad|
# -8 0.0003 1 0.9997 0.000335 <- confidently wrong, MSE barely learns
# -4 0.0180 1 0.9820 0.017345
# -1 0.2689 1 0.7311 0.143734
# 0 0.5000 1 0.5000 0.125000
# 4 0.9820 1 0.0180 0.000318
# 8 0.9997 1 0.0003 0.000000
#
# At z=-8 cross-entropy pushes with force ~1.0; MSE pushes with 0.0003.
# That is a 3000x difference in learning signal on the examples you most need to fix.Interpreting the coefficients: log-odds
So a coefficient of 0.7 means the odds multiply by per unit increase — the odds double. Note this is odds, not probability: doubling odds from 0.01 to 0.02 moves probability from ~1% to ~2%, but doubling odds from 1 to 2 moves probability from 50% to 67%. Getting this distinction right is a reliable senior signal.
Say these points
- ∂L/∂w = (p − y)x — the sigmoid derivative cancels against cross-entropy's denominator.
- MSE keeps a p(1−p) factor that vanishes exactly when the model is confidently wrong.
- MSE ∘ sigmoid is non-convex; cross-entropy ∘ sigmoid is convex.
- Coefficients are log-odds; e^w is the odds ratio, not a probability change.
- Perfect separation sends the MLE to infinity; L2 regularisation is the fix (and sklearn's default).
Avoid these mistakes
- Saying 'MSE is for regression, cross-entropy for classification' with no mechanism.
- Interpreting logistic coefficients as probability changes.
- Forgetting that sklearn regularises by default, then wondering why coefficients are shrunk.
Expect these follow-ups
- Show that softmax + cross-entropy gives the same (p − y) form.
- Why do frameworks fuse softmax and cross-entropy into one operation?
- How would you handle 200 classes efficiently in the output layer?
30-second answer
L2 (ridge) shrinks all coefficients smoothly, handles correlated features by splitting weight between them, and never zeroes anything. L1 (lasso) produces exact zeros for automatic feature selection but arbitrarily picks one from a correlated group. Elastic net combines both to get sparsity plus stable grouping. Tune λ by cross-validation over a log-spaced grid, always on standardised features.
Elastic net. α = 1 is pure lasso, α = 0 is pure ridge.
| L2 / Ridge | L1 / Lasso | Elastic net | |
|---|---|---|---|
| Sparsity | None | Exact zeros | Exact zeros |
| Correlated features | Splits weight evenly — stable | Picks one arbitrarily — unstable across resamples | Groups them together |
| Solution | Closed form | Requires coordinate descent / LARS | Coordinate descent |
| p ≫ n | Works | Selects at most n features | Can select more than n |
| Best for | Prediction with many weak, correlated signals | Interpretability, genuinely sparse truth | Correlated groups where you still want sparsity |
import numpy as np
from sklearn.linear_model import LassoCV, RidgeCV, ElasticNetCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
n, p = 300, 40
X = rng.normal(size=(n, p))
X[:, 1] = X[:, 0] + rng.normal(scale=0.05, size=n) # nearly duplicate features
X[:, 2] = X[:, 0] + rng.normal(scale=0.05, size=n)
w = np.zeros(p); w[[0, 1, 2, 10, 20]] = [1.5, 1.5, 1.5, -2.0, 1.0]
y = X @ w + rng.normal(scale=1.0, size=n)
# Standardisation is MANDATORY: the penalty is scale-dependent.
models = {
"Ridge": make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 50))),
"Lasso": make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0)),
"ElasticNet": make_pipeline(StandardScaler(), ElasticNetCV(l1_ratio=[.1,.5,.7,.9,.95,1],
cv=5, random_state=0)),
}
for name, m in models.items():
m.fit(X, y)
coef = m[-1].coef_
print(f"{name:11s} nonzero={np.sum(np.abs(coef) > 1e-8):3d}"
f" corr-group coefs={np.round(coef[:3], 2)}")
# Ridge nonzero= 40 corr-group coefs=[0.94 0.93 0.94] <- split evenly
# Lasso nonzero= 9 corr-group coefs=[2.66 0. 0. ] <- picked ONE
# ElasticNet nonzero= 14 corr-group coefs=[1.06 0.86 0.9 ] <- kept the groupTuning λ properly
Standardise inside the pipeline
The penalty is scale-dependent: a feature in dollars gets a different effective penalty than one in fractions. Scale inside the CV fold, never before the split, or you leak.Search on a log grid
λ effects are multiplicative.np.logspace(-4, 2, 50)covers six orders of magnitude in 50 fits. A linear grid wastes almost all its evaluations.Prefer the one-standard-error rule
Rather than the λ with the best CV score, take the largest λ whose score is within one standard error of the best. You get a simpler, more stable model at statistically indistinguishable performance — standard practice inglmnetand well worth naming.Never penalise the intercept
Shrinking the intercept toward zero just biases the predicted mean. Every good implementation excludes it; if you hand-roll the objective, exclude it yourself.
Say these points
- Ridge: smooth shrinkage, splits weight across correlated features, closed form.
- Lasso: exact zeros, but unstable selection among correlated features, capped at n features when p ≫ n.
- Elastic net: sparsity with grouping — the default when features are correlated.
- Standardise inside the pipeline; tune λ on a log grid; use the 1-SE rule.
- Never penalise the intercept.
Avoid these mistakes
- Regularising unstandardised features and getting nonsense selection.
- Presenting the lasso-selected feature list as a stable statement of importance.
- Fitting the scaler on the full dataset before cross-validation.
Expect these follow-ups
- Why does lasso select at most n features when p ≫ n?
- How does the group lasso differ, and when do you need it?
- Explain why weight decay in AdamW is not the same as adding L2 to the loss.
30-second answer
Batch GD uses the whole dataset per step: an exact gradient but one update per epoch and no ability to scale past memory. SGD uses one sample: very noisy but many fast updates. Mini-batch takes 32–1024 samples, which gives a low-variance gradient estimate, saturates GPU parallelism, and keeps enough noise to escape sharp minima. It is the only one of the three that maps well onto hardware.
| Batch | SGD (n=1) | Mini-batch | |
|---|---|---|---|
| Gradient quality | Exact | Very noisy | Low-variance estimate |
| Updates per epoch | 1 | N | N / B |
| Memory | O(N) | O(1) | O(B) |
| Hardware utilisation | Poor (one huge op) | Terrible (no parallelism) | Excellent |
| Escapes sharp minima | No | Yes (lots of noise) | Yes (tunable noise) |
| Convergence | Smooth, can stall | Erratic | Smooth enough, fast |
The statistical reason mini-batch wins: the standard error of the gradient estimate falls as . Going from to cuts gradient noise by ~5.7× for 32× the compute — a great trade. Going from to cuts it by another 5.7× for 32× the compute — a terrible trade, because you have already hit diminishing returns. That curve is why batch sizes cluster in the 32–1024 range rather than being as large as memory allows.
Scaling batch size correctly
- Linear scaling rule (Goyal et al.): when you multiply batch size by k, multiply the learning rate by k. A bigger batch gives a more reliable gradient, so you can afford a bigger step.
- Warmup: ramp the LR from ~0 over the first few hundred/thousand steps. Without it, the large initial LR blows up an untrained network before the linear scaling has anything to work with.
- Square-root scaling is the alternative rule, often better for adaptive optimisers like Adam where the update is already normalised per-parameter.
- Gradient accumulation simulates a large batch on small hardware: run k forward/backward passes, sum the gradients, then step once. Mathematically equivalent to batch k·B — but be careful whether your loss reduces with
meanorsum.
import torch
accum_steps = 8 # effective batch = 8 * micro_batch
optimizer.zero_grad(set_to_none=True)
for step, (x, y) in enumerate(loader):
with torch.autocast("cuda", dtype=torch.bfloat16):
loss = criterion(model(x), y)
# CRITICAL: divide, because criterion already averages within the micro-batch.
# Without this, the effective learning rate silently becomes 8x larger.
(loss / accum_steps).backward()
if (step + 1) % accum_steps == 0:
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
scheduler.step()
optimizer.zero_grad(set_to_none=True)Finally, be ready to place the modern optimisers on this map: momentum smooths the mini-batch noise with an exponential moving average of gradients; RMSProp/Adam additionally normalise per-parameter by a running estimate of gradient magnitude; AdamW decouples weight decay from that normalisation. They are all responses to the same underlying problem — a noisy, badly-conditioned gradient.
Say these points
- Gradient noise falls as 1/√B — diminishing returns explain the 32–1024 sweet spot.
- Mini-batch is the only variant that maps well onto GPU parallelism.
- Mini-batch noise is an implicit regulariser that favours flat minima.
- Scale LR linearly with batch size, and use warmup.
- Gradient accumulation requires dividing the loss by the accumulation count.
Avoid these mistakes
- Claiming 'bigger batch is always better' — it often degrades generalisation.
- Scaling batch size without scaling the learning rate, then reporting that the model got worse.
- Omitting the loss division in gradient accumulation.
Expect these follow-ups
- Why does Adam sometimes generalise worse than SGD with momentum?
- What does learning-rate warmup actually fix, mechanically?
- How would you find a good learning rate in one training run?
30-second answer
One-hot is out — 50k sparse columns. The realistic options are target/mean encoding with out-of-fold computation and smoothing, hashing to a fixed number of buckets, learned embeddings if you have a neural model, frequency encoding, or grouping the long tail into an 'other' bucket. The choice depends on model family, cardinality of the *useful* signal, and whether new categories appear at serving time.
| Technique | Pros | Cons | Use when |
|---|---|---|---|
| Target / mean encoding | One column, captures real signal, strong for GBDTs | Leaks badly if done naively; needs smoothing | Tree models, high cardinality, stable categories |
| Hashing trick | Fixed memory, handles unseen values for free, streaming-friendly | Collisions, no interpretability | Online/streaming systems, extreme cardinality |
| Learned embedding | Captures similarity, transfers across tasks | Needs a neural model and enough data per category | Deep models, RecSys, transferable representations |
| Frequency / count encoding | Trivial, leak-free, surprisingly effective | Loses identity — two equally common categories collide | Quick baselines, popularity really is the signal |
| Top-k + 'other' | Simple, robust, interpretable | Throws away the tail | When the top 100 cover 95% of traffic |
Target encoding without leaking
Naive target encoding — replacing each category with the mean of the target over all rows in that category — leaks the label directly into the feature. A category with a single row gets encoded as exactly that row's label. The two required corrections are out-of-fold computation and smoothing toward the global mean.
Smoothed target encoding. m is the prior weight: small categories are pulled toward the global mean.
import numpy as np, pandas as pd
from sklearn.model_selection import KFold
def target_encode(train, test, col, target, m=20, n_splits=5, seed=0):
"""Out-of-fold smoothed target encoding. No leakage."""
global_mean = train[target].mean()
oof = np.full(len(train), np.nan)
for tr_idx, va_idx in KFold(n_splits, shuffle=True, random_state=seed).split(train):
fold = train.iloc[tr_idx]
stats = fold.groupby(col)[target].agg(["mean", "count"])
smooth = (stats["mean"] * stats["count"] + global_mean * m) / (stats["count"] + m)
oof[va_idx] = train.iloc[va_idx][col].map(smooth).fillna(global_mean).values
# Test uses statistics from the FULL training set (test labels never touched)
stats = train.groupby(col)[target].agg(["mean", "count"])
smooth = (stats["mean"] * stats["count"] + global_mean * m) / (stats["count"] + m)
test_enc = test[col].map(smooth).fillna(global_mean).values
return oof, test_enc
# The leak, quantified:
# naive in-fold encoding : CV AUC 0.94 -> holdout AUC 0.71 (fantasy)
# out-of-fold + smoothing: CV AUC 0.78 -> holdout AUC 0.77 (honest)Embeddings and the hashing trick
import torch.nn as nn
# --- Learned embedding: rule of thumb for dimension ---
n_categories = 50_000
emb_dim = min(600, round(1.6 * n_categories ** 0.56)) # fastai heuristic -> 227
embedding = nn.Embedding(n_categories + 1, emb_dim, # +1 for the OOV bucket
padding_idx=n_categories)
# 50k x 227 floats = 45 MB. Often the largest tensor in a tabular net.
# --- Hashing trick: fixed memory, unseen categories handled automatically ---
import hashlib
def hash_bucket(value: str, n_buckets: int = 2 ** 16) -> int:
return int(hashlib.md5(value.encode()).hexdigest(), 16) % n_buckets
# Collision probability with 50k values in 65,536 buckets is high --
# use two independent hashes (feature hashing with signed sums) or more buckets.Say these points
- One-hot fails past a few thousand categories; pick by model family and serving constraints.
- Target encoding must be out-of-fold and smoothed, or it leaks the label.
- Hashing gives fixed memory and free handling of unseen categories, at the cost of collisions.
- Embedding dim ≈ 1.6 · n^0.56 capped at 600 is a solid starting heuristic.
- Check the head/tail distribution first — top-k + 'other' often wins.
Avoid these mistakes
- Computing target encoding before the CV split.
- Ignoring unseen categories at serving time (a KeyError in prod at 3am).
- Using label encoding (0…49999) for a linear model, which invents a false ordering.
Expect these follow-ups
- How do CatBoost's ordered target statistics avoid leakage differently?
- Design an embedding strategy for an item catalogue that grows daily.
- How would you handle a category whose meaning drifts over time?
Showing 5 of 5 questions for regression-and-regularization-interview.
Check your understanding
5 questions · no sign-up, nothing stored
Hands-on challenge
Build it — this is what you talk about in a deep-dive round.
Implement regularised logistic regression from scratch
Write a NumPy implementation and validate it against scikit-learn, then use it to demonstrate the MSE failure mode.
Requirements
- Numerically stable sigmoid and log-loss (no `log(0)`, use the log-sum-exp trick).
- Full-batch and mini-batch gradient descent with configurable L1/L2/elastic-net penalties, intercept unpenalised.
- Match `sklearn.linear_model.LogisticRegression` coefficients to 1e-3 on the same objective.
- Reproduce the confidently-wrong gradient comparison between cross-entropy and MSE, and plot it.
Stretch goals
- Add a Newton/IRLS solver and compare its iteration count to gradient descent.
- Implement out-of-fold smoothed target encoding as a scikit-learn transformer and prove it is leak-free.