Feature Engineering, Imbalanced Data & Unsupervised Learning

Five families that cover most of the value

Interactions and encodings worth adding

• Ratios beat differences for tree models, because a tree can already express a difference with two splits but needs many to approximate a ratio.
• Cyclical encodings for time: sin(2π·hour/24), cos(2π·hour/24) so 23:00 and 01:00 are close, instead of maximally far apart.
• Entropy features capture regularity: the entropy of a user's category distribution or day-of-week distribution distinguishes habitual from erratic users.
• Relative-to-cohort features: user_spend / cohort_median_spend normalises away seasonality and macro effects that would otherwise be learned as spurious level shifts.
• Lag and delta features for anything time-series-like — value_t − value_{t−1} and value_t / rolling_mean_28.

When imputation actively hurts

1. 1

   It erases MNAR signal

   The most common and most expensive case. Always add the indicator before deciding imputation is harmless.
2. 2

   It is fitted outside the CV fold

   A median computed on the full dataset leaks validation statistics into training. Imputers belong inside Pipeline, always.
3. 3

   It manufactures impossible rows

   Mean-imputing n_children gives 2.3 children; mean-imputing a bimodal distribution lands you in the empty valley between the modes. The model then learns from data points that cannot exist.
4. 4

   Serving-time missingness differs from training

   If a feature is 5% missing in training but 60% missing at serving (an upstream service is down), your imputed constant now dominates the input distribution. Monitor missingness rates in production as a first-class metric.
5. 5

   It shrinks variance and biases inference

   Single imputation understates uncertainty. If you need valid standard errors, use multiple imputation (MICE) and pool across imputations — not a single fill.

Restoring calibration after resampling

• Resample only the training fold, never validation or test. Your evaluation must reflect production prevalence.
• Resample inside the CV loop via imblearn.pipeline.Pipeline, or you leak synthetic neighbours across the fold boundary.
• Prefer class weights to physical resampling — same expected effect, no fabricated data, less pipeline complexity.
• Collect more positives if you possibly can. 140 positive examples is the actual constraint here; every algorithmic trick is working around a data problem.

k-means assumptions and when to use something else

• Assumes spherical, similarly-sized, similarly-dense clusters and uses Euclidean distance — so features must be scaled, and elongated or nested shapes will be split wrongly.
• DBSCAN/HDBSCAN for arbitrary shapes, no need to pick k, and a native notion of noise points. Needs a density parameter instead.
• Gaussian mixtures for elliptical clusters and soft assignments, with BIC/AIC giving a principled model-selection criterion.
• Hierarchical/agglomerative when you want a dendrogram and the ability to choose the granularity afterwards.
• Embedding + clustering for text/images: encode with a sentence or vision model, reduce with UMAP, then cluster with HDBSCAN. This is the standard modern pipeline for topic discovery.

Close by naming the business test: “A segmentation is successful when someone can act differently on each segment. If marketing cannot write a different email for cluster 3, the clustering has not earned its keep — no matter what the gap statistic says.”

The labelling flywheel is the real system

1. 1

   Log every review decision

   Analyst confirms/dismisses → a label with a timestamp and a reason code. This is free labelled data generated by work you were doing anyway.
2. 2

   Reserve a random-sample bucket

   Review ~1% of unflagged traffic too. Without it you only ever learn about cases your current detector already catches, and you can never estimate recall.
3. 3

   Migrate to supervised once you have enough

   With a few thousand confirmed positives, a supervised model on the same features almost always beats an unsupervised score, because it learns which anomalies actually matter rather than which are merely unusual.
4. 4

   Keep the unsupervised layer running

   The supervised model only knows the attack patterns in its training data. The unsupervised layer is your detector for genuinely novel behaviour — it is the thing that catches the attack nobody has seen.

Two production details worth mentioning: seasonality and drift mean thresholds must be recomputed on a rolling window, not fixed at launch (Black Friday is not an anomaly); and alert fatigue is a real failure mode — a detector nobody trusts is worse than no detector, so tune for precision at the team's real capacity and track the analyst confirmation rate as a first-class health metric.