Why “the model will learn it” keeps failing in production
You train a model that looks solid offline, then it hits production and the edges show up fast: a new traffic source, a logging tweak, a changed pricing rule. The model doesn’t “learn” those missing signals if they never make it into the inputs, or if they arrive in a different shape than training.
A lot of teams also confuse “automatic feature learning” with “no feature work.” Even deep models can’t fix wrong labels, stale timestamps, or a target that quietly shifted last week. And when you try to brute-force it with a bigger model, you often pay in latency, GPU cost, and harder debugging.
The practical question isn’t whether features matter. It’s which lever—features, model choice, or more data—actually moves your metrics under real constraints.
You shipped a baseline—now what’s the next lever: features, model, or more data?
That “which lever” question usually shows up right after a baseline ships and the graph stalls: AUC won’t budge, latency is tight, and people start proposing “just use a bigger model” or “let’s collect more data.” The fastest way to choose is to ask what kind of failure you’re seeing in real requests. If predictions are noisy for the same user or item across minutes, fix features and freshness. If the model is consistently wrong for an entire segment, you may need new signals or better labels, not a different architecture.
Reach for a stronger model when you already trust the inputs and the error looks like underfitting (it misses obvious patterns even on clean slices). Reach for more data when variance dominates (metrics swing run to run, rare cases hurt, or long-tail classes collapse). The annoying part: “more data” often means weeks of instrumentation and backfills, while a single bad join can erase any gain.
Make the next change the one you can validate quickly with an offline slice and a small production shadow before you commit to a bigger bill.
The unglamorous checks that beat fancy architectures (labels, leakage, and time)
That “validate quickly” plan usually breaks for a boring reason: you’re validating the wrong thing. Start by pulling 50–100 recent predictions and tracing each one back to the label and the raw inputs. If you see obvious label noise—refunds logged as “success,” support tickets closed automatically, “churn” defined differently by region—no architecture upgrade will rescue you. Fixing one label rule can beat weeks of tuning because it changes what “right” even means.
Then run a leakage check you can explain to a skeptic. If a feature exists only after the decision point (delivery timestamp, “resolved” status, post-click events), it can inflate offline AUC while doing nothing live. A quick test is to rebuild features using only data available up to the prediction time and compare the drop. If it collapses, you were training on the future.
Finally, treat time as a first-class input and constraint. Backfills, late events, and timezone bugs create silent training/serving skew. The cost is real: adding strict cutoffs often lowers offline metrics at first, but it gives you a model you can actually operate.
If you only have a day: which features usually move the needle?
That “model you can actually operate” usually comes down to a few features you can trust and refresh, not a long wishlist. In a one-day window, start with features that reduce obvious ambiguity in live traffic: recency (minutes since last action), frequency (counts in the last 1/7/30 days), and simple ratios (refunds per order, errors per request). If the same user looks different across sessions, add “as-of” time windows so the model sees the same facts at training and serving.
Then fix the boring encoding problems that hide signal. Canonicalize categories (one spelling per vendor), bucket rare values into “other,” and make missingness explicit as its own value. For text-ish fields like URLs or search queries, try cheap splits (domain, path depth, query length) before embeddings.
The constraint you’ll hit is plumbing: building windowed aggregates can hammer your warehouse and add serving latency. Pick one or two aggregates you can compute incrementally, validate on a recent slice, and only then expand the surface area.
When “learned features” are real—and when they’re just expensive
That “expand the surface area” moment is where teams often jump to embeddings or a larger model, hoping it will absorb messy inputs. Learned features are real when the raw signal is high-bandwidth and stable: text meaning, image content, audio, or dense click sequences where hand-built counts lose order. If your input is “what was said” or “what was seen,” a representation model can capture patterns you won’t design in a day.
But learned features get expensive fast when the problem is mostly bookkeeping. If vendor names are inconsistent, timestamps arrive late, or “refund” means three different things, an embedding just learns the mess. You pay twice: more GPU time and longer debugging when a slice breaks.
A quick gut-check: if a human can fix the signal with a deterministic rule (canonicalize, window, cutoff, join correction), do that first. Save learned features for the parts that truly resist rules—and measure the serving bill before you scale it.
Turning feature ideas into a pipeline you can operate
That “measure the serving bill” question usually lands when a feature idea works once, then falls apart the second time you retrain. The fix is to treat features like product code: version them, test them, and make “as-of” time a hard rule. If you can’t say, for any training row, exactly what raw tables and cutoffs produced it, you can’t debug drift or rerun results after a backfill.
Start with a short, owned feature list and a contract for each field: name, data source, window, refresh cadence, and default behavior when data is late. Build one path for training and serving, even if it’s slower at first, and add checks that fail loudly (null spikes, category explosions, timestamp outliers). The real pain is operational: windowed aggregates need compute, and backfills can change history. Lock datasets by date, schedule recomputes, and log feature values alongside predictions so you can trace a bad week without guessing.
After launch: knowing which features are breaking (and what to fix first)
When a metric drops on Tuesday and nobody changed the model, the usual culprit is a feature that shifted: a key stopped joining, an upstream enum got renamed, a late-events lag doubled, or a default started filling half your rows. You catch this fastest by logging feature values with each prediction and keeping a small “golden set” of requests you replay daily.
Then rank suspects with simple checks: null rate, top-k categories, basic ranges, and training-vs-serving distribution deltas on the slices that hurt. Fix the feature that explains the most bad predictions, not the one with the nicest importance plot.
The hard part is cost: replay jobs, backfills, and extra logging eat budget and can add latency. Put a ceiling on what you’ll monitor, and expand only when a new failure repeats.
