Why “just scale it” feels true—and still wastes your budget
You ship a prototype that looks good in a controlled demo, then real users hit it with messy inputs: half-finished sentences, odd formats, edge cases you didn’t test. Someone says the obvious fix is to “just scale it”—bigger model, more tokens, more training. It sounds right because it often nudges benchmark scores up.
But budgets don’t buy averages. They buy the ability to handle the exact failures your product sees. If the misses come from missing context, inconsistent labels, or a workflow that forces the model to guess, more compute can turn into a pricey way to get slightly nicer wrong answers.
The first clue is where errors show up in your product, not in your eval sheet
That usually means pulling a handful of real failures from logs and asking one blunt question: what did the user see, and what did the model not have? In a spreadsheet, an answer is simply “wrong.” In a product, “wrong” has a shape: it happens after a file upload, on long threads, on non‑English inputs, or when users paste tables and expect the model to preserve columns.
Once you map misses to a step in the workflow, the next lever often becomes obvious. If errors cluster around missing fields or ambiguous options, you likely need better inputs or tighter UI constraints, not a bigger model. If the model handles clean cases but falls apart on a specific customer segment, you probably need data coverage there—examples, labels, or retrieval sources that match that segment’s language.
The hard part is cost: instrumenting logs, reviewing outputs, and labeling edge cases takes time you won’t see on a benchmark chart. Do it anyway, then move from “accuracy” to “this screen, this input, this failure,” before you spend a dollar on scale.
When your best model is wrong in the same way every time
When you zoom in on “this screen, this input, this failure,” a pattern often jumps out: the model isn’t randomly missing—it’s repeating the same mistake. You’ll see it hedge with boilerplate refusals, misread the same column in a pasted table, or confidently choose the wrong option whenever two fields look similar. That consistency is useful. It usually means you’re not dealing with “not enough compute,” but a mismatch between what the model can represent and what your task demands.
If the error is systematic, try to write a one-sentence rule that describes it. If you can, you can often fix it with a change in structure: add retrieval for the missing facts, force a schema with function calls, or split one big prompt into two steps where the first extracts fields and the second decides. More examples may help, but only if they teach the missing distinction.
The catch is implementation cost: these fixes touch your UI, logging, and backend contracts, not just your prompt. Still, once you can name the repeated failure, you’re ready to test whether more data even has room to work.
If performance improves with more examples but plateaus fast, what’s missing?
That “room to work” shows up when you add a small batch of targeted examples and the model gets better—then stops getting better. You run a few prompt variants, sprinkle in more labeled cases, and the curve flattens after a quick win. In products, this usually happens when examples are patching surface phrasing, but the model still lacks the thing that makes the decision possible: the right context, a stable format, or a clean target.
Look at what the “better” examples actually changed. If they only help when the input looks similar to the new cases, you’re not learning a rule—you’re matching patterns. The missing piece is often upstream: users don’t provide a key field, the text is too long to fit, or the task depends on a private fact that isn’t in the prompt or retrieval index. More labels won’t create information that never arrives.
The practical snag is that diagnosing this costs time: you need per-slice charts and quick ablations (remove retrieval, shorten context, drop a field) to see what the model is leaning on. If those tests point to context limits or workflow gaps, paying for more compute may only buy a slightly smoother plateau.
The compute trap: spending more and only getting a slightly nicer demo
That “slightly smoother plateau” is where teams often reach for the credit card. You bump to a larger model, raise max tokens, add more retrieved chunks, or run a heavier reranker. The demo looks better because it is better—on the easy path. In production, the same missing field is still missing, the same long thread still gets truncated, and the same ambiguous UI choice still forces a guess.
Compute helps when the task is already well-posed and you’re paying for headroom: longer context to include the right evidence, more reasoning budget for multi-step logic, or a stronger model that can follow your schema reliably. If your failures come from absent information, conflicting sources, or labels that don’t match what users mean, scaling mostly polishes the edges. You get fewer obvious blunders and more confident near-misses.
The hard part is that compute costs recur. A 2% lift that looks great in a screen recording can turn into a monthly bill you can’t justify. Before you scale, decide what quarter of progress you’re buying: coverage, reliability, or speed.
Choosing the next lever when you can only afford one quarter of progress
That “quarter of progress” is the moment you stop asking “what’s best?” and start asking “what breaks the release?” If support tickets come from one slice—one document type, one customer tier, one language—buy coverage: collect 50–200 real cases from that slice, label them the way users judge success, and add retrieval sources that match their wording. If failures come from the model ignoring a required field or drifting from your format, buy structure: function calls, schemas, constrained UI inputs, and a two-step pipeline that extracts before it decides.
Compute is the right lever when you can point to a specific ceiling: you’re truncating the evidence, you need a heavier reranker to surface the right chunk, or latency is forcing you to under-run the model. Otherwise you’ll pay for nicer answers on the same missing information.
Pick one lever and write the acceptance test in product terms—“this screen, these inputs, this outcome”—so the spend can’t hide behind a benchmark bump. The next conversation is how much capability that test can realistically buy.
What capability gains are realistic next—and how to talk about them internally
That acceptance test is also your best language for setting expectations: name the slice, the workflow step, and the failure you’re targeting, then promise improvement there—not “a smarter model.” A realistic next gain is usually narrower: fewer misses on one document type, higher schema compliance on one API call, or better recall for one customer segment’s terms.
When you brief stakeholders, tie the spend to a measurable product outcome (“reduce ‘can’t process this file’ tickets by 30%”) and a cost shape (one-time labeling vs ongoing inference). The constraint to say out loud: even a strong lift on the target slice may not generalize, and compute-driven gains can raise your monthly bill before you see retention move.
