Fine-tuning isn't how you make a model smarter. It's how you make a proven task cheap.
Once something works reliably on a frontier model, a small tuned model can often do the same job for a fraction of the cost and a fraction of the latency. That's a fixed cost buying a lower marginal cost — which means there's a breakeven volume, and below it you shouldn't do this at all.
Most people who ask me for fine-tuning don't need it. If the model is getting things wrong because it lacks your information, that's retrieval and it's cheaper. Fine-tuning fixes how a model responds, not what it knows.
Rates are illustrative and move constantly. Frontier pricing has fallen repeatedly, and every drop pushes the breakeven further out — which is itself an argument for waiting unless your volume is genuinely high. Your actual bill replaces every number here at the viability check.
Four situations where tuning genuinely wins
All four share a property: the task is already solved, and the problem is what it costs to run it at scale.
Distilling a proven pipeline
You have something working on a frontier model with measured accuracy. A small model trained on that model's outputs can often match it on that narrow task at a tenth of the cost. This is the strongest case by a distance, because the teacher generates the training data for free.
Format and structure that must never vary
When output has to conform exactly — a schema, a house style, a regulated phrasing — prompting gets you to 95% and the last 5% costs more in retries and repairs than tuning would. Tuning is how you stop fighting the format.
Narrow classification at high volume
Routing, triage, tagging, sentiment against your own taxonomy. Small tuned models are extremely good at this and extremely cheap, and the labelled data usually already exists in your ticket history.
Latency that's part of the product
When a person is waiting. A small tuned model can respond several times faster than a frontier model, and sometimes the case isn't cost at all — it's that two seconds is unusable and three hundred milliseconds isn't.
Six reasons I'd tell you not to fine-tune
These come up more often than the cases above. I'd rather lose the work than take money for the wrong intervention.
- The model doesn't know your informationThat's retrieval, not tuning. Training facts into weights is expensive, goes stale the moment the fact changes, and can't cite a source. If the failure is "it doesn't know our refund policy," fine-tuning is the wrong tool at ten times the price.
- The task isn't working yetTuning makes a model more consistently itself on a task. It doesn't discover a capability that wasn't there. If accuracy is 60% with a frontier model and good prompting, tuning a smaller one will not get you to 90%.
- Your volume is lowBelow roughly half a million calls a month, the fixed cost rarely pays back before the pricing landscape moves under you. The calculator above will tell you plainly when this is your situation.
- The task changes oftenEvery meaningful change means retraining, re-evaluating and redeploying. A prompt is edited in a minute. If your requirements move monthly, tuning buys you rigidity.
- You have no evaluation setWithout a measured baseline you cannot tell whether the tuned model is better, worse, or differently wrong. This is non-negotiable and it's the first thing I'd build — for you or for someone else.
- You want it to "sound like us"Sometimes real, usually solved by a good system prompt and a few examples at a fraction of the cost. Try that properly first; if it genuinely fails at scale, then this conversation is worth having.
Four interventions to exhaust before this one
In rough order of cost. Most tasks that look like they need tuning are fixed somewhere in the first three rows.
Better prompting and examples
Unglamorous and frequently sufficient. A well-structured prompt with five good examples closes most consistency gaps, and it's edited in minutes rather than retrained in days.
Retrieval over your own content
If the gap is knowledge rather than behaviour, this is the answer — and it stays current automatically, cites its sources, and updates when your documents do.
A smaller frontier model, untuned
Often overlooked. The cheap tier of a frontier family may handle your task at a fifth of the cost with no engineering at all. Test this before assuming you need custom weights.
Fine-tuning or distillation
Once the task is proven, the volume is real, and the cheaper options have been tried and measured. At that point the case is arithmetic rather than judgement.
The evaluation set comes first, as always
You cannot fine-tune responsibly without a way to prove the result is at least as good as what you had.
Viability check
Your real volume, your real bill, your current accuracy. Ends with a number: the breakeven, computed from your figures rather than the illustrative ones above.
Baseline and data
An evaluation set with verified answers, and training data assembled — usually from your existing history, or generated by the frontier model already doing the job.
Train and compare
Several candidates trained and measured against the same set. The honest outcome is sometimes that the base model wins, and I'll tell you when it does.
Shadow, then switch
Run alongside the existing path on real traffic, compare outputs, then cut over with the old route kept as a fallback. Monitoring and retraining runbook handed over.
Published, and the first step often ends in "don't"
Does the arithmetic work at your actual volume?
$3,500One week · credited against a build
- Your real volume and spend, measured
- Breakeven computed from your numbers
- Cheaper alternatives tested first
- A written go or don't
- Yours to act on without me
Evaluation set, training, comparison, shadow run, cutover.
$20k – $34k5–7 weeks · check credited
- Evaluation set built with your team
- Training data assembled or generated
- Several candidates trained and compared
- Shadow run before any cutover
- Retraining runbook and monitoring
Models improve, data drifts, and a tuned model doesn't update itself.
from $3,800/moCancel any month · 30 days notice
- Scheduled retraining on fresh data
- Re-benchmarked against new base models
- Drift monitoring on inputs and outputs
- Told when to abandon the tune
- Your team trained to run it
The commitment, and the awkward part of it
The tuned model has to match or beat your current accuracy on your own evaluation set before anything switches over. If it doesn't, there's no cutover and you keep the evaluation set, the training data and the benchmark — which have standalone value regardless.
And the part most vendors leave out: a base model may get cheap enough that your tune stops being worth maintaining. That has happened repeatedly over the last two years. If it happens to yours, I'll tell you to retire it — which ends a retainer, and is still the right advice.
Training compute and inference are billed by your provider directly, not resold. Training a small model for a narrow task typically costs hundreds of dollars, not thousands.
What comes up on the first call
Can't we just fine-tune it on our documents so it knows our business?
This is the most common request and the one I most often decline. Training on documents teaches a model the style and shape of them far more reliably than the facts inside them — so you get something that sounds authoritative about your business and still invents details, without a source to check.
Knowledge belongs in retrieval, where it can be updated, cited and permission-controlled. Behaviour belongs in tuning. Mixing them up is the single most expensive mistake in this area.
How much training data do we need?
Less than people expect for narrow tasks — often a few hundred to a few thousand good examples, and quality matters far more than quantity. A thousand carefully checked examples beat fifty thousand scraped ones reliably.
For distillation you frequently need almost no manual labelling: the frontier model already doing the job generates the training data as a by-product of running.
What if a better base model comes out next month?
It will, and this is the genuine risk of tuning rather than a hypothetical one. It's why the ongoing option re-benchmarks against new base models, and why I'd rather you spent nothing than tuned a task with a thin margin.
What protects you is the evaluation set. With one, swapping to a new base and checking whether it beats your tune is an afternoon. Without one, you're guessing — which is how organisations end up maintaining a tuned model that a cheaper base model has quietly overtaken.
Open weights or a provider's tuning API?
Depends on volume, data constraints and whether you have anyone to operate infrastructure. Provider APIs are simpler and involve no serving to manage; open weights are cheaper at scale, work where data can't leave your network, and mean nobody can deprecate your model.
At the volumes where tuning pays at all, self-hosting open weights is usually the better answer — but it's a real operational commitment and I'd rather size that honestly in week one.
Will the tuned model be worse at everything else?
Almost certainly, and that's fine when understood. A model tuned hard for one task generally loses general capability — it becomes a specialist component, not a replacement assistant.
That's why tuned models sit inside a specific pipeline step rather than answering everything. If you need one model for many varied jobs, tuning is the wrong shape and a frontier model with good prompting is right.
Is our training data at risk of leaking through the model?
It's a real consideration, particularly with small datasets where memorisation is more likely. Mitigations exist — deduplication, filtering identifiers out of training data, and testing the finished model for regurgitation, which I do as standard.
If your data is sensitive enough that this is a governing concern, self-hosted open weights keep everything inside your perimeter, and we'd design for that from week one rather than retrofitting it.
Eight years of production systems where cost per run was somebody's line item
I'm Bahman Shadmehr. Automated trading platforms on the Texas wholesale power market and US equities, a national payment gateway I helped decompose while live, and the pipelines, monitoring and cost controls underneath all of it.
Those systems ran continuously, which makes marginal cost a first-class engineering concern rather than an afterthought — you learn quickly that the interesting question is never "can this work" but "what does it cost per run, and what happens when the volume doubles." That's the question this page is built around.
You deal with me from the first call to handover.
Python, Go, Kubernetes, Docker, AWS, GCP, Postgres, Redis, Prometheus. Remote, UTC+3. 76 technical posts · LinkedIn
Send me your monthly model bill and what the task is
Those two facts settle most of it. I'll tell you within a day whether the arithmetic works — and it frequently doesn't, which is a cheaper thing to learn from an email than from a project.
info@bshadmehr.meUseful in a first message
- The task, and whether it currently works.
- Calls per month, honestly — check the bill.
- What you spend on models now.
- Whether you have an evaluation set.
- Whether latency or cost is the actual problem.