Retrieval & LLM systems · Eight years of production engineering
Your company already knows the answer. It's just buried in six years of documents, tickets and threads.
I build retrieval and LLM systems that put that knowledge in front of the person who needs it, at the moment they need it — and the evaluation that tells you when the system is wrong. Not workflow automation. Production engineering.
Ask the company something
Retrieved to answer it
—
A 2-D picture of something that happens in about 1,500 dimensions, on an invented company's documents. Nothing here is keyword matching — "time off" finds the leave policy without sharing a single word with it.
01 — The problem
The demo always works. That's the trap.
Nearly every stalled LLM project I've seen failed in the same four places, and none of them are the model.
Nobody defined "right"
The pilot was judged on whether the answers felt good in a meeting. With no test set there was no way to tell whether the next change made it better or worse — so every improvement became an argument instead of a measurement.
The data was never prepared
Documents went in whole, or were split at arbitrary lengths. Retrieval returns something plausible from the wrong page, the model writes confidently around it, and trust is gone after two bad answers in front of a customer.
It was built as a demo, not a system
No permissions model, no cost ceiling, no latency budget, no record of what was retrieved and why. Perfectly fine for a prototype, and impossible to put in front of a real user.
Nothing watched it afterwards
Documents change, models get deprecated, the questions people ask drift. Without monitoring, quality decays quietly, and the first person to notice is the person it embarrassed.
Every one of those is an engineering problem, not a prompting problem. That is the entire reason I'm useful here.
02 — What I build
Four shapes of system, and the processes they fix
Most business problems worth solving with an LLM are one of these four, or a chain of two of them.
Answering from your own knowledge
UnderneathChunking strategy, embedding model choice, vector store, hybrid keyword and semantic retrieval, reranking, citation enforcement, permission-aware filtering.
Your policies, contracts, runbooks, past tickets and internal wikis become searchable by meaning rather than keyword — and a model answers from them, citing the source paragraph so anyone can check it in one click.
Typical case: a support team where the answer to 60% of tickets already exists in writing, and the real cost is the fifteen minutes it takes to find it.
Sorting and routing what arrives
UnderneathLabel taxonomy design, few-shot versus fine-tune decision, confidence calibration, human-in-the-loop thresholds, drift monitoring on the label distribution.
Inbound work — tickets, emails, applications, invoices, incident reports — read, categorised, prioritised and sent to the right queue, with a confidence score deciding what a human still needs to see.
Typical case: an ops team spending the first hour of every day deciding what the day is, before doing any of it.
Turning documents into data
UnderneathSchema-constrained output, validation rules, confidence per field, exception queues, reconciliation against the source document.
Contracts, invoices, forms, reports and PDFs converted into validated structured records that land in your database — with the fields the model wasn't sure about flagged rather than quietly guessed.
Typical case: a finance or legal process where someone re-types information that already exists in a file, and the error rate stays invisible until it matters.
Doing multi-step work reliably
UnderneathTool and function design, deterministic control flow around the model, retries and idempotency, cost and latency budgets, a full audit trail of every step.
Processes with several steps and real judgement in the middle — draft this, check it against that, look up the account, escalate when the numbers disagree. Built with hard boundaries on what the system may do unsupervised.
Typical case: a repeated internal process that takes a skilled person forty minutes and is 80% the same every time.
03 — The part most people skip
How you'll know it works, and keep knowing
An LLM inside a business process is a system that runs unattended and makes decisions. I've spent eight years building exactly that, and the alerting that proves it worked.
The difference between a demo and something you can put in front of a customer is almost entirely measurement. It's in every engagement, and it isn't a phase you can drop to save budget — without it, nobody can tell whether the next change helped.
A golden set, built with your team
A hundred or so real questions with agreed correct answers, written by the people who currently do the work. This is the definition of "right", and it's the first deliverable.
Retrieval scored separately
Did the right document come back at all? Most bad answers are retrieval failures wearing a generation costume, and separating the two tells you which half to fix.
Regression on every change
Prompt edits, model upgrades, newly added documents — all run against the golden set before they ship. Improvement stops being an opinion.
Monitoring in production
Answer rates, refusals, retrieval scores, cost and latency per request, plus sampled human review. The same instinct as any pipeline: if it can fail quietly, it will.
04 — How we'd work
Small, measured, and yours at the end
I'd rather prove one process works than plan a transformation. The first engagement is deliberately small enough to kill cheaply if the answer turns out to be no.
Map
Find the process actually worth doing first, and audit whether the data it depends on can support it. Sometimes the honest output of this week is "not yet, and here's exactly why".
Prototype
A working system on your real data, plus the golden evaluation set built with your team. You see a number, not a demo.
Harden
Retrieval quality, permissions, guardrails, cost ceilings, latency, logging, monitoring and the deployment pipeline. The part that makes it a system rather than a script.
Hand over
Runbook, evaluation harness and a working session with your engineers, so your team can change it without needing me. That's the point.
Available separately: a fixed-fee one-week assessment — which process to start with, whether your data is ready, and what it would realistically cost to build. No commitment afterwards, and I'd rather tell you in week one that the answer is no than spend four months of your budget arriving at the same place.
05 — Where this comes from
I came to this from the infrastructure side, which turns out to be the side that matters
For eight years I built backend systems that had to keep running when nobody was watching: an automated trading platform placing orders on the Texas wholesale power market, a US equities trading platform, a national payment gateway I helped break out of its monolith, and the Kubernetes clusters and data pipelines underneath all of it.
That work was never really about the code. It was about knowing what to do when something drifts. Metrics on every stage, alerts tuned to catch an anomalous run rather than a merely slow one, and enough logging to explain afterwards what happened and why.
An LLM inside a business process is the same problem in new clothes. It runs on a schedule or on demand, it makes decisions, it fails in ways that never raise an exception, and its quality decays silently as the world moves underneath it. The people who find that intuitive are the ones who have operated production systems — not the ones who found a good prompt.
So I'm not the cheapest way to get a demo. I'm who you call when the demo has to become something the business can depend on.
Python, Go, Kubernetes, Docker, AWS, GCP, PostgreSQL, MongoDB, Redis, RabbitMQ, Celery, Prometheus, Grafana — and 76 technical posts on DEV, several of them on database and retrieval fundamentals.
06 — Reasonable questions
The ones I get asked first
How is this different from Zapier, n8n or Make?
Those tools move structured data between apps according to rules you write in advance. They're genuinely good at that — if your problem is "when a form is submitted, create a row and send a Slack message", you don't need me.
What I do starts where the rules run out: unstructured company knowledge, judgement in the middle of a process, and answers that have to be checked rather than assumed. That needs a retrieval layer over your own content, a way to measure whether answers are right, permissions so the system can't surface what a user shouldn't see, and monitoring for when quality drifts. None of that is a node you drag onto a canvas.
Why not just give the team ChatGPT and let them get on with it?
Do that too — it's cheap and people learn fast. But a general assistant doesn't know your refund policy, can't see last quarter's tickets, can't be held to a measurable accuracy bar, and can't sit inside a process so the work simply happens without someone remembering to ask.
The value was never the model. It's the model connected to your knowledge, inside your workflow, with evidence that it's right.
Do we need to fine-tune a model?
Almost certainly not, and certainly not first. In most business processes the problem is that the model doesn't have your information, not that it lacks capability — and that's a retrieval problem, which is faster, cheaper and far easier to keep current than training.
Fine-tuning earns its place for consistent formatting, a narrow high-volume classification task, or squeezing cost out of something already proven. It's an optimisation, not a starting point.
What happens to our data?
It gets decided deliberately, early, and written down. Depending on your requirements that means enterprise API tiers with no-training guarantees, a cloud region you specify, self-hosted embeddings and vector storage inside your own infrastructure, or fully local models where the data genuinely cannot leave.
Access control matters as much as storage: retrieval has to respect the permissions your documents already carry, or the system quietly becomes a way to read things you couldn't open directly.
Won't this all be obsolete in six months?
The models will change — that's exactly why they're built to be swapped. What doesn't go obsolete is the work around them: knowing which process is worth automating, preparing the data, defining what "correct" means for your business, and the evaluation harness that lets you upgrade a model on a Tuesday and know by Wednesday whether it helped.
That harness is the durable asset. Everything else is replaceable on purpose.
How do we start without committing to a big programme?
The one-week assessment, or a single prototype on one process. Both are small enough to stop cheaply, and both end with something concrete — a written recommendation, or a working system with a number attached to how well it performs.
Then you decide, with evidence rather than a pitch.
Start here
Tell me about one process that eats more time than it should
info@bshadmehr.meThirty minutes is usually enough for me to say whether this is worth doing — and to be specific about why, if it isn't.
Useful in a first message
- The process, and roughly how many hours a week it costs you.
- Where the knowledge lives today — wiki, tickets, drive, someone's head.
- Whether the data can leave your infrastructure.
- Whether you have engineers who'd own this afterwards.