The question that sinks most AI projects in front of the executive committee isn’t “Does it work?” It’s “How do you know it works?” Without an answer backed by numbers, AI in production is an act of faith that someone will eventually pull the plug on.
Why testing it by hand isn’t enough
An AI system doesn’t fail like classic software. It doesn’t crash — it starts giving worse answers. A change to the instructions that fixes one case breaks three others. A model update from the provider shifts behavior without anyone touching anything. And the team testing it by hand always tries the same five examples — exactly the ones that already work.
What you need is your own test bench and a way to see what’s really happening in production.
The four pieces
A custom evaluation set. Between 50 and 300 real cases from your company with the expected result, including the hard ones and the ones that once went wrong. It’s built from historical data and grows with every incident. It’s the most valuable asset in the project, and no provider can supply it.
Automated judges. For outputs without a single right answer — a summary, a reply to a customer — we define what makes an answer valid and use a model as the evaluator against those criteria, checked against human judgment on a sample. It isn’t perfect; it’s what lets you go from five tests to two hundred.
Regression detection. Every change — instructions, model, tools — runs through the full set before it ships. If the score drops, it doesn’t ship. This is the practice that turns a pilot into a maintainable system.
Production traces. What the user asked, what the agent decided, which tools it used, what they returned, how long it took, and what it cost. Without traces, there’s no way to diagnose problems when something goes wrong — and no defense in a complaint or an audit.
What we measure
Beyond quality, three indicators that tend to get overlooked: completion rate (how many tasks the agent finishes without help), escalation rate (how many end up with a person), and cost per resolved task, which ties into tokeneconomics. An agent with excellent quality and a 70% escalation rate isn’t solving the problem.
For systems covered by the EU AI Act, much of this material — traces, metrics, change logs — is also the evidence the regulation requires. We cover it in detail in AI compliance and governance.
When to set it up
Before building the agent, not after. The right order is: define how success will be measured, build the evaluation set, and then develop. A project that reaches production without evaluation can’t be improved — only rebuilt.
You don’t need it when the system is internal, occasional, and low-stakes: human review is enough there.
How we approach it
We build the evaluation set with the team doing the work by hand today — they’re the ones who know what a good answer looks like — and set the bar before writing any code. Tracing is instrumented from day one. The result is a dashboard tracking quality, escalation, and cost over time: the material you need to defend or cancel the investment in front of the committee.
Frequently asked questions
How many cases do you need to get started? Fifty well-chosen cases are enough to catch the big regressions. What matters is that they’re real and include the hard ones, not that there are many of them.
Can one model judge another? With explicit criteria and checked against human judgment on a sample, yes — and that’s what makes evaluation at scale viable. Without that check, no.
Don’t the providers’ platforms already do this? They provide the instrumentation. The cases, the quality criteria, and the bar are yours — and they’re the part that really decides.