The Eval Set Is the Product Spec

Two engineers can read the same product brief and build two different systems. One treats "handle refund requests sensibly" as permission to refund anything under a stated limit. The other reads the same sentence and escalates everything above a much smaller amount. Neither engineer is careless. The brief simply never decided the question — and with a language model in the loop, every undecided question becomes behaviour in production, discovered by a user.

That is why we treat the eval set — the collection of test cases an AI system must pass, with the scoring rules attached — as the real product specification. A roadmap describes intent. An eval set forces the decisions the roadmap skipped: which inputs count, what a correct outcome looks like in the environment (not just in the reply text), and who judges the borderline cases. Write the evals first and the argument about what to build happens once, in a document, instead of repeatedly, in incidents.

Start from failures, not from features

An eval set written from imagination tests what the builders already believe. An eval set written from failures tests what the system actually does wrong. The method that experienced practitioners converge on starts with error analysis: collect real traces — the full record of what the system received, did, and returned — and read them, one by one, noting the first place each one went wrong.

Those notes then get grouped into a failure taxonomy: a short, named list of the distinct ways this system fails, with a count next to each one. The counts are the point. They replace "the model is unreliable" — a claim nobody can act on — with "most failures are the system citing a policy it did not retrieve," which an engineer can fix this week. Hamel Husain and Shreya Shankar's widely used evals FAQ puts error analysis at the centre of the discipline for exactly this reason: you cannot choose what to measure until you have looked at your own data and named what breaks.

In practice the starting set is small. Anthropic's engineering guidance on evaluating AI agents suggests beginning with a few dozen tasks drawn from real failure modes — the manual checks the team already runs, the bug reports, the support queue — rather than waiting for a large, polished benchmark. A small set built from real failures beats a large set built from guesses: every case in it already mattered once.

Two suites, two different jobs

A mature eval setup keeps two suites apart, because they answer different questions.

A capability suite asks what the system can do at its best. It is deliberately hard: pass rates start low, and the team works to raise them. A regression suite asks the opposite question — does the system still do everything it already did yesterday? That suite should sit near a perfect pass rate, and any drop is treated as breakage, not as noise.

The two connect over time. When a capability case starts passing consistently, it graduates into the regression suite, where it stops measuring ambition and starts guarding against backsliding. That graduation is what turns progress into something permanent. Without it, teams re-win the same ground after every model change, prompt edit, or retrieval tweak — and usually find out from users that they lost it.

One distinction matters for agents in particular. Because model output varies between runs, a single success proves little, so practitioners track both the chance of succeeding at least once across several attempts and the stricter measure of succeeding on every attempt. A demo cares about the first number. A production system that moves money or changes records lives and dies by the second.

Grade the outcome, and pick the grader honestly

For each task in the set, something has to decide pass or fail. That decision-maker is the grader, and choosing it is an engineering decision with trade-offs, not a default.

Code-based graders — checks on the final state, test suites, schema and format validation — are fast, cheap, and consistent. Use them wherever the outcome can be verified mechanically: did the record actually change, did the tool get called with valid arguments, did the answer stay inside the permitted structure? Model-based graders, where another model scores the output against a written rubric, handle the cases code cannot: tone, completeness, whether a summary covers what mattered. They are flexible, but they are themselves models, so they earn trust only by being checked against human judgements on a labelled sample — and re-checked when anything changes. Human review remains the reference standard: slow and expensive, which is precisely why it belongs at the calibration points rather than in the loop for every case.

Two habits keep graders honest. First, grade the outcome in the environment, not the story the system tells about itself. An agent that announces a booking it never made has failed, whatever its transcript claims. Second, keep most judgements binary — pass or fail against a stated criterion. A five-point "quality" score feels nuanced and behaves like noise: two reviewers rarely land on the same number, and a team cannot gate a release on a score nobody can reproduce. One binary judgement per failure mode, each tied to the taxonomy, is measurable in a way a blended score never is.

The regression gate is where evals pay for themselves

Evals earn their keep at the moment of change. Models get swapped, prompts get edited, retrieval indexes get rebuilt, tools change their behaviour. Each change is a hypothesis — this will make the system better — and the regression suite is how the hypothesis gets tested before users test it for you.

Run as a gate, the rule is simple: a change ships only if the regression suite holds and the capability suite does not fall. That rule changes the politics of a deployment. Model upgrades stop being week-long manual re-checks and become measured comparisons, and whether a new prompt is "better" stops being a matter of taste — the set both sides agreed to, back when it was written as the spec, produces the answer.

The gate only works if the team also reads the transcripts behind the scores. Aggregate numbers say that something changed; the trace of an individual failure says what, and often reveals that the task or the grader — not the system — was wrong. A suite nobody reads becomes a ritual: green, meaningless, and quietly drifting away from what users actually experience.

Where evals mislead

Evals are not a guarantee, and treating them as one creates its own failures.

A suite can saturate: once every case passes, it stops producing information about improvement and only catches regressions — which is useful, but teams that keep optimising against a saturated set are polishing the test, not the product. A suite can also go stale. User behaviour shifts, the underlying data changes, new failure modes appear in production that no case covers; a set that is never refreshed with fresh traces slowly becomes a museum of last year's problems. And a model-based grader can drift or be gamed: systems optimised hard against a rubric learn to satisfy the rubric's letter, which is why human calibration is a recurring cost, not a one-time setup step.

Finally, evals measure what the suite contains — nothing else. Safety, cost, and latency need their own cases and thresholds, deliberately added, or they will be traded away silently in pursuit of the scores the team watches. The honest position is that a good eval set narrows uncertainty; it does not remove it. Production monitoring, staged rollout, and human escalation still have to exist behind it. That is a separate layer of the discipline — and the subject of our next post, on guardrails.

Bring us a problem

If your team is arguing about whether an AI system is ready — or discovering in production what it actually does — the missing piece is usually not a better model. It is a written, testable definition of "ready." Start with Trida AI's intake — a few questions, then talk with founders and experts from Trida AI.