WTF is an Eval

A team ships an agent into customer support. It scored 94% on the benchmark they picked, so the review was short. Two days later it refuses a routine refund because the customer asked for it sarcastically, and the agent read the sarcasm as a reason to escalate.

Nothing about the model was wrong. The 94% was real. It just measured a different thing than the one that broke.

That gap has a name in most teams: we should write some evals. Almost nobody agrees on what that means.

A benchmark grades the model. An eval grades your system.

An eval is a test for a system whose output you cannot predict in advance. That is the whole definition, and everything awkward about evals comes out of it.

A benchmark is a fixed set of questions with known answers, run against a model, scored once. It tells you something real about the model in isolation. It tells you nothing about your retry logic, your tool descriptions, your context budget, your system prompt, or the twelve rounds of tool calls between the user’s message and the answer.

You do not ship a model. You ship a model plus a harness, and the harness is the part you wrote. When the refund agent failed, the model had no trouble reading the tone. The harness routed anything with a negative sentiment flag to escalation, and nobody had a test for that path.

So the useful split is:

The benchmark is a fact about the model. The eval is a fact about your system. Only one of them changes when you edit your own code.

The scores are saturated, and that is not a scandal

Public leaderboards are still worth reading. They are just worth reading for what they are.

Two things degrade them over time. Saturation: once the frontier clusters above 90%, the test cannot separate the models any more, and the remaining gaps are inside the noise. Contamination: older questions leak into training data, so a right answer might be recall rather than reasoning, and you cannot tell which from the score.

None of this makes the numbers fake. It makes them a fact about a model on a fixed set of questions, which is exactly what they claim to be. The mistake is downstream, when a vendor’s 94% becomes your deployment decision.

Your tests assume three things that stopped being true

Traditional tests carry assumptions that are free in normal software and false here.

Deterministic output. Same input, same output, or it is a bug. With a model, the same prompt can produce ten different answers that are all correct. If your test asserts on an exact string, you are testing the sampler.

Binary correctness. Pass or fail, nothing between. But an answer can be right and cost forty tool calls to get there. That is a real regression and no assert catches it.

Static behavior. The system behaves today as it did yesterday. Your code does. The model behind the API does not.

Every practical eval design is a workaround for one of those three.

Assert on shape and cost, not on the answer

Here is what that looks like in code I actually run. Aster ships a small set of live eval cases, and a case is not “the answer must equal this”. It is four constraints:

pub struct Case {
    pub name: String,
    pub prompt: String,
    /// Substring the answer must contain.
    pub must_mention: Option<String>,
    /// Tools that must run at least once.
    pub calls: Vec<String>,
    /// Tools that must never run.
    pub avoids: Vec<String>,
    /// Per-tool ceilings.
    pub at_most: Vec<(String, usize)>,
}

must_mention handles nondeterminism. The model can phrase the answer however it likes as long as the load-bearing word is in there. avoids catches the dangerous shape: a read-only question that calls edit_file has failed even if the prose is perfect. at_most is the one that earns its keep.

One case asks what activation events the VS Code extension declares, and caps find_files at five calls. That number is not arbitrary. editors/ is gitignored, and before I added an ignore fallback every discovery tool came back empty, so the model kept guessing at paths. Nine calls, still wrong. Observed runs now land between one and three. The ceiling sits above the noise and still fails the moment the fallback breaks again.

That is the property worth designing for:

A regression shows up in the round count before it shows up in the answer.

The model is good enough to paper over a broken harness for a while. It will burn eight extra tool calls compensating for a search that returns nothing, and hand you a correct answer at the end. Assert only on the answer and you will not notice until the compensating stops working.

The same idea runs over recorded sessions rather than fixed cases. A round is barren if every tool result in it was empty, an error, or a path that does not exist. Counting barren rounds across real transcripts tells me which tools answer nothing, which is a question no benchmark will ever ask about my code.

The rubric is the eval, not the judge

Some things cannot be asserted with a substring. Tone, completeness, whether an explanation is actually correct. Human review does not scale past a few dozen samples a week, so the common move is to have a second model grade the first.

It works better than it sounds like it should. In the MT-Bench work (arXiv 2306.05685), GPT-4 judging model responses agreed with human preferences more than 80% of the time, which is roughly the rate two humans agreed with each other. The ceiling on judging is not the judge, it is that people disagree too.

The thing that moves the number is the rubric. A judge prompt that says “rate this answer 1 to 5” is asking the model to invent a scale, and it will invent a different one each run. A rubric that defines each point with a worked example is asking it to classify against a fixed reference. Same judge model, very different reliability.

So write the rubric like a spec: what counts as a 3, what specifically drops something to a 2, one example of each. Then check the judge against a small human-labeled set before you trust it on the other ten thousand. A judge you have never calibrated is a random number generator with good manners.

The model moves while you sleep

This is the assumption that breaks hardest. Your code is frozen. The endpoint behind it is not.

The documented case is arXiv 2307.09009, which tracked GPT-4 across three months in 2023. On one task, identifying whether a number is prime, accuracy went from 97.6% in March to 2.4% in June. Be honest about what that means: follow-up analysis showed a lot of the drop came from the model no longer producing its reasoning before answering, and from formatting changes that broke the parser, not from the model forgetting arithmetic. The capability was mostly intact. The behavior was not.

That distinction does not help you. Your pipeline consumes behavior. If the model stops emitting a code fence, your extractor returns nothing, and it does not matter that the answer was in there somewhere.

Which is why the version pin is not a fix either. Pinning buys you stability until the version is retired, and then you take the whole delta at once, on a deadline, instead of in pieces you could have measured. The eval suite is what makes that migration a measurement instead of a gamble.

What this costs, and what it does not buy you

The honest version: this is a standing cost, not a launch task. A suite you ran once at ship time is a photograph of a system that has since moved. Teams that treat it as a checklist find out how far it drifted at the same moment their users do.

And evals only test what you thought to encode. The refund case at the top is the point. Nobody writes an eval for sarcasm before sarcasm breaks something. The suite grows out of incidents, which means it is always slightly behind reality, and no amount of coverage closes that gap completely. That is the real limitation and it does not go away.

So the number I would watch first is not a score at all. It is how often a human has to step in and correct the agent. That rate does not care whether the cause was the model, the prompt, a tool that started returning empty, or a case nobody imagined. It moves before anything else does, and it moves for all of them.

Point-in-time certification is over

When the agent fails in production, the thing that failed is almost always yours. A tool description that argued nine to one for gathering. An ignore rule that made discovery return nothing. A sentiment flag wired to the wrong branch. The model was fine in all three, and the leaderboard had nothing to say about any of them.

The best test for your agent is not a set of PhD-level physics questions. It is your own messy, ambiguous, half-typed production data, run on a schedule instead of at a milestone.

So the question to sit with is the one the version pin does not answer. If the model behind your agent shifts behavior tonight, while you are asleep and your code has not changed by a single line, do you find out before your customers do?