Evals for agents: how to test what has no right answer

In a survey with 1,340 responses, 89% of organizations implemented some form of observability for their agents. Half evaluate what those agents produce.

And 29.5% evaluate nothing at all. Among those with an agent already in production, 22.8%.

Almost everyone wanted to know what happened. Far fewer wanted to know whether it was any good.

The queue moves slower here for a specific reason. An ordinary test compares the output against the right answer, and for a good share of what an agent produces no right answer exists.

An ordinary test asserts equality. You expect 42 and receive 42. The premise is that one correct output exists and only one.

Ask an agent to summarize a report and no correct summary exists. Dozens of acceptable summaries exist and hundreds of bad ones, and no string comparison separates the two groups.

None of that makes the work untestable. It means the question changes. Instead of "is the output equal to the expected one?", you start asking three things: does the output respect the contract, does it contain or avoid what it should, and did the calculable indicators land inside the band.

Separating this from a neighboring layer pays off. The verification loop answers whether the work ran: it compiled, it passed the tests, it broke nothing that worked. An eval answers whether the work was any good. Different questions, and the second is the one almost nobody automates.

Three things get called "eval" and they measure different objects.

The confusion costs a lot in one specific situation. A new model comes out, gains three points on MMLU, and the team switches. Two weeks later, perceived quality has fallen.

A capability benchmark measures a general ability on standardized tasks. It knows nothing about your prompt, your tools, your domain. A model can improve on average and get worse in your case. The only thing that answers that is a test set of your own, which is the middle layer.

Four kinds of check, from cheapest to most expensive. The order matters: start at the top and go down only when you have to.

Contract. Does the output have the agreed shape? Required field present, correct type, value inside range, reference pointing at something that exists. It is deterministic, runs in milliseconds and catches most of the gross failures.

Assertion. Does the output contain or avoid something specific? Does it mention the required source, avoid inventing a field name, avoid returning empty text, respect the size limit. Deterministic too, and where most business rules fit.

Proxy. A calculable metric correlating with quality without being quality. Readability, average sentence length, repetition density, step count. It never says whether something is good, and it says when something changed.

Judge. A model evaluating with an explicit rubric. It is the most expensive and the most fragile, and 53.3% of those who evaluate use it anyway. Tomorrow's post covers when to trust its score, which is why it gets one paragraph here. A judge's output is also generated text, with the fragilities described in prompt injection when the content under evaluation comes from outside.

One data point that contradicts the expectation of full automation: among those who evaluate, the most used method remains human review, at 59.8% (LangChain, a survey with 1,340 responses collected between 18 November and 2 December 2025, published 12 June 2026; accessed 28 August 2026). A mature eval eliminates no person. It decides what the person no longer has to look at.

One test case, whole

An example makes it clearer. An agent that receives an issue and returns a diff.

case: fixes-broken-import
input: "Issue #482: build fails with 'Cannot find module lib/covers'"

contract:
  - output_is_valid_diff: true
  - files_touched_max: 3

assertion:
  - contains_file: "lib/covers.ts"
  - does_not_contain: "node_modules"
  - does_not_change: "package.json"

trajectory:
  - called_tool: ["find_file", "edit", "run_test"]
  - steps_max: 8
  - ran_test_before_finishing: true

proxy:
  - lines_changed: [1, 40]

None of those checks needs a model to run. All of them are ordinary code reading the output and the trace.

Note the trajectory block. It is what separates "the diff is right" from "the agent got there the right way". An agent that returned the correct diff without running the test passes the first three sections and fails this one, and failing is the good outcome, because next time it gets it wrong.

And note what is absent there: no check on the diff being elegant, idiomatic or well written. That is judgment, and judgment enters no automatic gate without someone paying the price of a judge.

An agent can reach the right result by the wrong path. If you look at the output alone, it passes.

The problem is that the wrong path never repeats. It got it right because the tool order happened to work in that case. On the next variation, it fails, and your test set said everything was fine.

What you can evaluate in the trajectory:

• The correct tool got called, rather than one that returned the answer by accident.
• The order of calls matches the expected one, when order matters.
• The step count landed inside the band.
• No tool got called in a loop.

The good news is that this data already exists if you followed the previous layer. The span tree, with invoke_agent at the top and execute_tool for each tool, is the record of the trajectory. Evaluating the path becomes reading what the trace already stored.

Start with the cases that already failed.

It is the cheapest advice in this post and the most ignored. Every bug that reached production is a ready-made test case, with real input and a known result. A set assembled that way reflects your genuine risk, and not an idea of what might go wrong.

Three rules that matter more than the size:

Small and stable beats large and shifting. Thirty cases that never change let you compare rounds. Three hundred cases that change every week let you compare nothing, because the score went up or down with nobody knowing whether it was the system or the set.

A new case enters when a new bug appears. It is the same discipline as this cluster's pillar: the error that escaped becomes a permanent change in the environment. Here the change is one line in the set.

Keep a set you never tune against. If you optimize against the same set you measure with, the number rises without the system improving. Split part of it off and look there only now and then.

An eval in CI dies of one cause: it fails when it should not, the team disables it, and nobody turns it back on.

Three techniques prevent that. They come from current practice reported by people who operate these gates, and not from a controlled study.

What remains is deciding what happens on a failure, and the answer is not always "block".

A contract and assertion check deserves a block: both are deterministic and a failure there is a genuine failure. A proxy deserves a warning, since it signals change, and change can have an explanation. A judge's score deserves a record and human review before it becomes a rule.

That talks to the four metrics of the guardrails layer: nothing says everything you measure has to block something. A metric that blocks without need meets the same fate as a guardrail stopping legitimate traffic.

You need no framework for the first eval. You need three files.

A file with the cases. Start with five, taken from things that already went wrong. Each case has an input and the checks it should pass.

A script that runs it. It executes the agent on each case, collects the output and the trace, and applies the checks. It prints one line per case.

One line in CI. The script runs on the pull request. A contract failure breaks the build. A proxy outside the band prints a warning.

Once that exists, three things become possible. You know whether the prompt change made something worse. You know which case broke, rather than knowing only that something broke. And you have somewhere to add the next bug.

One decision to make early, because it is expensive later: store the result of every round, with the date and the version of what got tested. Without a history, you compare against memory, and memory always thinks things were better before.

A framework enters when the case count passes a few dozen, or when you want to compare providers in a matrix. Before that, it solves a problem you do not have yet, by the same reasoning as never connecting a tool you will not use.

What follows is what each tool documents and what users report. I tested none of them, and I treat this as a decision map, not as an evaluation of my own.

One market fact matters for the choice. On 9 March 2026, OpenAI announced the acquisition of Promptfoo, with integration into OpenAI Frontier, launched on 5 February that year. Ian Webster and Michael D'Angelo founded Promptfoo in 2024, over 25% of Fortune 500 companies use it, and OpenAI stated it continues developing the open source components (OpenAI, accessed 28 August 2026).

The judgment here is lukewarm on purpose: the acquisition invalidates the tool in no way, and the open source history continues. It is a data point to weigh for anyone needing neutrality between model providers, and irrelevant for anyone who already picked one.

<!-- [ORIGINAL DATA] -->

Text is the hardest case to evaluate, because no right answer exists. And this operation still runs an automatic gate before each publication.

What it measures:

The seven posts of this cluster, measured:

The gate failed two of seven. And the two cases teach more than the five approved ones, because they failed for opposite reasons.

The prompt injection post landed at 60.1 with an average sentence of 18.4 words. Sentences too long. The fix would be writing shorter, and I chose against rewriting because the subject asked for the chaining.

The observability one landed at 61.6 with an average sentence of 13.4 words, the shortest in the cluster. There the prose was fine and the metric fell anyway, because names like gen_ai.usage.cache_read.input_tokens count as long words. Writing shorter would fix nothing; only removing the content would.

That is the previous section's tolerance band, found in practice before I read the recommendation anywhere.

A footnote with a charm of its own. This post, about evals, passed the gate before going out: Flesch 70, the highest of the eight, an average sentence of 14.2 words, seven em dashes, no worn phrases. The high number has a prosaic explanation rather than a literary one. Here there is almost no technical name with dots and underscores to weigh the count down.

And one detail appeared that only shows up when the text talks about itself. I wrote the exact value with one decimal place, ran the measurement again, and it had changed, because the sentence announcing the number entered the number's own count. I rounded to 70 and moved on. It is the miniature version of the problem of measuring a system that reacts to the measure, and one more piece of evidence that the proxy measures what it measures.

And the confession that remains: none of that measures whether the post is good. It measures format, tics and proxies. If I write three thousand correct, readable words with nothing to say, the gate approves. Quality evaluation stays mine, and the gate exists to spare me from arguing about what a counter settles.

The layer has a cost, and cases exist where it never pays for itself.

A task that runs once. Building a set, a script and a gate for something that happens once is more work than the task. Look at the result and move on.

A prototype whose output shape still changes every week. A test set presumes the output has a stable shape. While the shape shifts, the set breaks for the wrong reason and teaches people to ignore the gate.

Low volume with a reviewer available. If five items come out a week and someone looks at all of them, human review is already the eval. Automating there trades something that works for something that needs maintenance.

The trigger for building is volume or repetition. Once you start looking at the same thing for the twentieth time, or once the number of outputs passes what you can review, the arithmetic flips.

And one case makes it worth building even at low volume: when the error is expensive. One output a week that could take production down justifies a gate, even where human review could handle the volume.

One last caveat about what the layer misses. An eval measures the output against criteria you chose. If the criterion is wrong, the gate approves the wrong thing with a high score, and with consistency. No metric warns you that the question was a different one.

What are evals?

They are tests for systems whose output is not deterministic. Instead of comparing against a single expected answer, they assess whether the output respects a contract, contains or avoids defined elements, and keeps calculable indicators inside a band. They run against a stable set of cases, in CI most of the time, and they serve to detect regression when the prompt, the model or the tools change.

What is the difference between an eval and a benchmark?

A capability benchmark, such as MMLU or HumanEval, measures a general ability of the model on standardized tasks. An application eval measures your system in your domain, with your prompts and tools. The distinction is practical: a model can rise on a benchmark and get worse in your case, so switching models based on a benchmark, with no test set of your own, is deciding in the dark.

How do I create a test set for an LLM?

Start with the cases that already failed in production, because they have real input and a known result. Keep the set small and stable, so scores from different rounds stay comparable. Add a new case whenever a new bug appears. And split part of the set off so you never tune against it, which lets you notice when the number rose through overfitting.

Should an eval block the deploy?

It depends on the type. A contract and assertion check can block, because it is deterministic and a failure there is a real failure. A proxy deserves a warning rather than a block, since normal oscillation would fail it for no reason. A judge's score deserves a record and a human look. A gate that fails on noise ends up switched off, and a switched-off gate protects less than a gate that only warns.

How do I know whether my test set is good?

Three signals show it is working. It fails now and then, because a set that never fails is measuring nothing. Once it fails, you can say which case broke and why, rather than only that the number fell. And the cases came from real failures, not from scenarios imagined at a desk. A set holding only the happy path approves anything that avoids exploding.

Do I need a paid tool to start?

No. A script that runs the case set, computes the checks and prints the result already delivers almost all the value at the start. The three cheapest checks, meaning contract, assertion and proxy, are ordinary code with no model involved. A paid platform solves aggregation, history and comparison across versions, which are problems of scale.

• 89% instrumented, 52.4% evaluate, 29.5% evaluate nothing. Knowing what happened became standard; knowing whether it was good did not.
• A benchmark measures the model, a metrics framework measures your system. Switching models on a benchmark is deciding in the dark.
• Evaluate the trajectory. Getting it right by accident with the wrong path is a failure that passes the test and returns on the next variation.
• In CI: a tolerance band, a judge with a pinned version, a stable set. A gate that shouts for no reason is a gate someone switches off.
• Not every metric deserves a block. Contract blocks, proxy warns, judge asks for review.
• An eval measures a proxy, not quality. The judgment stays human.

Tomorrow's post covers the kind of check left out here: using a model as a judge, and when its score deserves trust. The map of the five layers continues in the pillar on harness engineering.