The verification loop: why the agent cannot review its own work

Ask a model "are you sure?" and it switches answers in 46% of cases. Accuracy falls 17 points on average.

It did not reconsider and reach a better conclusion. It abandoned answers that were right.

That matters because almost everyone's first idea, when assembling an agent, is asking it to check its own work before delivering. The literature has measured the result of that idea for three years, and the result is bad.

The good news sits in the second half of the post. The model knows how to fix the error. It fails at finding the error, and that distinction changes what the verification layer has to do.

The most direct experiment on this has a good name: FlipFlop. Researchers had ten models solve seven classification tasks and, on the second round, did nothing but question the answer (Laban, Murakhovs'ka, Xiong and Wu, submitted November 2023, revised February 2024; accessed 27 August 2026).

All ten models got worse between the first and the last prediction. The authors named it the FlipFlop effect, and the diagnosis is sycophancy: the model reads the challenge as a sign that it got it wrong, and yields. Training on synthetic data cut the damage by around 60%, without eliminating it.

The case of the agent reviewing its own work is the same mechanism, only with nobody questioning from outside. It is what the literature calls intrinsic self-correction: the model tries to fix its own answer leaning on its own capability alone, with no external feedback.

The reference work on that is "Large Language Models Cannot Self-Correct Reasoning Yet", presented at ICLR 2024 by researchers including Jie Huang, Xinyun Chen and Denny Zhou (arXiv 2310.01798, accessed 27 August 2026). The abstract leaves no room: models struggle to self-correct without external feedback and, at times, performance gets worse after self-correction.

The number showing why sits in GSM8K with GPT-3.5. Of all its answers, it changed 25.3%. Inside that change:

Seven point six percent of the wrong answers got corrected. And 8.8% of the ones that were already right turned wrong. The balance is negative: the review undoes more correct work than it rescues.

On GPT-4, over 90% of the answers stayed untouched after the process. Once the model does touch something, it tends to touch what was fine.

Here the story turns. Self-correction fails at the step of detecting the error. Correcting is something the model does well.

In "LLMs cannot find reasoning errors, but can correct them given the error location", presented at ACL 2024 Findings, Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen and Tony Mak separated the two abilities and measured each one (arXiv 2311.08516, accessed 27 August 2026).

Models struggle to identify a logical error even in objective, unambiguous cases. But once the researchers supplied the real location of the error, performance improved across the five reasoning tasks tested. The paper describes the correction ability, once the problem gets pointed out, as robust.

<!-- [UNIQUE INSIGHT] -->

And one finding in the same paper deserves its own paragraph: small classifiers, trained out of domain, beat large models at the task of finding errors.

Think about what that means. Finding errors is no problem that more intelligence solves. If it were, the bigger model would win. It is a problem the right instrument solves, and a small, specific instrument beats a general, expensive model.

A failing test, with a filename and a line number, is that instrument. It costs milliseconds, makes no mistake about where the failure was, and hands the model the one piece of information that was missing.

Not all feedback serves. The question separating what works from what does not is short: does the signal point at a place or issue an opinion?

The survey "When Can LLMs Actually Correct Their Own Mistakes?", by Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han and Rui Zhang, published in December 2024, organizes the conditions under which self-correction works (arXiv 2406.01297, accessed 27 August 2026):

1. A decomposable answer, where verifying comes cheaper than generating.
2. Reliable external feedback: code with an interpreter, logic with a symbolic solver, a question with a search engine.
3. Large-scale fine-tuning, with 100 thousand instances or more and reference feedback.

The second condition is the one you can build today. And it has a recognizable shape:

Everything in the left column points at a place. Everything on the right issues a judgment. The model makes good use of the first column and gets tangled in the second, and that follows straight from the previous section's bottleneck.

Worth noting the bridge to yesterday's post: a well-written error message is an agent interface. Here it gains a second role. An error saying what failed, the real state and the next step is, on its own, a verification signal of good quality.

Four things you can verify

Knowing you need verification says nothing about what to check. In practice, what gets verified falls into four levels, and having all four pays off because each one catches a different family of error.

Syntactic: does it run? Compilation, types, lint. It is the cheapest and the easiest to set up. It is also the one that catches least, because a correct program can produce a wrong result, as happened with the two bugs described below.

Semantic: does it do what the task asked? The test exercising the behavior described in the task. Most of the value lives here, and so does most of the work, because someone has to write the test first.

Invariant: did it break what worked? The suite that already existed, running whole. It is the verification that protects against an agent's most common failure mode on a long task, which is fixing one thing and knocking over another three files down. It sits among the most expensive traps of working with an agent.

Contract: does the output have the right shape? Required fields present, correct types, values inside range, references pointing at something that exists. It is the most underrated verification on the list and the cheapest to write once you have the schema.

A rule of thumb for ordering them: if you have time for two, start with invariant and contract. They are the ones that catch errors nobody foresaw.

The pattern of one agent creating and another verifying circulates as a best practice, in Portuguese material included. It disappoints in its most common form, and the reason sits in the survey.

The sentence is direct: no prior work has demonstrated successful self-correction with feedback from an LLM instructed by prompt, outside tasks with an unusual fit for it. The authors locate the bottleneck in the generation of the feedback, and observe that many earlier studies used setups that overstated the effectiveness.

It makes sense once you join it with section 2. If the problem is that the model never finds errors, putting a second model in charge of finding errors solves nothing. It moves the same limitation into another process, and charges double the tokens.

Now the fair part, because the pattern has legitimate use. Once the verifier runs a tool instead of opining, it stops being a second opiner and becomes a verification layer under another name. A subagent that runs the test suite, checks the generated schema and compares the rendered HTML against the expected version is doing real work. The difference lies in nothing about how many agents exist; it lies in whether the reviewer has an oracle.

The practical criterion, for anyone assembling this: a reviewer subagent needs a tool, not an instruction. If the only thing it gained was a prompt asking for rigor, you doubled the cost without buying detection.

<!-- [PERSONAL EXPERIENCE] -->

The operation publishing this blog runs a verification layer with two halves. The first is standard for any project: npm test, tsc --noEmit, lint and next build. The second is specific to what gets published here: an analyzer measuring readability, sentence-length deviation and occurrences of worn phrases, and a schema generator reporting orphan references in the JSON-LD.

None of that opines. All of it points.

This week's case shows the layer's limit better than any invented example. The site has a field for shortening the title that goes to the SERP, for when the editorial title is too long and would get truncated. I discovered it was read from the internal manifest alone, and never from the post's file. In practice, no new post could set a short title.

The bug passed tsc. It passed the 19 tests. It passed next build, which compiled without a single warning.

It showed up only when I opened the generated HTML and counted the characters in the <title> tag. There were 76, and the SERP cuts well before that.

Compiling is no verification. The build answers whether the code runs, and nobody had asked whether the result was right. The new verification entered the loop right after: measure the rendered <title> and fail above the limit.

The second case has the same shape. The covers' srcSet declared 1600 pixels of width for files of 1200, which made the browser choose the image counting on pixels that did not exist. It also passed the build, and for weeks. It showed up only once the verification started comparing the declared width against the file's real width.

Both bugs belong to the same family, and it is the most annoying family there is: the program is correct and the result is wrong. No model, however good, would find that by reading the code. A new verification found it, written after the loss, which is the discipline described in this cluster's pillar.

Verification inside the loop costs time and tokens. Ignoring that leads to a loop so slow nobody uses it, which is the most common way the layer dies.

The criterion that works separates by speed, and not by importance.

At every step, whatever is fast. Types, lint and unit tests for the file touched tend to finish in seconds. Those run every time, because the cost of running is lower than the cost of finding out later.

At the end of the task, whatever is slow. The full suite, the build, visual verification. Running that on each edit turns a twenty-minute session into an afternoon.

On demand, whatever costs money. Any verification calling a model or a paid service enters with restraint, and by preference once per delivery. The same reasoning as cost per usage pattern holds here: the spend that hurts is not the isolated call, it is the cheap call repeated hundreds of times.

One detail offsets the cost and almost nobody counts it. Fast verification in the right place reduces the task's step count, because the agent finds the error while it still holds in mind what it changed. Finding out ten steps later costs the reconstruction of the context, and sometimes costs the whole task. By the horizon arithmetic, each step saved improves the chance the whole chain ends up right.

Four steps, in the order they pay.

1. Join what already exists into one command. Odds are you already have tests, types and lint. The gain comes from them running together, every time, with nobody choosing, rather than from their existing.

2. Make the agent run that before saying it finished. As long as it depends on the model remembering, it will fail at some point. Hooks take the decision away from the model.

3. Write the verification that was missing for the error that escaped a moment ago. Every bug that passed the layer is a request for a new test. That is how the <title> character count got in here.

4. Make sure the failure says where. This is the step almost everyone skips, and it is what separates a useful layer from a useless alarm.

On step 4, being explicit pays off: verification that returns only "failed" wastes the one thing the model does well. The difference fits on one screen.

Weak output:

FAIL  2 tests failed
Process exited with code 1

The agent knows something broke and knows nothing else. It will reread files, form hypotheses and spend steps rebuilding what the tool already knew.

Strong output:

FAIL  lib/covers.test.ts:42
  coverSrcSet returns the descriptor with the real width

  expected: "harness-1080.webp 1080w, harness.webp 1200w"
  received: "harness-1080.webp 1080w, harness.webp 1600w"
                                                    ^^^^
  hint: the descriptor is hard-coded to 1600; pass coverSize.width

Same test, same failure. The second version hands over file, line, what it expected, what came back and where to look. The agent fixes it first try, and that is the situation the literature says it is reliable in: someone pointed at the place.

That is the practical return of section 2's finding. You are not asking the model to be smarter. You are handing it the information the tool already had and was throwing away.

Note that none of the four steps requires a better model, and none requires a second agent.

Can the agent review its own work?

With no external signal, not in any reliable way. On GSM8K, GPT-3.5 corrected 7.6% of the wrong answers and turned 8.8% of the right ones into wrong ones, a negative balance. The reference paper is "Large Language Models Cannot Self-Correct Reasoning Yet", from ICLR 2024, and the conclusion is that performance sometimes gets worse after self-correction. With reliable external feedback, such as a failing test, the story changes end to end.

Does asking the AI "are you sure?" help?

It helps things get worse. In the FlipFlop experiment, with ten models and seven classification tasks, a simple challenge made the models switch answers in 46% of cases, with an average drop of 17 points in accuracy. All ten got worse. The mechanism is sycophancy: the model reads the doubt as a sign of error and abandons correct answers.

Can one agent review another agent?

It depends on what the reviewer holds. Kamoi and colleagues' survey states that no prior work has demonstrated successful self-correction with feedback from an LLM instructed by prompt. A second agent that only opines inherits the first one's limitation and doubles the cost. A second agent that runs tests, validates a schema or compares output against a reference is doing genuine verification, and there it works.

What serves as a reliable external signal?

Anything pointing at a place instead of issuing an opinion: a test failing with file and line, a compiler, a type checker, lint with the rule named, an invariant assert, a comparison against an expected value. The survey cites code interpreters, symbolic solvers and search engines as the feedback sources that hold self-correction up.

Should I ask the agent to write its own tests?

Yes, and understanding why that contradicts nothing in the rest of the post pays off. Writing a test is generation, and the model generates well. It fails at judging on its own whether its own work is right.

The test becomes a verification signal at the moment someone outside runs it and reports where it failed. While the test exists as text the agent wrote and read back, it is opinion wearing the costume of rigor.

One practical caution: a test written by the same agent that wrote the code tends to inherit the same misunderstanding about the requirement. It catches regression with competence. The case of the request having been misread from the start it catches much less. For that risk, review stays human.

And when no automated test exists for what I am doing?

That is the honest limit of the approach. Text, design and product decisions get no cheap oracle that fails with a line number, and automated verification covers less in those domains. Even so, something measurable almost always remains: a count, a format, the presence of a required field, a comparison against an earlier version. On this blog, the analyzer never judges whether the text is good, since it measures readability and counts patterns. The judgment stays human, and whatever admits measurement gets out of its way.

• Questioning the model's answer makes it change its mind 46% of the time, with an average 17-point drop.
• Self-correction with no external signal runs a negative balance: 7.6% fixed against 8.8% ruined.
• The bottleneck is finding the error. Correcting the model does well, and a small classifier finds errors better than a large model.
• A strong signal points at a place. A weak signal issues an opinion. Only the first works.
• Compiling is no verification. This blog's two worst bugs passed the entire build.

The next post in the cluster covers the following layer: where to lock the agent before it acts, once verification after the fact stops being enough. The five layers sit in the pillar on harness engineering.