TDD with an agent: the test pays, the ritual does not

Two recent measurements look like a fight.

One says tests written by an LLM detect faults in 69% of cases, against 17.2% for general-purpose human tests. The other says telling the agent to follow TDD produced no discernible difference against not telling it, and cost three to eight and a half times more tokens.

Both are right, and the reconciliation is this post's subject: they measure different things. One measures the test. The other measures the ceremony around the test.

This text separates the two, shows why coverage is the wrong metric, and ends on what to use in place of the ritual. It is the verification layer of harness engineering, measured instead of assumed. It continues why the agent cannot review its own work: there it got established that the agent needs an external signal. Here you see that the test is that signal, and it works, while red-green-refactor is not what makes it work.

The result is stronger than the public discussion suggests.

In LLM vs. Human Unit Tests: Fault Detection on Real Python Bugs, by Phouvadeth Vathana, Prapti Bhatt, Rishi Patel and Nasir U. Eisty, tests generated by an LLM detected faults in 69% of cases, against 17.2% for general-purpose human tests. The difference is solid in statistical terms: Fisher exact with p below 0.001 and an effect size of 1.10 on Cohen's measure (arXiv 2606.08588, 7 June 2026).

The method matters for knowing the reach. It covered 29 historical bugs from BugsInPy, plus function-level benchmarks on the python-slugify and packaging libraries. The model was Gemini 2.5 Flash, with context augmented by retrieval, and the authors record that the retrieved context was critical to the result.

It is a strong and narrow study at once: one model, one language, 29 bugs. It supports no law that "AI writes better tests than humans". It supports that, under those conditions, the artifact the machine produced found four times more defects.

How much of that is already happening

It is no laboratory scenario. A study presented at the 23rd International Conference on Mining Software Repositories analyzed 2,232 commits with test changes, from the AIDev dataset. It found that 16.4% of all commits that add tests came from AI (arXiv 2603.13724, Yoshimoto, Fujita, Horikawa, Feitosa, Kashiwa and Iida, March 2026).

The same work describes those tests' structural profile: longer code, higher assertion density and lower cyclomatic complexity than the human ones, since they use linear logic. The coverage achieved is comparable.

Hold on to the assertion density. It looks like quality and it returns in this post as a risk: many assertions guarantee nothing about any of them checking the right thing.

The detail that reorganizes the discussion sits in the same study's coverage numbers.

The LLM-generated tests had lower coverage than the human ones: 84.8% against 88.5% on lines, and 75.2% against 82.1% on branches. Less coverage, four times more detection.

The authors themselves conclude that coverage metrics on their own measure no test quality. That has a direct consequence for anyone using coverage as a continuous integration gate: the number that blocks the merge is the one telling you least about finding defects.

Coverage measures whether the line ran. It measures nothing about whether anything got checked while it ran. A test that calls the function and asserts nothing covers everything and finds nothing.

Now the other end, and it is uncomfortable for anyone defending TDD on principle.

In TDD inside the agent loop, published on 10 August 2026, Birgitta Böckeler investigated whether instructing an agent to follow TDD inside its own loop improves the resulting code and tests. The conclusion is direct: "there was no clearly discernible difference between the TDD flow and the non-TDD flow" (martinfowler.com).

The details make the picture worse for the ritual. Across five batches of tasks, the solutions without TDD placed first and second on the small and medium tasks; the ones with TDD, third and fourth. Only when the prompt put design review in the foreground did a TDD solution place first.

And the cost: the TDD path consumed 3 to 8.5 times more tokens. It is a difference that changes budgets, and the subject of where the money leaks in production covers what that means at scale.

The table is the post's thesis. The artifact is good. The ceremony failed to pay for itself.

Where the ritual still makes sense

Before throwing it out, three honest caveats. Two come from the study itself.

When the prompt asks for design review. It was the one condition where a TDD solution placed first. That suggests the gain came from somewhere other than the order of the steps, and instead from the explicit attention to architecture. If attention produces the effect, asking for it on its own works, with no full cycle required.

When the task is large. The ranking result cited holds for small and medium tasks. Nothing in the material supports the conclusion for long refactors or features crossing many files, and the cost of being wrong there is another matter.

When a human writes it. None of that measures TDD done by a person. The method was born to give rhythm and short feedback to whoever codes, and the measurement here covers instructing an agent to imitate it inside its own loop. They are different questions, and conflating them would be the same error this post charges others with.

The evidence authorizes something narrow and useful: telling the agent to follow TDD, on a small or medium task, showed no gain and cost a lot. It authorizes no claim that TDD died.

Böckeler names five behaviors that explain why the ritual fails to take.

Note the pattern: four of the five are the agent failing to do the discipline it was told to do. TDD failed as no method; it went unexecuted, instruction and all.

The fifth, the tautological test, is different and graver. It deserves its own section.

A tautological test asserts what the code does, not what it should do. Someone writing the test while looking at the implementation produces that: a copy of the current behavior, wearing the look of verification.

With an agent, the problem worsens for a structural reason: when the same model writes the code and the test, both share the same blind spot. If it misunderstood the business rule, it will implement wrong and check wrong, with dense assertions, flawless syntax and high coverage.

Here the assertion density the MSR study measured stops being a signal of quality. Twenty assertions about the wrong behavior remain twenty wrong assertions, and the high number gives the opposite impression. Density measures how much the test asserts, not whether it asserts the right thing, which is coverage's defect one floor up.

It is the same limit described in why the agent cannot review its own work, applied to the test. The agent never finds its own error because the error was born from its understanding. Writing the test first solves nothing, because the understanding is the same at both moments.

The defense has nothing to do with process, it has to do with source: the test needs to come from something the model never invented. A specification written beforehand, a real input-output example, a bug reproduced with production data, an API contract. That is the very thing the detection study observed when it called the retrieved context critical, since the tests were good because they had somewhere to take the truth from.

One second-order piece of evidence is worth more than argument: someone had to build a guardrail for this.

tdd-guard is an open source tool whose declared function is enforcing TDD on Claude Code by blocking actions when the agent tries to skip tests or implement past what the task needs. Meaning: someone automated the containment of two of the five failure modes Thoughtworks named.

If the discipline held up through instruction, the tool would need no existence. It is the confession that saying "follow TDD" in the prompt makes the agent follow TDD in no way.

And a more useful reading sits in that. A lock depending on no agreement from the model is the definition of a guardrail rather than an instruction, and the distinction sits in where to lock before the agent acts. If you want genuine TDD in the loop, you need mechanism, not a request. The question left is whether it pays.

Böckeler proposes four substitutions, and all of them share a property: they measure results instead of conformance to the process.

Mutation testing. It alters the code on purpose and checks whether any test breaks. A test that fails to detect the mutation was checking nothing. It is the direct answer to the coverage problem: it measures detection, not execution.

Access to static analysis. It serves as a continuous refactoring trigger, filling the place the cycle's refactoring step should fill and never does.

Approved scenarios. Instead of demanding a test before the code, keep a list of human-validated scenarios the system has to satisfy. The truth comes from outside the model.

Structural review. Track file count and modularity drift, to notice when the design is degrading, which is the problem TDD promised to prevent through the local path and never did.

Testing and evaluating became synonyms in conversation about agents, and they are not. A test verifies deterministic behavior of code. Evaluating an agent on the real task measures non-deterministic output. The techniques above belong to the first family.

The part that stays standing is short, and it fits in four instructions.

Give it the truth from outside. A specification, an input-output example, a reproduced bug, a contract. The detection study showed the retrieved context was critical, and it is the difference between the test checking the specification and the test checking the implementation.

Ask for assertions about behavior, not about structure. "Check that the function rejects an email with no at sign" produces a useful test. "Write tests for this function" produces whatever the model thinks the function does. The principle is the same as in tool design for agents: what you hand the model defines the ceiling of what it can do.

Do not use coverage as a target. If the number becomes the goal, the agent reaches it without finding defects, and a measurement now exists showing that higher coverage coexisted with four times lower detection.

Run mutation testing when the cost justifies it. It is the only measure on the list that answers "are these tests good for anything?" without depending on judgment.

The order matters. The first three are free and they change the result; the fourth costs machine time and it holds for code that cannot break. The operating traps that show up along the way sit in the traps in a real project, and integrating this with commits and review sits in git with an agent.

<!-- [PERSONAL EXPERIENCE] -->

I went to look at the repository before writing this section, because an honest answer required data rather than recollection.

The blog has 24 tests across five files: frontmatter, link guard, newsletter, publication date and search. Looking at the git history, all of them landed in the same commit as the feature they test. None of them had a red commit first. TDD never happened here, by accident or otherwise.

And the most recent case is the most interesting, because it contradicts what I expected. I wrote the five link guard tests after the implementation, and they still produced the benefit TDD promises: while trying to test, I discovered the function sat inside a large module the test runner could not import. The way out was extracting the logic into a module of its own, pure, with no runtime dependency.

Meaning: the pressure for testability improved the design. Except that it arrived at the moment of writing the test, not at the moment of writing the code first. The design gain TDD claims appeared with no ritual.

The explicit judgment: I keep writing tests afterward, and I keep considering tests non-negotiable. The thing I abandoned is the idea that the order is what produces the value. And the uncomfortable part is that 24 tests for a site this size is few, since the gap lies in the coverage of decisions that still have no test, not in the method.

Is imposing TDD on the agent worth it?

The available measurement says it fails to pay off. In TDD inside the agent loop (August 2026), the TDD flow produced no discernible difference against the non-TDD flow, and it consumed 3 to 8.5 times more tokens. The test keeps paying; the order it gets written in showed no gain.

Are AI-generated tests reliable?

Better than the public discussion suggests. In a study with 29 real bugs from BugsInPy, tests generated by an LLM detected faults in 69% of cases, against 17.2% for general-purpose human tests. The decisive condition was the retrieved context: the tests were good because they had somewhere to take the specification from.

Does high coverage mean good tests?

No. In the same study, the LLM tests had lower coverage than the human ones (84.8% against 88.5% on lines) and detected four times more faults. Coverage measures whether the line ran, not whether anything got checked while it ran.

What is a tautological test?

It is the test that asserts what the code does, instead of what it should do. With an agent, the risk rises because the same model that wrote the code writes the test, and both inherit the same understanding, the times it is wrong included.

How do I stop the agent from gaming the test?

An instruction in the prompt is not enough, and the existence of tdd-guard, which blocks the agent when it skips a test or implements beyond the ask, is the evidence. Mechanism is what works: the truth coming from outside the model, and the check running in the environment.

Is mutation testing worth the cost?

It is for code that cannot break. It is the only technique on the list answering head on whether the tests detect a change in behavior, which is the question coverage leaves unanswered. For low-risk code, the machine cost seldom pays for itself.

• An AI test finds defects: 69% against 17.2%, with lower coverage.
• Coverage is the wrong metric. It sees execution, not verification.
• The TDD ritual in the agent loop showed no gain and cost 3 to 8.5× more.
• Four of the five failure modes are the agent failing to execute the discipline it was told to follow.
• The tautological test is the grave mode: same model, same blind spot.
• The test's truth has to come from outside the model.

If you change one thing after reading this, take coverage out of the target position. It is the number that blocks merges, it is the one telling you least about finding bugs, and a measurement now exists showing it can rise while detection collapses.