Harness engineering: what separates a demo agent from a production one

In March 2026, LangChain's engineering team took its own coding agent, sitting in 30th place on Terminal Bench 2.0, and moved it to 5th. They did not change the model. They changed only what surrounds it.

Two months later, Scale AI's SWE-Atlas measured the same variable and reached the opposite conclusion: harness choice is noise inside the margin of error. Opus 4.6 scores 2.5 points higher inside Claude Code than in a generic agent, and GPT 5.2 goes the other way.

Both studies are right. Understanding why is the difference between an agent that impresses in a demo and one that survives Monday.

Harness engineering is the practice of designing everything that surrounds the model in an agent, so the agent fails less without the model changing. The harness is that surrounding layer: the system prompt, the set of tools the agent can call, the memory it carries between steps, the tests that run before it declares itself done, the limits on what it cannot do, and the telemetry that records what happened.

The term came from a specific place. In February 2026, Mitchell Hashimoto (co-founder of HashiCorp and creator of Terraform) described a habit he had developed working with agents: every time the agent got something wrong, he did not write a better prompt. He changed the environment so that the mistake became impossible by construction. He called it engineering the harness. Within weeks, OpenAI and Anthropic published material expanding on the idea.

The formula that stuck is short: agent = model + harness.

The most common confusion is treating this as prompt engineering with a new name. They are different things. Prompt engineering picks words. Harness engineering writes code and configuration: which tool exists, what it returns when it fails, what runs on its own before a commit, what the agent cannot execute even if it decides to.

If you have ever tuned a CLAUDE.md that guides the agent, or set up hooks so that something always runs, you have already done harness engineering. You called it something else.

Both are right because they are not measuring the same thing.

LangChain optimised its own agent's harness and climbed 25 places on Terminal Bench 2.0 in March 2026, with the model untouched (Faros AI, 22 May 2026, retrieved 27/08/2026). In the same piece, Faros reports that open models with an optimised harness matched or beat frontier-model setups across 211 real engineering tasks.

Scale AI's SWE-Atlas compared models running inside different harnesses and found the opposite: the difference sits inside the margin of error, and it does not even point the same way twice. Opus 4.6 does better in Claude Code; GPT 5.2 does worse. Latent.Space, summarising the finding, headlined the issue with the question the market would rather not ask: "Is Harness Engineering real?"

That pattern has appeared on this blog already: the piece on Claude Code in production opens with METR and Microsoft reaching opposite results on productivity gains. Serious measurements disagree when they measure different things under the same name.

What changes between the two columns is the type of task. Terminal Bench measures terminal work with many chained steps. SWE-Atlas compares performance on isolated tasks. It is the same variable measured across two horizons, and the horizon decides whether the harness shows up or disappears into the noise.

Anyone who read only the headline that "harness engineering is the future" has half the information. Anyone who read only SWE-Atlas and concluded it is not worth investing in has the other half. I spent a week thinking the two studies cancelled each other out, until I looked at what each one had measured.

The harness matters in proportion to the number of steps the agent has to get right in a row.

The number that closes the argument: an agent's success rate falls from 60% on a single pass to 25% when it has to get eight consecutive steps right on the same task (Fiddler AI, 29 April 2026, citing arXiv 2511.14136v1; retrieved 27/08/2026). Error compounds. At every step, the chance that the previous one came out crooked enters the calculation.

The arithmetic of an agent chain is crueller still: three agents at 70% accuracy each deliver 0.7 × 0.7 × 0.7 = 34% end to end. You do not need a bad model to get there. Three good ones in a row will do it.

Before promising a deadline, run that calculation for your own case. It is a simple power: accuracy per step raised to the number of steps.

The practical reading is in the third row. A tool with 90% reliability looks fine when you test it on its own, and leaves 12% of a twenty-step task standing. For an agent to deliver four out of five long tasks, each step has to succeed 99% of the time. That number is what the verification layer exists to buy: every test that runs inside the loop turns an uncertain step into a checked one.

This is why SWE-Atlas could not see the harness. On a one-step task, the model carries almost the whole result. From the fifth or eighth step onward, what decides the outcome is whether anything is checking the work along the way.

And it is why the production failure rate is what it is. Fiddler puts it between 70% and 95%, depending on how you measure, and backs the range with three independent references: on the WebArena benchmark, a GPT-4 agent completed 14.41% of tasks end to end against 78.24% human performance; Carnegie Mellon measured failure on about 70% of office tasks; and MIT reported that 95% of generative-AI pilots produced no measurable impact on financial results.

The first two layers give the largest return, and in that order.

Tools set the ceiling. An agent is only as capable as the set of things it can call, and only as reliable as what those things return when they fail. A tool that returns Error without saying what happened forces the agent to guess, and guessing at step three becomes a disaster at step six. An error message is an agent interface. Connecting a tool is the easy half, and the practical guide to MCP covers it; designing what it returns is the half that decides the outcome.

The difference shows up in a single line. A file-reading tool that fails like this:

Error: ENOENT

leaves the agent with three hypotheses and no clue: the path is wrong, the file does not exist, or it lacks permission. It will try all three, spending steps. The same failure, written for whoever has to read it:

File not found: lib/posts.ts
The lib/ directory exists and contains: posts-meta.ts, markdown.ts, site.ts
Did you mean lib/posts-meta.ts?

The agent corrects on the first attempt. That cost three extra lines of error handling and saved two steps. By the arithmetic in the previous section, that is the difference between 60% and 25%.

Verification is the layer almost nobody implements and the one that changes the result most. The idea is simple: the build, the tests, and the linter run inside the loop, not after the agent has declared itself finished. Without that, "done" only means the model stopped generating tokens.

Faros's data shows the size of the effect: open models with an optimised harness matched or beat frontier-model setups across 211 real tasks. The verification layer made up for the difference in model.

This layer carries an important limit, and it is structural: the agent cannot review its own work with the reliability it brings to execution. The environment has to be the one verifying, with a build, a test and a lint. Rather than another pass of the same model over the text it wrote a moment ago.

For a task too large for a single loop, delegate part of it to a subagent with its own context.

Underneath these two layers sits an infrastructure decision made by inertia as a rule: how the tool reaches the agent at all. MCP, A2A and WebMCP solve different problems and sit at very different levels of maturity, and choosing wrong here costs a rewrite later.

Context is what the agent knows. A guardrail is what it cannot do. The two get confused all the time, and they solve opposite problems.

The context layer covers indexing what exists, the session history, and whatever survives between sessions. The failure mode is familiar: the agent forgets mid-task and starts again with half the information. Organising that has four pillars, and they resolve in order.

Memory is the part of context that outlives the session, and the instinct to store everything is the most expensive mistake in this layer. What deserves to persist is decisions and constraints: why one library lost out, which convention the repository follows, what someone tried already and what failed. Conversation transcripts grow without limit and push the things that matter out of the window.

How to retrieve what fell outside is the question that changed answers over the past two years. With large context windows, retrieval stopped being mandatory on every task. It still wins on cost in a good share of cases.

Guardrails are hard limits: a sandbox, a spend ceiling, human approval for irreversible actions, a list of what never executes. The difference from context is that a guardrail does not depend on the model agreeing. It is the only layer that still holds when the model interprets the instruction some other way, and it is the subject of permissions and auto mode.

The starting list is short and almost always the same: git push --force, rm -rf, database migrations, any call that spends money. Each item either requires human confirmation or does not exist in the tool set. Writing "be careful with force push" in the prompt covers the case where the model read the instruction. Removing the command covers the rest.

The reference for designing this layer today has a date: on 9 December 2025, OWASP published the Top 10 for Agentic Applications, built with more than 100 specialists, cataloguing the ten risk classes specific to agents that plan, hold memory, call tools, and act with delegated authority (OWASP GenAI Security Project, retrieved 27/08/2026). It is the list to design guardrails against, and the ten threats are worth taking one at a time.

One of them does not leave the list through engineering: prompt injection has no definitive fix, which pushes the defense into system design rather than input filtering. And the authority the agent carries to act needs an owner: the credential nobody revokes is the most common hole in this layer in 2026.

Without telemetry you do not know whether the harness improved anything. You know the last task worked, and nothing more.

The problem is that agent bugs do not reproduce. The same input does not produce the same output, so the trace has to exist before the error, not after. That means recording what the agent asked for, what the tool returned, how many steps it took, and where it started over.

A record useful for investigation has five fields: the task as it arrived, each tool call in order, the raw return of each one, the number of steps to completion, and the point where the agent switched strategy. The fifth is the one almost nobody keeps and the one that explains the most. An agent that abandons a path and tries another did one of two things: a tool returned something useless, or the context filled up.

Step count deserves separate attention, because it is the number that moves before all the others. A task that used to take six steps and now takes eleven is telling you that some tool started answering worse, or that the window is at its limit. Success rate drops after that, and by the time it drops the week is gone.

The metrics that matter sit on the business side. Faros proposes four: dollars per merged PR, time to merge, first-pass success rate, and defects that escape to production. None of them mentions tokens or latency, and that is why they are useful for deciding whether to keep investing.

Telemetry tells you what happened. Telling you whether it is good is a different problem, and it calls for evaluating the agent on the real task rather than trusting the impression left by the last session. With another model as the evaluator, it is worth knowing when to trust the score it gives.

If the question is the invoice at the end of the month, cost per usage pattern moves the needle more than the choice of model. And on the API at scale, the money leaks in places the dashboard does not show.

Between 21 and 27 August 2026, this blog published 23 articles written with an agent. The caveat is worth stating: this is operational experience, with a small N and the bias of the person who built it. It still shows Hashimoto's discipline working in practice: every time the agent got something wrong, the fix went into the environment.

The harness has all five layers. Tools: access to the repository, the browser, and the build. Verification: npm test, ESLint, tsc, and next build running inside the loop, plus a 16-item checklist that inspects every published post: single H1, canonical, cover with alt text, JSON-LD on the right date, sitemap entry. Context: an AGENTS.md at the root. Guardrails: explicit permissions and confirmation for irreversible actions. Observability: Lighthouse and a route sweep on every build.

What it let through is the useful part.

ScrollTriggers took their measurements before the fonts loaded. The site's web fonts use display: swap. As they arrive, every text block changes height and the calculated positions become stale numbers: pinned sections released in the wrong place, and a gallery ran over the terminal. No test caught it, because the build passed. The fix was ScrollTrigger.refresh() after document.fonts.ready, plus a re-measure when returning from a background tab. A mistake made impossible by construction to repeat.

The cover srcSet declared 1600w for 1200px files. The w descriptor is the number the browser uses to choose which image to download, and it was counting on pixels that did not exist in 20 of the 35 covers. Nobody noticed for weeks: the page looked right. It surfaced only when verification started comparing real naturalWidth against the declared value.

And then the optimisation that broke the page. Trying to improve blocking time, I deferred three components until the thread went idle. The gain was a lab number; the real effect was breaking initialisation order: displaced pins, invisible sections. Reverted in full. The rule that stuck: an optimisation that changes mount order needs visual verification rather than a metric alone.

Five steps to add a harness to a project already running, ordered by return: from what pays off on day one to what only pays off after months.

1. Write the context file. AGENTS.md or CLAUDE.md at the root, holding what is true about the project and not obvious from the code: which package manager, which command runs the tests, which directories the build generates and nobody hand-edits, which commit convention the repository follows. It is the cheapest fix available and the only one that pays off on the first day. The common waste is filling that file with what the agent works out on its own by reading package.json.

2. Put verification inside the loop. If the agent can run the test, it should run it before calling the job done. In practice that is a single command chaining what the project already has: npm test && npx tsc --noEmit && npm run lint. The test was already there; the gain comes from the agent seeing the failure while it still holds in mind what it changed. Hooks do this without depending on the model remembering.

3. Close off what must not happen. One guardrail against an irreversible action is worth more than ten instructions asking for care. Start with the list from the previous section and add whatever is specific to your domain: in my case, publishing to production without going through the checklist.

4. Record. Without a trace you will be arguing opinions about what the agent did. The minimum fits in one file per session, with the five fields described above. Start with step count if you are only going to start with one.

5. Adopt the discipline. A new error becomes a change in the environment. Every time. It is the only item on the list that never ends, and it is what separates having a harness from having configuration.

None of these steps requires changing the model. That is the point.

The strongest version of that argument is the one that drops the API altogether: with a decent harness, what fits on your laptop in 2026 handles a good share of drafting and text transformation. The constraint there stops being money and becomes memory.

What is harness engineering?

It is the practice of designing everything that surrounds the model in an agent: tools, verification, context, limits, and telemetry. The formula is agent = model + harness. Mitchell Hashimoto coined the term in February 2026, out of the discipline of turning every agent error into a permanent fix in the environment.

How is it different from prompt engineering?

Prompt engineering picks words; harness engineering writes code and configuration. A better prompt asks the agent to be careful. A better harness makes the build fail when it was not.

Does harness engineering actually improve results?

It depends on the task horizon, and that is the honest answer. On long tasks, yes: LangChain climbed 25 places on Terminal Bench 2.0 without changing the model. On short, isolated tasks, SWE-Atlas measured harness choice inside the margin of error. The more consecutive steps the agent has to get right, the more the harness decides the outcome: success falls from 60% on one pass to 25% across eight.

Do I need an agent framework for this?

No. The five layers are architecture, and they fit inside what the project already has: a markdown file at the root, the test command you run every day, a permissions list, and one log per session. A framework starts to earn its place when you need to orchestrate several agents or swap models without rewriting the integration. Start without one, and adopt it when its absence starts to hurt.

Where do I start if I have very little time?

With the context file at the project root and verification inside the loop. Both fit in an afternoon and cover the two highest-return layers.

• The harness is everything that is not the model, and it is where the gain lives once a task has many steps.
• Two serious measurements disagree because they measure different horizons. Be suspicious of anyone citing only one of them.
• The investment order is tools, verification, context and memory, guardrails, observability.
• The discipline is worth more than any tool: an agent error becomes a change in the environment, not a note in the next prompt.

This is the first of 16 articles about building agents that survive production. This text references the other fifteen throughout, each in the layer it belongs to: tools and protocols in the first, verification in the second, context and memory in the third, security and identity in the fourth, measurement and cost in the fifth. They arrive in English one at a time; the Portuguese edition holds all of them already. If you want the next ones, subscribe to the newsletter.