Agent observability: instrumenting what you cannot reproduce

The agent failed yesterday. You ran it again today, with the same input, and it worked.

There is no bug to investigate. There is an event that never returns.

That is the difference between observing an ordinary system and observing an agent. In the first, logs serve to reconstruct what happened because you can repeat the run and compare. In the second, the second run is a different run.

The record has to exist before the error. And it has to exist with the right names, because a standard already defines those names, and it is still moving.

The difficulty is no shortage of logs. It is that repetition stopped being an investigation tool.

In a deterministic system, you reproduce the error and observe it as many times as you need. Add a log, run again, look. The record can be incomplete, because improving and repeating is always available.

With an agent, each run picks a path. The tool it calls can be another one, the order can change, the number of steps varies. Something goes wrong, and that specific run is the only evidence that will ever exist.

Hence the practical inversion: in an ordinary system, you investigate and then instrument what was missing. With an agent, you instrument first and investigate with what you kept.

That is why observability comes last among the harness's five layers and still has to be built early. It depends on the other four to have anything to record. And without it, none of the four has a way to improve.

A standard exists, and it is specific. The OpenTelemetry GenAI special interest group has worked on it since April 2024. It defines operation names, attributes and types. It covers model calls, agent steps, tool use and token counts.

An agent trace has this shape:

invoke_agent research-assistant   (INTERNAL)
├── chat gpt-4o                   (CLIENT)
├── execute_tool web_search       (INTERNAL)
├── chat gpt-4o                   (CLIENT)
├── execute_tool summarize        (INTERNAL)
└── chat gpt-4o                   (CLIENT)

The agent span uses kind INTERNAL when the run is local and CLIENT when it calls a remote service.

The convention names eight operations: chat, text_completion, generate_content, embeddings, create_agent, invoke_agent, invoke_workflow and execute_tool (Greptime, 9 May 2026, accessed 28 August 2026; structure also described on the OpenTelemetry blog).

The hierarchy is what makes the trace useful. A total latency number says nothing. The same information, spread across the tree, shows something else. The task took 40 seconds. Thirty-one sat in one search. The model waited idle that whole time. Without the tree, you switch models. With the tree, you fix the tool.

Every attribute in the convention starts with gen_ai.. This table is worth copying:

Two pairs exist for a reason. The first keeps the model you asked for and the model that answered. They are not always the same, because the provider routes. The second separates a token read from cache from a token charged in full. It is the difference between an explainable invoice and a fright.

<!-- [UNIQUE INSIGHT] -->

One warning is worth more than the whole table. Part of the Portuguese material cites these attributes as llm.usage.prompt_tokens and llm.usage.completion_tokens. Those names are not the OpenTelemetry GenAI convention's. The origin looks like OpenInference, which has a schema of its own, or naming from old library versions.

This is no pedantry. An attribute name is a query key. Instrument with a name outside the standard and your dashboards stop talking to the ecosystem's tools. Migrating later costs you rewriting queries, alerts and dashboards.

One recommendation almost nobody follows and everybody regrets. Tag every span with at least one business attribute from day one. A user_id, a workflow_id, a task identifier. Add that after an incident and the incident that motivated the change is the only one without the field.

The token attributes are also the bridge between trace and invoice. They are how you discover which usage pattern is eating the budget, by the same reasoning as what each usage pattern costs.

Instrument now, and count on renaming.

The convention is real and large platforms are adopting it. Datadog was among the first to give it native support, from version 1.37 of the conventions, and OpenAI's Python SDK carries the most mature instrumentation of the set.

And it remains under construction. Three verifiable facts show that better than any label:

In June 2026, the specification got its own repository. The GenAI conventions page on the OpenTelemetry site now redirects to the semantic-conventions-genai repository. Splitting off a document under construction tends to happen when it changes at a different pace from the rest.

That repository held 139 open issues and 45 pending pull requests when I checked, on 28 August 2026.

One attribute already changed names between versions. In v1.37, the gen_ai.system field became gen_ai.provider.name. Anyone who instrumented before that had dashboards stop finding data over a name swap.

Greptime's analysis, from May 2026, describes the GenAI and MCP conventions as still in development status, with no public stabilization timeline. I could not confirm that label in the official documentation, which was migrating repositories when I checked, but the three facts above hold the conclusion up without it.

The response to that is simple: isolate the attribute mapping in a layer of its own. One function that takes your data and returns the attributes with the convention's names. Once a name changes, you edit one file instead of hunting strings through the code.

An explicit judgment: anyone waiting for the standard to stabilize before instrumenting will spend the year without a trace. Renaming an attribute costs an afternoon. Having no record of the incident that happened costs the whole incident.

The list of what is worth recording on each run:

• The task as it arrived, before any processing.
• Every tool call, with its argument and return.
• The number of steps to completion.
• The version of the model that answered, and not only the one requested.
• Input, output, cache and reasoning tokens.
• The stop reason, via gen_ai.response.finish_reasons.
• The point where the agent changed strategy.

The last one is the most informative and the least collected. It is also the signal that precedes cascading failure, one of the threats cataloged in the OWASP Top 10 for agents. The agent abandons a path and tries another for one of two reasons: the tool returned something useless, or the context filled up. Both are actionable, and neither shows up in a log that records only success.

What to keep out of the raw record

A trace holds gen_ai.tool.call.result, and a tool's result often contains text written by third parties: the page the agent opened, the email it read, the issue comment.

That carries two consequences. The first is privacy: user content crosses into an observability platform, which tends to keep a retention policy different from your database's. The second is that a trace gets read by people and by tools, and third-party text on any surface is an injection vector.

The middle ground works like this. Metadata always. Content by sampling. Sensitive content never. Response size, hash, origin and status code solve most investigations without carrying the text.

Sampling

Keeping everything at maximum fidelity gets expensive and it is seldom necessary. Two approaches circulate: head sampling, deciding at the start of the trace, and tail sampling, deciding at the end and letting you keep the traces that went wrong.

Current practice in production converges on the second, because errors are rare and errors are what matter. I treat that as market practice, not as a measured result, since I found no study comparing the two in an agentic system.

The cost has one detail that changes the arithmetic. On several platforms, only LLM spans enter the invoice. Tool, retrieval and agent spans come free. If that is the case on yours, a system heavy on agents costs less to observe than the total span count suggests. Check the price table before assuming. It varies by vendor.

A trace serves to investigate one case. A metric serves to discover that a case exists.

Calls per task is the number that moves first. It rises before the success rate falls, which gives a week of advantage to whoever is watching.

The third row deserves the bold sitting on it. An aggregate error rate hides the case you need to find: one tool failing 40% of the time among nine healthy ones disappears in the average, and it is the one eating the steps. Measuring per tool is the difference between knowing something got worse and knowing what did. It is also what turns tool design into data-guided work.

Worth noting that these four talk to the four metrics of the guardrails layer: those measure friction and coverage of the lock, and these measure execution health. These four answer "what broke". They do not answer "is this worth continued investment", which is a different question with different metrics, such as dollars per delivery, time to completion and defects that escaped, described in the pillar.

You need no new platform to start. You need three decisions.

1. Wrap the model call. A function of yours calls the API. It opens a chat span. It closes the span with the tokens and the stop reason. Nothing beyond that on day one.

2. Wrap the tool call. Same pattern, execute_tool span. Keep the tool name and the time. The argument and the return come later, once you decide the content policy.

3. Put both inside an invoke_agent. That span is what gives you the tree. Without it you have loose events, not a trace.

Done that, three numbers already come free: calls per task, tokens per task and time per step.

One choice to make early, because it is expensive to change later. Decide the business identifier now. Every span carries it. It can be the slug of whatever you are producing, the task number, the user. Without it, the trace answers "what happened" and fails to answer "on which piece of work".

And do not wait for the pretty dashboard. A structured file, one line per span, already answers the question that matters on incident day.

<!-- [PERSONAL EXPERIENCE] -->

Honesty costs something here. The operation publishing this blog has strong verification and weak observability.

The part that exists is good and it covers the product: Lighthouse on every build, a sweep of every route, a readability analysis of each text, a schema check with a report of orphan references. If the site regresses, I find out.

Of the agent, almost nothing. There is no structured trace, no named span, no systematic count of steps per task. The session history exists. It is a trail in prose. Rereading it and reconstructing one specific task works. Aggregating, comparing across weeks or firing an alert does not.

The contradiction is mine and it deserves saying in full. I recommended step counting as a leading indicator in this cluster's pillar. I collect it in no consistent way. Writing about the layer is what made me look at the gap.

The reason it exists is banal and common enough. The four earlier layers pay back where you can see it, on the next task: a better tool saves steps today, a test in the loop catches the error today. Observability only pays in the week something gets worse, and until then it looks like work with no result.

The plan, stated as a plan. Wrap the calls in something that emits invoke_agent and execute_tool with the convention's attributes. Start with three numbers: calls per task, tokens per task and error per tool. Once that runs for a few weeks, I come back here with the data.

The layer has a limit, and knowing it saves you from asking it for what it does not do.

The trace records what happened. The tool that got called, the argument it got, the time it took, the return it gave. That is mechanics.

It does not record why the model chose that. No field for intent exists. finish_reasons says the generation stopped for a tool call, and says nothing about why that tool looked like the right one.

It also fails to record whether the result was good. A flawless execution trace, all spans green, low latency, zero errors, can end in a wrong answer. The absence of technical failure is no signal of quality.

Those two gaps have different answers. For intent, what helps is recording the reasoning text when the model exposes it, and accepting that it remains a report, not a cause. For quality, the answer is an entire other layer: evaluation, which compares the output against a criterion.

It is the difference between asking "did it run?" and asking "was it any good?". Observability answers the first with precision. The second asks for evals, the subject of the next post.

In the meantime, a cheap habit helps: attach the result of the verification you already have to the trace. Whether the build passed, whether the test passed, whether the checklist closed. It is no measure of quality, and it is the closest the observability layer gets to one on its own.

What is LLM observability?

It is the practice of instrumenting applications built on language models. Each call, tool use and reasoning step becomes a traceable span, with standardized attributes for model, tokens, latency and result. The difference from traditional monitoring lies in the goal. It is not only about detecting downtime. It is about reconstructing a run that no one can repeat, because the same input produces a different output.

What is the difference between a log and a trace in an agent?

A log records isolated events, with a timestamp and a message. A trace records the structure of the run: which call happened inside which, in what order, and how long each took. In an agent, the structure is the main information, because it lets you attribute latency and cost to the specific step instead of to the total. A well-formed trace answers where the time went; a log answers that something happened.

Is OpenTelemetry worth using while the convention is still moving?

It is. The alternative is going without a trace or inventing a schema of your own, which will also change and talks to no tool yet. The practical care is isolating the attribute mapping in a single layer, so that a rename like gen_ai.system to gen_ai.provider.name, which happened in v1.37, costs one file edit.

Which attributes should an LLM span record?

At minimum: gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.response.finish_reasons. For tool spans, add gen_ai.tool.name and the result. And tag every span with at least one business attribute, such as user_id or workflow_id, because adding that field after an incident leaves that exact incident without it.

Do I need a paid platform to start?

No. The first useful step is wrapping the model and tool calls in spans and writing one structured line per span. A file already answers the questions of incident day. A platform solves aggregation, alerting and retention, which are problems of scale, not of starting. If you do buy, check how the vendor charges: on several plans only model spans enter the invoice, which changes the arithmetic of a system with many tool calls a great deal.

Is keeping the prompt content in the trace safe?

It depends on what enters it. A tool result tends to contain text written by third parties, and the observability platform keeps a retention policy different from your database's. The defensible practice is keeping metadata always, meaning size, origin, status code and hash, content by sampling, and sensitive content never.

• An agent bug does not reproduce. The trace has to exist before the error, because no second observation will come.
• The tree is invoke_agent at the top, with chat and execute_tool as children. It is what lets you attribute latency to the right step.
• The names start with gen_ai.. An attribute prefixed with llm. belongs to no convention, and a wrong name costs you rewriting dashboards.
• The standard is under construction: a separate repository in June 2026, 139 open issues, one attribute already renamed. Instrument anyway, with the mapping isolated.
• Measure errors per tool. The average hides the tool that is costing you the steps.

The next post in the cluster covers this layer's natural pair: observability says what happened, and evaluation says whether what happened was good. The full map continues in the pillar on harness engineering.