Trace the decision, not just the call.

Most teams instrument agents the way they instrument web services. Every model call gets a log line with latency, token counts, and a status code. Every tool call gets a span.

Then an agent does something wrong, and none of it helps. The logs show that a tool was called with certain arguments and returned successfully. They do not show why the agent chose that tool, what it believed when it chose it, or whether it was supposed to be able to make that choice at all. A service trace answers "what happened and how long did it take." An agent trace has to answer a harder question: "why did this happen, and should it have?" We design observability around that second question.

The unit is the decision

In a conventional service, the unit of work is a request. In an agent, the unit that matters is a decision: the point where the agent looked at its context and picked a next step.

We record each decision as its own event. It holds the step the agent took, the alternatives it had available, and the reason it gave. The reason is not ground truth, since models can rationalize. But a stated reason that does not match the evidence is itself a signal, and it is far easier to catch when it sits next to the action it is supposed to explain.

Capture what the agent saw

Most agent failures are context failures. The model was not wrong about the world. It was wrong about what it had been shown. A retrieval step returned a stale document. A memory with no source outranked a fresh one. A tool result was truncated and the agent filled in the rest.

So every decision event points to the exact context it was made against: which instructions were loaded, which memories were retrieved and why, which tool results were in the window, and which of those came from untrusted sources such as web pages, emails, or shared files. Where the underlying data already lives somewhere durable, we store references, not copies. The goal is to be able to rebuild the agent's view of the world at that moment, not to duplicate it.

Record permission, not just action

A trace that says "sent a message" is incomplete. The question an operator actually asks is whether the agent was allowed to send it.

Each action event carries the scope it ran under: which permission granted it, where that permission came from, and whether the action crossed an approval gate. If a human approved it, the trace records who approved it, what exactly they saw when they approved, and when. If the action was blocked, that is recorded too. Blocked actions are some of the most useful data an operating layer produces, because they show where an agent's intent and its authority diverged.

Keep the model out of the identity

Models change. When a provider ships a new version, or a task is routed to a different model, behavior shifts in ways that are hard to predict. Observability has to make those shifts visible.

We tag every decision with the model and version that produced it, alongside the prompt and tool definitions in effect. That keeps the agent's identity separate from the model behind it. The agent is the role, the scope, and the memory. The model is one input. When a swap changes behavior, the traces from before and after can be compared step for step, and the regression is something you can point to rather than a feeling that things got worse.

Traces become evals

A complete decision trace is more than a debugging aid. It is a test case.

When a run goes wrong and someone works out what the right decision would have been, that trace, with its context and the corrected outcome, goes into the eval set. The next time the prompt changes, the tools change, or the model changes, the eval replays the same context and checks the decision. Over time the eval set becomes a record of every way the system has failed, and a guard against failing that way again. This only works if the trace captured enough to replay. A log line with a token count cannot be replayed.

Watch the cost of watching

Decision-level tracing is heavier than request logging, and it is easy to overdo. Storing every full context window on every step gets expensive quickly, and much of it is redundant.

Our rule is to separate what is always kept from what is sampled. Decision events, permission checks, approvals, and blocked actions are always kept, because they are small and they are what an audit needs. Full context snapshots are kept on failures, on actions that crossed an approval gate, and on a sample of normal runs. Retention follows the same logic: the record of who decided what lasts longer than the raw material behind it.

Make it readable by a person

The final consumer of a trace is usually a human trying to understand what went wrong, or deciding whether to trust an agent with more scope. If the trace is only readable by a query engine, it fails that person.

We design traces so a single run reads as a sequence of plain steps: what the agent was asked, what it looked at, what it decided, what it was allowed to do, and where a human stepped in. Every step links down to the raw detail for anyone who wants it.

An operating layer is only as governable as it is observable. Approval gates, scoped permissions, and sourced memory all depend on a record that shows they held. We build that record around decisions, because decisions are where agents either earn trust or lose it.