When an AI workflow produces a wrong answer, there are two kinds of teams. The first opens a trace, sees exactly which step failed, what data it used, and what the model returned, and fixes it in minutes. The second re-runs the workflow a few times, cannot reproduce the problem, and starts guessing. The difference between them is not talent. It is whether they can see what happened.
A trace is the record of a single end-to-end run: every step, in order, with its inputs, outputs, timing, and cost. In a modern agent that step list is long: retrieve data, mask PII, call a tool, invoke the model, check a guardrail, route for approval, deliver. Without a trace, all of that happens inside a black box, and you are left reasoning about a probabilistic system from its final output alone. Several disciplines already solved this problem, and they all point the same way.
The observability school: spans, not print statements
Site reliability engineering gave distributed systems a vocabulary for this: a trace is made of spans, where each span is one unit of work with a start, an end, and attributes, nested to show what called what. OpenTelemetry, now with semantic conventions for generative AI, made this a standard rather than a bespoke log format. The point of a span is that it captures structured facts (model, token count, latency, cost, tool name) instead of a wall of unstructured text. When every step is a span, "where did the 40 seconds go?" and "which call cost the most?" become queries, not archaeology.
The scientific school: a trace is a lab notebook
Science has a rule: an experiment you did not record cannot be reproduced, and a result you cannot reproduce is not a result. An AI run is an experiment with a stochastic instrument. The trace is the lab notebook: which inputs, which context, which model version, which prompt version, which output. Reproducibility is the deepest crisis in applied AI, because the same input can yield different outputs. Capturing the full context of each run is what lets you re-create a failure instead of hoping it happens again while you watch.
The forensics school: the black box recorder
Aviation does not improve by assuming crashes are rare. It improves because every aircraft carries a recorder, and every incident is reconstructed from data, not memory. A trace is that recorder for AI. When something goes wrong in front of a customer, you do not want a debate about what probably happened. You want the flight recorder: the exact sequence, the exact data, the exact model output, timestamped and immutable.
The accounting school: attribute every cost
Finance has always insisted that costs be attributable to a line item. AI spend without attribution is a single opaque number on a monthly invoice. When each span records its token count and cost, that number decomposes: this workflow, this step, this model, this customer. You cannot manage a cost you cannot attribute, and traces are what make AI cost attributable.
What a trace actually captures
A useful trace is structured. A single run might look like this:
{
"trace_id": "b7f3",
"workflow": "quote-drafter",
"prompt_version": 14,
"spans": [
{ "step": "retrieve_crm", "ms": 210, "status": "ok" },
{ "step": "mask_pii", "ms": 8, "redactions": 3 },
{ "step": "llm_call", "model": "claude-sonnet-4.6", "tokens": 1840, "cost_usd": 0.031, "ms": 4200 },
{ "step": "guardrail_output", "check": "grounding", "status": "pass" },
{ "step": "approval", "user": "manager@acme", "status": "approved" }
],
"total_ms": 4630,
"total_cost_usd": 0.033
}Notice what this makes possible: the trace links to the prompt version, so a bad output points straight at the configuration that produced it. That link between traces, prompts, and guardrails is what turns observability from a dashboard into a debugging tool.
What breaks when you fly blind
Skipping tracing is invisible right up until you need it, and then it is the only thing that matters.
- The silent wrong answer. A finance workflow reports a cash position that is off by a large margin. With no trace, you cannot see that a connector returned stale data three steps back, so you re-run and guess for two days. A trace would have shown the stale fetch immediately.
- The cost blowup. The monthly model bill is five times normal. Without per-run cost in traces, you cannot tell which of forty workflows is responsible, so you throttle everything and slow the whole business while you hunt.
- The unreproducible bug. A customer reports an offensive reply, but it only happens sometimes. Without the captured input and context, you cannot recreate it, so you cannot confirm a fix. The issue stays open for weeks.
- The compliance question you cannot answer. An auditor asks, "what data did the AI use to make this decision, and who approved it?" With no trace, you have no answer, and "we think it was fine" is not one.
- The invisible drift. Output quality slowly degrades as a data source changes upstream. With no metric mined from traces, nobody notices until customers complain, by which point the damage is weeks deep.
The business case
Tracing moves four numbers that executives care about. It cuts mean time to resolution, because engineers debug from evidence instead of guesses. It controls cost, because spend becomes attributable and therefore optimizable. It improves quality, because traces are a mine of real failure cases that feed better prompts and evals. And it satisfies audit, because every decision has a reconstructable record. None of these requires a bigger model. They require being able to see.
Conclusion
You cannot operate what you cannot see. A model without tracing is a system you are running on faith, and faith does not survive the first production incident. Treat every run as an experiment worth recording, capture it as structured spans, and link it to the prompt and guardrails that shaped it. Then, when something goes wrong, and it will, you open the recorder instead of opening a debate.
Want end-to-end tracing across connectors, guardrails, models, and approvals? Book a 30-minute call and we'll wire up observability for your workflows.