AI Coding Agents Need Traces, Not Just Transcripts

GitHub has added OpenTelemetry support to the Copilot app so organizations can follow agent sessions across model requests and tool use in their existing observability systems. Prompt and response content is excluded by default, with content capture requiring a separate decision. I think this is the right direction. A chat transcript tells us what the agent said. A trace can tell us which tools it called, how long each step took, where it retried, and which boundary failed. As agents take longer and more autonomous tasks, that operational view matters as much as the final patch.

A transcript is a narrative, not a system map

Conversation history is useful for reviewing intent and decisions, but it can hide timing, parallel work, failed calls, and repeated operations. It is also awkward to aggregate across teams without collecting sensitive content.

Tracing represents a session as related spans. A model request, repository read, test run, browser action, and deployment check can each have timing and status while remaining part of one task.

I would begin with metadata-only telemetry

The default exclusion of prompts and responses is important. Source code, customer data, secrets, and internal decisions can all appear in content. Most operational questions do not need that payload.

I would start with tool name, duration, success state, model class, repository, environment, retry count, token or cost bands, and a privacy-safe task identifier. Content capture should have a separate owner, retention policy, and access review.

Useful traces answer operational questions

• Which tool or service causes the most failed sessions?
• Where do agents spend time waiting?
• How often does a task retry the same action?
• Which repositories require repeated clarification?
• Do browser tasks fail at authentication, navigation, or verification?
• Which automated checks prevent bad changes before review?

These questions lead to better permissions, documentation, test environments, and workflows. A generic “agent productivity” score rarely does.

Observability does not replace review

A trace can show that tests passed and the deployment tool returned success. It cannot decide whether the feature matches the customer’s intent or whether a test suite protects the right behavior. Human review still owns scope, tradeoffs, and release judgment.

I see traces as evidence for that review. They make hidden automation visible and help reviewers focus on uncertain or privileged steps.

Telemetry needs its own safety budget. Observability systems can become a second copy of sensitive data. I would define allowed attributes, scrub error messages, sample routine sessions, limit retention, and restrict dashboards by role. Correlation identifiers should not expose customer identities.

I would also monitor the monitor: export failures, dropped spans, and telemetry cost should be visible so the organization does not mistake missing data for clean behavior.

Start with one decision the trace can improve. The GitHub announcement describes centralized OpenTelemetry configuration and agent-session analysis. I would not enable it merely to collect a new dashboard.

A good pilot might target failed browser-verification tasks or slow test runs, where tool timing and error spans can lead to a concrete workflow change.

Trace boundaries can mirror responsibility. I would create spans around meaningful ownership boundaries rather than every tiny internal call: planning, repository inspection, implementation, verification, privileged action, and external service. That structure lets a reviewer see where the session changed from reading to writing and where a human approval should have occurred.

Attributes can capture the declared environment and mutation class so teams can detect a production action in a task expected to remain local. Links between the agent trace and CI run, pull request, or browser evidence make the final result easier to audit without copying their full contents into telemetry. Over time, the trace model can reveal whether failures come from vague requests, missing project instructions, unstable tools, or inadequate tests. The point is to improve the system around the agent, not assign a simplistic score to individual developers.

Service-level objectives should remain modest at first. I might track the percentage of sessions with a complete verification span or the rate of tool failures that receive a classified cause. I would not promise that every agent task finishes within one duration; research, implementation, and incident work naturally have different shapes and should not be optimized into the same behavior.

Developers should be able to see what telemetry is collected about their sessions. A published attribute dictionary, access policy, and deletion path make the program easier to trust and reduce the temptation to add opaque content capture later. I would include the telemetry configuration in ordinary security and architecture reviews.

Observe one workflow before instrumenting everything

I would instrument one agent workflow with metadata-only traces, set a short retention period, and review the first two weeks with developers and security owners. Keep only the fields that change a decision. The goal is not surveillance or a complete replay; it is enough evidence to make automated work understandable and improvable.

Photo by Vito Goričan on Pexels.