Use Traces to Debug AI Coding Agent Workflows

When an AI coding task fails, the chat history may not show which tool timed out or how often an operation retried. A trace records related steps with their timing and outcome, making those questions easier to investigate. GitHub Copilot app now supports enterprise-managed OpenTelemetry configuration, including export to compatible monitoring tools. Prompt and response content is excluded by default. I would begin with that limited collection and one practical question, such as why browser verification fails or test runs take too long. The useful result is a change to the workflow, rather than a dashboard full of activity nobody acts on.

Record tool timing and outcomes

Conversation history is useful for reviewing intent and decisions, but it can hide timing, parallel work, failed calls, and repeated operations. It is also awkward to aggregate across teams without collecting sensitive content.

Tracing represents a session as related spans. A model request, repository read, test run, browser action, and deployment check can each have timing and status while remaining part of one task.

Start without capturing prompts or responses

The default exclusion of prompts and responses is important. Source code, customer data, secrets, and internal decisions can all appear in content. Most operational questions do not need that payload.

I would start with tool name, duration, success state, model class, repository, environment, retry count, token or cost bands, and a privacy-safe task identifier. Content capture should have a separate owner, retention policy, and access review.

Choose questions the trace can answer

• Which tool or service causes the most failed sessions?
• Where do agents spend time waiting?
• How often does a task retry the same action?
• Which repositories require repeated clarification?
• Do browser tasks fail at authentication, navigation, or verification?
• Which automated checks prevent bad changes before review?

These questions lead to better permissions, documentation, test environments, and workflows. A generic “agent productivity” score rarely does.

Keep review and access controls in place

A trace can show that tests passed and the deployment tool returned success. It cannot decide whether the feature matches the customer’s intent or whether a test suite protects the right behavior. Human review still owns scope, tradeoffs, and release judgment.

I see traces as evidence for that review. They make hidden automation visible and help reviewers focus on uncertain or privileged steps.

Limit what telemetry stores. Observability systems can become a second copy of sensitive data. I would define allowed attributes, scrub error messages, sample routine sessions, limit retention, and restrict dashboards by role. Correlation identifiers should not expose customer identities.

I would also monitor the monitor: export failures, dropped spans, and telemetry cost should be visible so the organization does not mistake missing data for clean behavior.

Start with one decision the trace can improve. The GitHub announcement describes centralized OpenTelemetry configuration and agent-session analysis. I would not enable it merely to collect a new dashboard.

A good pilot might target failed browser-verification tasks or slow test runs, where tool timing and error spans can lead to a concrete workflow change.

Group trace steps by the work being done. I would create spans around meaningful stages of the workflow rather than every tiny internal call: planning, repository inspection, implementation, verification, privileged action, and external service. That structure lets a reviewer see where the session changed from reading to writing and where a human approval should have occurred.

Attributes can capture the declared environment and type of change so teams can detect a production action in a task expected to remain local. Links between the agent trace and CI run, pull request, or browser evidence make the final result easier to audit without copying their full contents into telemetry. Over time, the trace model can reveal whether failures come from vague requests, missing project instructions, unstable tools, or inadequate tests. The point is to improve the system around the agent, not assign a simplistic score to individual developers.

Service-level objectives should remain modest at first. I might track the percentage of sessions with a complete verification span or the rate of tool failures that receive a classified cause. I would not promise that every agent task finishes within one duration; research, implementation, and incident work naturally have different shapes and should not be optimized into the same behavior.

Developers should be able to see what telemetry is collected about their sessions. A published list of collected fields, access policy, and deletion path make the program easier to trust and reduce the temptation to add opaque content capture later. I would include the telemetry configuration in ordinary security and architecture reviews.

Try tracing on one workflow first

Pick one recurring failure or delay and identify the minimum trace fields needed to investigate it. Set access, retention, and error-scrubbing rules before collecting data. Review a short pilot with the people running the workflow, then remove fields that do not help a decision. Link to existing test or review evidence where possible instead of copying its contents into telemetry. Keep monitoring transparent so developers know what is collected and how it is used.

Photo by Vito Goričan on Pexels.