[Proposal] End-to-end tracing for agentic use cases (GenAI spans)

[randilt]

Summary

The gateway now serves LLM providers, LLM proxies, MCP proxies and A2A agents. This proposal makes one agent turn through the gateway show up as one trace, and adds OpenTelemetry GenAI spans that carry the model, token usage and tool for each call.

It has two parts:

1. Fix trace continuity in the current tracing, so that Envoy's spans, the policy engine's spans and the upstream's spans form one tree ([Improvement]: Nest policy spans under their phase span and pass each policy its own span context #3733, [Bug]: Policy-engine spans are not parented under Envoy's span, so a request produces two traces #3734, [Bug]: Policy-engine sampler ignores the parent's sampling decision, producing orphan spans and partial traces #3735, [Improvement]: Streamed responses create a phase span and a span per policy for every chunk #3736).
2. Add GenAI spans for LLM, MCP and A2A calls, following the OpenTelemetry GenAI and MCP semantic conventions.

No new component, control-plane API or database change is needed. This builds on the original tracing design in # .

What is the problem that we are trying to solve?

An agent turn through the gateway does not appear as one trace, and no span carries model, token or tool data.

The following was observed by running the gateway built from main (gateway-runtime 2026.09.24, Envoy 1.39.0) with tracing enabled and inspecting the traces in Jaeger.

Case	What happens today	Tracked in
Client sends no traceparent (also the LLM proxy's loopback hop)	Envoy's spans and the policy engine's spans form two separate traces	#3734
Client sends traceparent	One trace, but the policy-engine root is a sibling of Envoy's ingress span, not a child of Envoy's ext_proc span	#3734
Client sends traceparent with the sampled flag 00	Envoy records nothing; the policy engine still exports 13 spans whose parent was never recorded	#3735
Any request	Phase spans and policy spans are all flat under the policy-engine root	#3733
Streamed chat completion, 22 chunks	79 spans: every chunk gets a phase span plus one span per streaming policy	#3736
MCP tools/call	Trace context in params._meta is ignored and forwarded unchanged	This proposal
Any LLM, MCP or A2A request	No gen_ai.* attributes on spans. They exist only on the OTel analytics log records, which carry no trace ID	This proposal, #3738
Policy side-calls	Calls from policies to guardrail, embedding and other services emit no spans	#854

Trees observed today:

No client traceparent: two unrelated traces

Trace A                                  Trace B
ingress (Envoy, SERVER)                  external_processing.process (policy engine, SERVER)
├─ ext_proc egress (Envoy, CLIENT)       ├─ process_request_headers
└─ router egress (Envoy, CLIENT)         ├─ policy.request.<name>
                                         └─ … all flat

Client sends traceparent: one trace, wrong shape

client span
├─ ingress (Envoy, SERVER)
│  ├─ ext_proc egress (Envoy, CLIENT)        ← the policy engine should be under this
│  └─ router egress (Envoy, CLIENT)
└─ external_processing.process (policy engine)   ← sibling of ingress

Who are we solving the problem for?

Engineers who run agents through the AI Gateway. They need one trace per agent turn that shows each model call and tool call, its tokens and latency, and any policy that blocked or changed it.

Secondarily, teams that display gateway spans next to agent-side spans, such as WSO2 Agent Manager.

Why should it be solved?

• Agent turns now cross several gateway hops: LLM proxy, provider, MCP proxy, A2A agent. Without one trace, latency and failures cannot be attributed to a hop.
• Other gateways, such as Agent Router (formerly Envoy AI Gateway) and agentgateway, already emit OpenTelemetry GenAI spans.
• The data already exists. The gateway extracts the model, token usage and MCP method for analytics today; tracing mostly needs to put that data on spans.

Proposed solution

The policy engine emits OpenTelemetry GenAI spans from data it already holds, and trace context handling is fixed so that all gateway spans form one tree. Envoy stays the source of HTTP spans.

flowchart LR
  client["Agent or client<br/>sends traceparent"]
  ctrl["Gateway controller<br/>injects the genai-tracing system policy"]
  subgraph runtime["Gateway runtime (one container)"]
    envoy["Envoy router<br/>ingress, ext_proc and upstream spans<br/>(unchanged)"]
    subgraph pe["Policy engine (root span parented on Envoy's span)"]
      rec["GenAI span recorder (new)<br/>chat, tools/call, invoke_agent spans"]
      chain["Policy chain<br/>policy spans; side-call CLIENT spans (new)"]
    end
  end
  subgraph up["Upstreams"]
    llm["LLM provider"]
    mcp["MCP server"]
    a2a["A2A agent"]
  end
  col["OTel Collector"]
  be["Any OTLP backend"]

  client --> envoy
  envoy -- "ext_proc stream<br/>traceparent in gRPC metadata" --> pe
  envoy -- "traceparent header<br/>MCP _meta rewritten" --> up
  ctrl --> pe
  envoy -. "OTLP" .-> col
  pe -. "OTLP" .-> col
  col --> be

Loading

1. Trace continuity (prerequisite)

Change	Issue
Parent the policy-engine root span on the traceparent that Envoy injects into each per-request ext_proc gRPC stream. Use the HTTP headers only when Envoy tracing is off.	#3734
Make the policy-engine sampler ParentBased, so it follows Envoy's sampling decision.	#3735
Nest policy spans under their phase span, and pass each policy its own span context.	#3733
Replace per-chunk spans with one response-body span per policy, carrying a chunk count.	#3736
Agree which resolver attributes may be recorded on spans.	#3737
Add traceId and spanId to the OTel analytics log records.	#3738
Take service.version from the build.	#2096

Envoy 1.39 creates a CLIENT span for the ext_proc call and injects its context into the stream's metadata (async_client_impl.cc). It opens one stream per HTTP request (ext_proc.cc). The policy engine reads the downstream HTTP headers instead, which carry only the client's context.

2. GenAI spans

A new span recorder in the policy-engine kernel opens one span per AI operation. The gateway controller injects a genai-tracing system policy on LLM, MCP and Agent routes, the same way it injects the analytics system policy, so that the policy engine sees the response stream.

Spans follow the OpenTelemetry GenAI and MCP semantic conventions, pinned at commit 06ec68e (schema gen-ai-dev/1.42.0-dev). Gateway-only facts use wso2.* keys, reusing the keys the OTel analytics publisher already emits where they exist.

Route	New span (kind CLIENT)	Key attributes	Filled from
LLM provider	{operation} {model}, for example chat gpt-4o	gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.response.id, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.reasoning.output_tokens, gen_ai.request.stream, gen_ai.response.time_to_first_chunk, server.address, error.type	Provider template and sdk/ai/llmusage
MCP proxy	{mcp.method.name} {target}, for example tools/call get_invoice	mcp.method.name, gen_ai.operation.name=execute_tool (tool calls only), gen_ai.tool.name, mcp.protocol.version, jsonrpc.request.id, rpc.response.status_code, error.type	MCP resolver
Agent (A2A)	invoke_agent {agent}	gen_ai.operation.name=invoke_agent, gen_ai.agent.name, gen_ai.agent.version, gen_ai.conversation.id (the A2A contextId)	A2A resolver

Existing policy-engine spans keep their names and gain wso2.ai.proxy, wso2.guardrail.name and wso2.policy.action.

Rules:

• Operation names. The standard values are used: chat, text_completion, embeddings, generate_content, execute_tool, invoke_agent. For LLM routes these are the values the OTel analytics publisher already derives from the route.
• Span timing. A GenAI span runs from when the request leaves the policy chain to the last response byte. Gateway time and model time therefore show separately, and a streamed response closes at stream end.
• LLM proxies. Only provider routes emit the inference span. An LLM proxy's own route never does, because its upstream is the loopback provider route, so tokens are reported once.
• MCP context.
  • When params._meta carries a context in the same trace as the HTTP request, the MCP span nests under the gateway's span.
  • Otherwise the MCP span is parented on the _meta context and linked to the HTTP context, as the MCP conventions specify.
  • Before forwarding, the gateway rewrites _meta.traceparent and _meta.tracestate to its own span. These keys are reserved for this by MCP SEP-414.
• Guardrails. No OpenTelemetry convention exists yet (semantic-conventions-genai PR 427 is open). wso2.* keys are used until it merges.
• Content. Prompts, completions and tool arguments are not recorded by default. When enabled, they are captured after the masking and guardrail policies have run, and truncated.
• Policy denials keep today's behaviour: the HTTP status and terminal.reason are recorded, and the span status stays unset for 4xx, as the OpenTelemetry HTTP conventions specify.

3. Target trace

One agent turn: an LLM call through an LLM proxy, then an MCP tool call. Envoy's ext_proc span above each policy-engine root is omitted for readability.

4. Configuration

One new block under the existing [tracing] section. The Helm chart mirrors it under gateway.config.tracing.genai.

[tracing]
enabled = true
sampling_rate = 0.1          # existing; the policy engine now follows Envoy's decision

[tracing.genai]
enabled = true               # GenAI, MCP and A2A spans; default true when tracing is on
capture_content = false      # prompts, completions, tool input and output (opt-in)
max_content_bytes = 16384    # truncate captured content

5. Compatibility with WSO2 Agent Manager

Agent Manager instruments agents in-process with Traceloop, and its trace parser reads the standard gen_ai.* names. The gateway therefore needs no Agent Manager specific keys: it emits the standard names, sets error.type on failures, and never emits traceloop.* keys.

For gateway spans to appear inside an agent's trace in the Agent Manager console, two further things are needed, both listed under open questions: the calling agent's identifiers on the spans, and export to Agent Manager's collector.

Alternatives considered

Alternative	Why it was not chosen
Put GenAI attributes on Envoy's own spans, using custom_tags fed from dynamic metadata	Tags are strings only in Envoy 1.39, but token counts must be integers. The tag list is fixed per listener, and Envoy cannot rewrite MCP params._meta.
OpenInference conventions (Agent Router's default)	Most OpenTelemetry backends, and Agent Manager, read gen_ai.*. An OpenTelemetry Collector can remap these spans to OpenInference if a user needs it.
A new tracing sidecar or component	The policy engine already has the OpenTelemetry SDK, the request context and the extracted data.

Challenges and how the design handles them

• Convention churn. Every GenAI and MCP convention is at Development stability, and names have changed since semantic conventions v1.41.0 (for example cache_creation became cache_write). A commit is pinned, and every key lives in one Go package, because no official Go package exists after semconv/v1.41.0. GenAI metrics wait until the token metrics settle.
• Privacy. Content capture is opt-in. Inbound traceparent, _meta and baggage are used only to correlate spans, never as identity.
• Token double counting. An agent's SDK span and the gateway's span can both report usage. Backends can separate them by service.name; Agent Manager counts usage only on the outermost span.
• Performance. Token counts need the response body. The policy engine already reads it when analytics, token rate-limit or cost policies run; otherwise enabling GenAI spans adds one pass-through streamed read. Collapsing per-chunk spans reduces the span count on streams.
• Envoy owns the upstream span. The provider's parent is Envoy's router egress span, and the GenAI span sits beside it in the same trace.
• MCP clients that send context only in _meta. Envoy's HTTP spans for those requests land in a separate trace that the MCP span links to, because Envoy picks its trace before the request body is read.
• Backward compatibility. Existing span names stay, and GenAI spans are additive. The visible changes are that policy-engine spans move under Envoy's span, and streamed responses produce fewer spans.

Delivery plan

Phase	Scope	Main code
0. Trace continuity	Section 1 above	gateway-runtime/policy-engine/internal: kernel/extproc.go, tracing/tracer.go, executor/chain.go, kernel/resolution.go, analytics/publishers/otel.go
1. LLM spans	Span recorder, genai-tracing system policy, [tracing.genai] configuration, LLM proxy rule, time to first chunk	New kernel recorder, sdk/ai/llmusage, gateway-controller/pkg/utils/system_policies.go, gateway/configs/config-template.toml, Helm chart
2. MCP and A2A spans	MCP span, _meta read and rewrite, invoke_agent span	internal/resolver/mcp.go, internal/resolver/a2a.go
3. Policies and content	Guardrail attributes, client spans and propagation for policy side-calls (#854), opt-in content capture	httpkit/httpclient/client.go, guardrail policies in wso2/gateway-controllers
4. Ecosystem	Agent Manager export, GenAI metrics	To be agreed

No tracing integration tests exist today. Each phase adds tests that run the gateway against an OTLP sink and assert the span tree and parent IDs.

Open questions

1. PR fix(policy-engine): extract trace context from request headers #3061 reported no trace context in the ext_proc gRPC metadata, though Envoy 1.39 injects it into every stream. This should be confirmed with a debug log before [Bug]: Policy-engine spans are not parented under Envoy's span, so a request produces two traces #3734 is implemented.
2. The existing IsSpanSafeAttribute rule keeps tool names and JSON-RPC IDs off spans, but the MCP conventions require both. Should the conventions be followed on the new MCP span only ([Task]: Decide which resolver attributes may be recorded on spans (IsSpanSafeAttribute is defined but never applied) #3737)?
3. Content capture: is a gateway-wide switch enough, or is a per-proxy setting needed? A per-proxy setting changes the control-plane API.
4. Agent Manager integration: how does the gateway get each agent's identifiers, and which export route should it use? Agent Manager's collector accepts OTLP over HTTP only, while the gateway exports OTLP over gRPC today.
5. Should the gateway forward traceparent to A2A agents by default? An agent that adopts it gets the gateway's span as its parent.
6. Should GenAI metrics be exported as OTLP metrics or as Prometheus series?
7. Routes that carry telemetry (for example an OTLP ingest route deployed on the gateway) would themselves be traced. Should such routes be excluded from tracing, as health-check routes are today?

References

• Original tracing design: Distributed Tracing for Gateway #494
• Related issues: [Improvement]: Nest policy spans under their phase span and pass each policy its own span context #3733, [Bug]: Policy-engine spans are not parented under Envoy's span, so a request produces two traces #3734, [Bug]: Policy-engine sampler ignores the parent's sampling decision, producing orphan spans and partial traces #3735, [Improvement]: Streamed responses create a phase span and a span per policy for every chunk #3736, [Task]: Decide which resolver attributes may be recorded on spans (IsSpanSafeAttribute is defined but never applied) #3737, [Improvement]: Add traceId and spanId to OTel analytics log records #3738, Use build-time Version constant for OTel service.version attribute #2096, Submit traces from Policies #854, [Task]: check the frequent "No valid trace context extracted" #3026
• OpenTelemetry GenAI and MCP semantic conventions at 06ec68e, including MCP context propagation
• Upstream discussion on gateway span topology: semantic-conventions-genai issue 231 and issue 299
• MCP SEP-414: trace context in _meta
• Envoy v1.39.0 gRPC client trace injection
• Agent Router
• Agent Manager instrumentation contract