Telemetry

The SDK has two observability layers:

OpenTelemetry export — distributed traces, metrics, and structured logs pushed to any OTLP-compatible backend (Jaeger, Grafana, Datadog, etc.). Configured once at agent creation.
Run-level telemetry — AgentTelemetry on every AgentRunResult. LLM round counts, tool breakdowns, and storage operation summaries returned inline with each result. No configuration required.

Enable OpenTelemetry

Pass WithObservabilityConfig to wire traces, metrics, and logs in one block:

a, err := agent.NewAgent(
    agent.WithLLMClient(llmClient),
    agent.WithObservabilityConfig(&agent.ObservabilityConfig{
        Endpoint: "localhost:4317",        // gRPC OTLP endpoint
        Protocol: agent.OTLPProtocolGRPC,  // or OTLPProtocolHTTP
        Insecure: true,                    // dev only — omit for TLS
    }),
    agent.WithLogLevel("info"),
)
defer a.Close() // flushes all OTLP exporters — required

Use the same WithObservabilityConfig on NewAgent and NewAgentWorker. LLM calls and tool executions run on the worker process — without matching config those spans and metrics are dropped. Opt out of individual signals:

agent.WithObservabilityConfig(&agent.ObservabilityConfig{
    Endpoint:       "localhost:4317",
    DisableTraces:  false, // default — enabled
    DisableMetrics: false, // default — enabled
    DisableLogs:    false, // default — enabled
})

See Tracing, Metrics, and Logs for per-signal details and collector configuration.

Run-level telemetry (`AgentTelemetry`)

Every completed run populates AgentTelemetry on AgentRunResult. This is run-level behavioral data returned with the result — separate from OTLP export. Use it for app-side logging, dashboards, and eval harness assertions.

Structure

AgentTelemetry has three sections:

Section	Tracks
`Run`	Orchestration lifecycle — timing, LLM round count, finish reason
`Tools`	Tool invocation totals, failures, per-tool breakdown
`Storage`	Retriever searches and memory recall/store operations

Fields are zero when the corresponding feature is not configured (no retriever → retriever counts are 0; no memory → memory counts are 0).

Run telemetry

Field	Description
`StartedAt`	Run start time
`CompletedAt`	Run completion time
`TotalLLMCalls`	LLM calls across all tool rounds
`FinishReason`	`complete` or `max_iterations`

max_iterations means the run hit WithMaxIterations before producing a final answer.

Tool telemetry

Field	Description
`TotalCalls`	Total tool invocations
`FailedCalls`	Tools that returned an error — excludes approval-denied and unauthorized cases
`Breakdown`	Per-tool invocation counts — key is tool name
`FailedBreakdown`	Per-tool failure counts

Covers native tools, MCP tools, retriever tools, and sub-agent delegation tools.

Storage telemetry

Field	Description
`TotalRetrieverSearches`	Total RAG searches
`FailedRetrieverSearches`	Failed retriever searches
`PrefetchSearches`	Searches from prefetch or hybrid mode
`AgenticSearches`	Searches from agentic or hybrid mode (LLM-initiated)
`TotalMemoryRecalls`	Memory load operations before the run
`FailedMemoryRecalls`	Failed recall operations
`TotalMemoryStores`	Memory persist operations
`FailedMemoryStores`	Failed store operations

See Retrieval and Memory for when each counter increments.

Run

agentRun, err := a.Run(ctx, "What's the weather in Tokyo?", nil)
if err != nil {
    return err
}
result, err := agentRun.Get(ctx)
if err != nil {
    return err
}

t := result.Telemetry
fmt.Printf("llm_calls=%d finish=%s\n", t.Run.TotalLLMCalls, t.Run.FinishReason)
fmt.Printf("tool_calls=%d failed=%d\n", t.Tools.TotalCalls, t.Tools.FailedCalls)
fmt.Printf("retriever=%d prefetch=%d agentic=%d\n",
    t.Storage.TotalRetrieverSearches,
    t.Storage.PrefetchSearches,
    t.Storage.AgenticSearches)
fmt.Printf("memory recall=%d store=%d\n",
    t.Storage.TotalMemoryRecalls,
    t.Storage.TotalMemoryStores)

Example: Simple Agent · Observability.

Stream

Telemetry is on Result.Telemetry inside the AgentEventTypeRunFinished event:

for ev := range eventCh {
    if ev.Type() != agent.AgentEventTypeRunFinished {
        continue
    }
    finished, ok := ev.(*agent.AgentRunFinishedEvent)
    if !ok || finished.Result == nil || finished.Result.Telemetry == nil {
        continue
    }
    fmt.Println(finished.Result.Telemetry.Run.TotalLLMCalls)
}

Token usage vs telemetry

Data	Where	Purpose
Telemetry	`Result.Telemetry`	Run behavior — LLM round count, tool breakdown, storage ops
Token usage	`Result.LLMUsage`	Aggregate token counts for cost and quota tracking

See Token Usage.

Eval harness

Telemetry fields are designed for eval assertions — total LLM calls, tool breakdown, finish reason, and storage counts. See Eval Harness.

Example

SHOW_TELEMETRY on run results

Metrics

OTLP counters and histograms

Tracing

Distributed spans per LLM and tool call

Overview

Getting Started

Runtimes

Features

Advanced

Observability

Testing & Evaluation

Production

Reference Apps

Telemetry

Enable OpenTelemetry

Run-level telemetry (`AgentTelemetry`)

Structure

Run telemetry

Tool telemetry

Storage telemetry

Run

Stream

Token usage vs telemetry

Eval harness

Example

Telemetry

Metrics

Tracing

​Enable OpenTelemetry

​Run-level telemetry (AgentTelemetry)

​Structure

​Run telemetry

​Tool telemetry

​Storage telemetry

​Run

​Stream

​Token usage vs telemetry

​Eval harness

​Example

Telemetry

​Related

Metrics

Tracing

Enable OpenTelemetry

Run-level telemetry (`AgentTelemetry`)

Structure

Run telemetry

Tool telemetry

Storage telemetry

Run

Stream

Token usage vs telemetry

Eval harness

Example

Related