AI Agent Observability: Logs, Traces, and Metrics in Production

Monitoring traditional applications is a solved problem. You track CPU cycles, memory usage, API response times, and database transaction latencies. If an endpoint errors, you look at the stack trace.

AI agents break this traditional Application Performance Monitoring (APM) paradigm. An agent is non-deterministic. It executes non-linear reasoning loops, calls dynamic sets of external APIs, updates short-term memory stores, and runs multiple sequential LLM calls. If an agent outputs garbage, you cannot diagnose it using a CPU spike or a simple stack trace. The system did not crash; it just reasoned incorrectly.

To run agents reliably in production, you must build an observability architecture centered on logs, traces, and metrics designed specifically for non-deterministic software.

---

Traditional APM vs. Agent Observability

Traditional APM tools check if the system is up and performing. Agent observability checks if the system is reasoning correctly and behaving efficiently.

Dimension	Traditional APM	Agent Observability
Primary telemetry	HTTP latency, CPU, error rates	Spans, prompt versions, token counts
Error definition	HTTP 5xx, uncaught exceptions	Hallucinations, tool loops, bad formatting
Trace structure	Linear HTTP client-to-server calls	Non-linear reasoning and tool graphs
Data type	Structured JSON, plain text logs	System prompts, model outputs, tool inputs

---

Logs: Capture the Trajectory

In agentic systems, logs must capture the reasoning trajectory. A trajectory is the exact sequence of thoughts, decisions, tool invocations, and raw API payloads that led to the final output.

A production agent log must store:

• The System Prompt: The exact prompt template and variables used at the start of the session.
• The Internal Thoughts: The model’s reasoning steps, often extracted from ReAct thoughts or XML tags.
• Tool Input and Output: The raw arguments sent to a tool and the exact string or JSON returned.
• Model Configuration: The model identifier, temperature, frequency penalty, and top-p settings.

Avoid storing these as raw, unstructured text files. Write them as structured JSON documents containing a session_id and a run_id to make them searchable. When a user reports that an agent gave an incorrect answer, you can query by the session_id to inspect the exact prompt, thoughts, and tool inputs that caused the bad output.

---

Traces: Map the Execution Graph

Tracing is the most critical component of agent observability. Because agents call tools that might trigger nested LLM calls, you need to visualize the hierarchy of operations.

Using OpenTelemetry, you can represent each agent run as a trace. Every LLM call, vector database query, and tool execution becomes a span. By setting parent-child relationships, you create a nested execution graph.

graph TD
    ParentSpan["Parent Span: User Query (Trace ID: 0x9f3)"]
    ParentSpan --> ChildSpan1["Child Span: LLM Call 1 (Reasoning)"]
    ParentSpan --> ChildSpan2["Child Span: Tool: db_search"]
    ParentSpan --> ChildSpan3["Child Span: LLM Call 2 (Synthesis)"]
    ChildSpan2 --> SubChildSpan1["Sub-Child Span: Vector DB Retrieval"]

This hierarchy lets you isolate bottlenecks. If a query takes 15 seconds, the trace will show exactly where that time was spent: 2 seconds querying the vector database, 8 seconds on the first LLM call, and 5 seconds on the second.

OpenTelemetry Python Implementation

The following Python sample shows how to use the standard OpenTelemetry SDK to instrument an agent loop with nested spans for tool execution and LLM inference.

import time
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

# Initialize OpenTelemetry Tracer
provider = TracerProvider()
processor = SimpleSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent-observability")

def call_vector_db(query: str):
    """Simulate a vector database search."""
    with tracer.start_as_current_span("vector_db_retrieval") as span:
        span.set_attribute("db.system", "chromadb")
        span.set_attribute("db.query", query)
        time.sleep(0.5) # Simulate latency
        return ["customer_record_1", "customer_record_2"]

def execute_tool(tool_name: str, args: dict):
    """Execute an agent tool with custom span tracking."""
    with tracer.start_as_current_span("tool_execution") as span:
        span.set_attribute("tool.name", tool_name)
        span.set_attribute("tool.arguments", str(args))
        
        if tool_name == "db_search":
            results = call_vector_db(args.get("query", ""))
            span.set_attribute("tool.output_length", len(results))
            return results
            
        time.sleep(0.2)
        return "success"

def call_llm(model: str, prompt: str):
    """Simulate an LLM API invocation."""
    with tracer.start_as_current_span("llm_call") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.prompt", prompt)
        
        # Simulate token usage metrics
        span.set_attribute("llm.prompt_tokens", len(prompt) // 4)
        time.sleep(1.2)
        completion = "The user records search was successful."
        span.set_attribute("llm.completion_tokens", len(completion) // 4)
        return completion

def run_agent_loop(user_input: str):
    """Main agent execution trace containing nested child spans."""
    with tracer.start_as_current_span("agent_run") as span:
        span.set_attribute("agent.input", user_input)
        
        # Step 1: LLM decides to search
        thought = call_llm("gpt-4o", f"Decide action for: {user_input}")
        
        # Step 2: Agent runs search tool
        tool_result = execute_tool("db_search", {"query": user_input})
        
        # Step 3: LLM synthesizes final answer
        final_answer = call_llm("gpt-4o", f"Synthesize result from {tool_result}")
        span.set_attribute("agent.output", final_answer)
        return final_answer

if __name__ == "__main__":
    run_agent_loop("Find details for account 9410")

---

Metrics: Track Costs and Performance

Agent metrics help you monitor operational cost, system health, and token usage trends over time.

Export the following metrics to your dashboards:

• Token Consumption: Track input, output, and cached tokens per model. Since providers charge per token, this metric correlates directly with cost.
• Token Efficiency: Monitor the ratio of cached tokens to total tokens. High cache hits reduce both costs and latencies.
• Trace Latencies: Measure the duration of the entire agent loop alongside individual LLM and tool spans to catch performance regressions.
• Tool Error Rates: Track the percentage of tool invocations that return exceptions. High tool failures usually indicate that the model is generating bad inputs or that tool APIs are unstable.
• Step Count per Run: Monitor the number of reasoning steps the agent takes. An unexpected spike in average step counts points to agents getting stuck in infinite loops.

---

Frequently Asked Questions

How do you trace nested tool calls in agentic workflows?

Nested tool calls are traced by creating child spans under the parent LLM reasoning span, passing the parent span context using OpenTelemetry context propagation.

How do you track LLM token usage and costs asynchronously?

Token usage is extracted from the LLM API metadata response and recorded using custom counter metrics in OpenTelemetry, which are then aggregated and sent to a telemetry collector.

What is the difference between system metrics and semantic metrics in AI observability?

System metrics measure infrastructure properties like latency, CPU usage, and network errors, whereas semantic metrics track output attributes such as accuracy, hallucination score, and token cost.