Observability for AI Applications: Monitoring, Tracing and Debugging LLM Calls

The Bottom Line: AI Observability Is Not “Add a Few More Logs”

After integrating LLM calls into production, teams quickly discover that the traditional observability trinity (logs, metrics, traces) falls short for AI workloads.

LLM calls have three characteristics that break traditional approaches:

1. Extreme latency variance — same model, same prompt, response times can differ by 10x
2. Cost and output are coupled — slow requests are also expensive (more tokens)
3. Output cannot be asserted — free-text cannot be schema-validated as “right or wrong”

This article covers three layers: metrics collection → tracing with context → debugging and evaluation.

---

1. Metrics: The Foundation Layer

1.1 Metrics You Must Collect

Category	Metric	Description	Aggregation
Latency	TTFT	Time to first token	P50/P95/P99
Latency	TPOT	Time per output token	P50/P95/P99
Latency	End-to-end	Full request to complete response	P50/P95/P99
Cost	Input tokens	Tokens per call	Sum/daily average
Cost	Output tokens	Tokens per call	Sum/daily average
Cost	Per-call cost	Calculated by model unit price	Daily/monthly
Quality	Error rate	Grouped by status/error type	Percentage
Quality	User feedback	Thumbs up/down/report	Ratio
Quality	Response length	Output character count	Mean/distribution

1.2 Implementation

async function tracedLLMCall(params: {
  model: string;
  messages: ChatMessage[];
  metadata?: Record<string, string>;
}): Promise<LLMResponse> {
  const start = performance.now();
  const traceId = crypto.randomUUID();

  try {
    const response = await openai.chat.completions.create({
      model: params.model,
      messages: params.messages,
      stream: true,
    });

    let ttft: number | null = null;
    let outputTokens = 0;
    let fullResponse = '';

    for await (const chunk of response) {
      if (!ttft) ttft = performance.now() - start;
      outputTokens += chunk.usage?.completionTokens ?? 0;
      fullResponse += chunk.choices[0]?.delta?.content ?? '';
    }

    const endToEnd = performance.now() - start;

    metrics.record('llm.ttft', ttft, { model: params.model });
    metrics.record('llm.e2e', endToEnd, { model: params.model });
    metrics.record('llm.output_tokens', outputTokens, { model: params.model });
    metrics.record('llm.cost', calculateCost(params.model, inputTokens, outputTokens), { model: params.model });

    return { response: fullResponse, traceId, usage: { inputTokens, outputTokens } };
  } catch (error) {
    metrics.record('llm.error', 1, { model: params.model, errorType: error.code });
    throw error;
  }
}

1.3 Metric Aggregation Notes

• Do not average latencies — LLM latency follows a long-tail distribution; P95 is the meaningful metric
• Group by model and prompt pattern — different models and contexts produce vastly different metrics; mixing them is meaningless
• Cost and latency must be correlated — a “slow” request is almost always an “expensive” request

---

2. Tracing and Context Preservation

2.1 Call Chain

A typical AI application call chain:

User request → Gateway → Agent/orchestration → LLM call → Tool call → LLM call → Response

Traditional tracing tools (Jaeger, Zipkin) handle RPC tracing well, but LLM context is far more complex — we need to know:

• What prompt was used
• What content was returned
• How many tokens were consumed
• Whether there were intermediate tool call (Function Calling) results

2.2 Implementation

Extend OpenTelemetry with custom Span attributes for LLM calls:

import { Span, trace, SpanStatusCode } from '@opentelemetry/api';

async function tracedLLM(model: string, messages: ChatMessage[]): Promise<LLMResponse> {
  const tracer = trace.getTracer('llm-instrumentation');
  const span = tracer.startSpan('llm.call', {
    attributes: {
      'llm.model': model,
      'llm.request.messages': JSON.stringify(
        messages.map(m => ({ role: m.role, content: truncate(m.content, 500) }))
      ),
    },
  });

  try {
    const response = await openai.chat.completions.create({ model, messages });
    const usage = response.usage;

    span.setAttributes({
      'llm.response.tokens.input': usage?.promptTokens ?? 0,
      'llm.response.tokens.output': usage?.completionTokens ?? 0,
      'llm.response.total_tokens': usage?.totalTokens ?? 0,
      'llm.response.finish_reason': response.choices[0]?.finishReason ?? 'unknown',
      'llm.latency_ms': performance.now() - start,
    });

    // Store full response in a separate Trace Store (not in span attributes — too large)
    await traceStore.put(span.spanContext().spanId, {
      messages,
      response: response.choices[0]?.message,
      usage,
    });

    span.setStatus({ code: SpanStatusCode.OK });
    return response;
  } catch (error) {
    span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

2.3 Sensitive Data Handling

Full prompts and responses may contain PII. Handling strategy:

Data Level	Example	Handling
Metadata	Model name, latency, token count	Write to Span attributes, aggregatable
Sanitized	PII-stripped prompt excerpt	Write to Trace Store, 7-day retention
Raw	Full prompt and response	Do not persist; store only a hash for audit

---

3. Debugging and Evaluation Workflow

3.1 Online Debugging

When a user reports “the AI gave a wrong answer”, traditional logs tell you “input X, output Y” — but you need to answer “why did the AI give that answer.”

Context required:

• Complete prompt (System + User + conversation history)
• Raw model output
• Token usage and latency
• Function calling results (if applicable)
• Temperature and other parameters

Recommended practice: retain full call context in Trace Store, searchable by User ID, Session ID, or Trace ID. When a user reports an issue, ops can reconstruct the full call context in one click.

3.2 Regression Testing (Golden Dataset)

After a model upgrade or prompt change, how do you ensure output quality has not regressed?

Build a Golden Dataset:

golden-dataset/
  test-cases/
    - case-001.json    # Simple Q&A
    - case-002.json    # Multi-turn conversation
    - case-003.json    # Tool calling
    - case-004.json    # Edge cases (empty input, very long input)
  expected/
    - expected-001.json  # Expected output (or evaluation criteria)

Run regression tests on every change:

llm-eval run --dataset golden-dataset/ --model gpt-4o-mini --output results/
llm-eval diff --baseline results/v1.0/ --current results/v1.1/

3.3 Automated Evaluation Metrics

Dimension	Method	Description
Accuracy	Judge LLM scoring	Another LLM evaluates “did it answer correctly”
Hallucination rate	Factual consistency	Does the output contradict the given context
Instruction following	Completeness check	Expected format, all required fields
Safety	Content safety scan	Harmful content, system prompt leakage
Semantic similarity	Embedding distance	Vector distance between output and expected output

---

4. Toolchain

Purpose	Tool	Description
Metrics	OpenTelemetry + Prometheus	Standard metrics pipeline
LLM tracing	OpenTelemetry + custom Spans	Standard approach, define your own LLM Span attributes
Dedicated LLM observability	Langfuse / Arize Phoenix / LangSmith	Purpose-built for LLM, works out of the box
Evaluation	deep-eval / Ragas	Golden Dataset evaluation and auto-scoring
Debug UI	Langfuse / Weights & Biases Prompts	Visual prompt/response/token consumption view

---

Summary

Layer	Key Points	Common Mistake
Metrics	Group by model and prompt pattern, watch P95 not average	Added latency monitoring but not cost or quality
Tracing	OpenTelemetry Span + separate Trace Store	Writing full prompts into logs (data leak risk)
Debugging	Retain full call context, one-click replay	Only “user says it’s wrong” — no reproducible context
Evaluation	Build a Golden Dataset, run regression on changes	Never measured output quality after going live

AI observability is not a “cost” — it is the only tool you have to answer “why did the AI answer that way” in production. Without it, every user report of “the AI got it wrong” is a blind diagnostic exercise.