ModelRefs / AI Observability (Prometheus, Grafana) — Tutorial
AI Observability (Prometheus, Grafana) — Tutorial
Monitor LLM latency, token costs, error rates, and drift in production with Prometheus and Grafana. Covers Why AI observability is different.
Overview
Monitor LLM latency, token costs, error rates, and drift in production with Prometheus and Grafana
Level: Expert. Estimated reading time: 40 minutes.
Why AI observability is different
Traditional software observability: is the service up? Is latency < 200ms? Are errors < 0.1%? These are sufficient for CRUD APIs.
AI systems have additional failure modes invisible to standard metrics:
Hallucination rate: the model returns confident but wrong answers. Not visible in error rates — the request succeeds (HTTP 200) but the answer is wrong.
Prompt drift: input distributions shift over time. A model trained on 2023 data starts receiving 2025 queries with different vocabulary and context.
Latency spikes: LLM latency varies dramatically by output token count. p99 latency may be 10× the median. You need token-aware latency histograms.
Cost explosion: token usage drives API costs. An unexpected spike in input length (a user sending a 50,000-word document) can cost $10/request instead of $0.01.
The three pillars — metrics, logs, traces — all apply but need AI-specific extensions: Metrics: token counts, latency per token, refusal rate, tool call success rate. Logs: full prompt/response pairs (with PII redaction), sampling strategy for cost. Traces: end-to-end latency breakdown across retrieval, LLM, post-processing.
Prometheus metrics for LLM APIs
Prometheus collects time-series metrics via HTTP scraping. Your FastAPI service exposes a /metrics endpoint; Prometheus scrapes it every 15 seconds.
Key metrics to instrument for an LLM API:
Counters (only go up): - llm_requests_total{model, status} — total inference calls - llm_tokens_total{model, type="input/output"} — token consumption (drives cost) - llm_errors_total{model, error_type} — timeouts, context_overflow, content_filter
Histograms (distribution): - llm_request_duration_seconds{model} — end-to-end latency (compute p50, p95, p99) - llm_tokens_per_second{model} — throughput (output tokens / latency) - llm_input_length_tokens{model} — prompt length distribution
Gauges (can go up or down): - llm_active_requests{model} — in-flight requests - llm_context_utilization{model} — fraction of context window used
Export with the prometheus_client library: expose_metrics creates the /metrics endpoint that Prometheus scrapes.
Grafana dashboards and alerting
Grafana connects to Prometheus and renders dashboards. For LLM APIs, a production dashboard should have:
Row 1 — Health: request rate (req/s), error rate (%), p95/p99 latency.
Row 2 — Cost: input tokens/min, output tokens/min, estimated cost/hour (multiply by model pricing).
Row 3 — Model behaviour: refusal rate, average output length, tool call rate, fallback rate (when main model fails and a backup is used).
Row 4 — Infrastructure: GPU utilisation, GPU memory, CPU, pod restarts.
Alerting rules (PagerDuty/Slack integration): - Error rate > 1% for 5 minutes → critical - p99 latency > 10s for 5 minutes → warning - Token rate > 2× baseline → cost anomaly, warning - GPU memory > 90% → warning (OOM risk)
Distributed tracing with OpenTelemetry: instrument each request with a trace ID that propagates through retrieval → LLM → post-processing. Visualise in Jaeger or Grafana Tempo to find latency bottlenecks across the pipeline.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AI Observability (Prometheus, Grafana) — Tutorial.