Test an AI Agent Safely

New to the concepts? Start with what an AI agent is and what tool calling is.

Why testing an agent is different

A chat model returns one answer you can grade. An agent runs a loop, so testing has to cover the whole path, not just the destination.

Three properties make it hard. Agents are non-deterministic, so the same input can produce different runs. They are multi-step, so a wrong turn early can be silently corrected or silently compounded. And they take real actions, so a bad test can delete data or send an email.

The consequence is that final-answer testing is not enough. A pass can look green while the trajectory was a mess, and reliability problems only show up when you run the same task many times.

Prerequisites

Pin your setup. Stating what you tested on is good practice and an honest signal to readers.

• A defined task with a clear, checkable goal.
• A sandbox where tool calls hit fake data and reversible side effects, never production.
• An evaluation harness to run tasks, capture full trajectories, and score them.
• Test credentials and test data, never live secrets.

Step 1. Define state-based success

The most common testing mistake is grading the agent's final message. An agent can say "I booked your flight" without booking anything, and a text match will happily pass it.

Grade the world instead of the words. Check the resulting state: was the record created, the ticket closed, the file written? This is exactly why tau-bench verifies the database state rather than the agent's closing sentence.

Where the task has rules, score compliance separately from completion. An agent that resolves the request by breaking a policy has not really succeeded.

Step 2. Build a representative evaluation set

A public benchmark tells you how an agent does on tasks you already knew about. It tells you little about your traffic.

Build a set from real cases: the tasks your users actually bring, including the messy and ambiguous ones. Cover the common paths, the edge cases, and the failure cases where the right move is to stop and ask a human.

Keep it a living set. Every production failure becomes a new test, so the suite grows toward the problems you actually have.

Step 3. Sandbox the agent

Never test an agent where it can do real harm. A test that runs against live systems is not a test, it is a production incident waiting to happen.

Run the agent in an isolated environment with fake data and reversible actions. Apply least privilege: give it only the tools the task needs, and nothing that can touch real money, real messages, or real infrastructure. Anthropic's agent guidance makes the same point, recommending sandboxed testing with guardrails before you widen an agent's autonomy.

This containment is also what makes aggressive red-teaming safe, which comes later.

Step 4. Score the trajectory, not just the answer

Outcome tells you whether the agent succeeded. Trajectory tells you whether it succeeded for the right reasons, and whether it will keep succeeding.

Capture and score the full path: which tools it called, with what arguments, in what order, and how it recovered from errors. Two things hide in a single pass/fail. Looping is invisible in a trace total, since a 20-step run and a 3-step run both return one answer, so attribute cost per step to see it. And a lucky recovery can mask a wrong first move that will not always self-correct.

A messy trajectory behind a green result is a warning, not a win.

Step 5. Measure reliability, not best-case

Because agents are non-deterministic, one successful run proves very little. The question is whether it succeeds consistently.

Run each task several times and measure reliability, not the best attempt. A useful metric is pass^k, the probability of succeeding on the same task k times in a row, which surfaces the flakiness a single run hides. An agent that passes once in five is not ready, however good that one run looked.

Report the spread, not just the peak. Consistency is the property production actually needs.

Step 6. Use an LLM judge carefully

Some qualities resist exact-match scoring, like whether an answer was helpful or a trajectory was sensible. An LLM judge can score these against a rubric.

Give the judge the task, the full trajectory, and clear criteria such as goal fulfilment, tool-use appropriateness, and response quality. The main risk is bias: a judge tends to favour outputs that resemble its own style. Reduce it by using a different model family for the judge than for the agent, and by spot-checking the judge against human labels.

Treat the judge as a useful signal, not ground truth. For anything high-stakes, keep a state-based check as the source of truth.

Step 7. Red-team for safety

A functional agent is not a safe agent. Before shipping, attack it the way an adversary would, inside the sandbox.

Probe the known weaknesses. Try prompt injection hidden in the content the agent reads, tool poisoning in tool descriptions, and inputs designed to trigger destructive or out-of-scope actions. Confirm that the agent refuses, asks for approval, or fails closed rather than doing damage.

Track an explicit unsafe-action rate. A test suite that only measures success will pass an agent that is helpful and dangerous at once.

Step 8. Guardrails and human-in-the-loop

Testing tells you where an agent is weak. Guardrails keep those weaknesses from becoming incidents in production.

Add controls the agent cannot talk its way past: input and output validation, permission gates on sensitive tools, and explicit human approval before irreversible actions. Keep the tool surface small and the scopes narrow, the same discipline covered in the guide to MCP security and tool poisoning.

Guardrails and evaluation work together. One reduces the blast radius while the other tells you where the blast is likely to come from.

Step 9. Monitor in production

Offline evaluation and online monitoring answer different questions, and a reliable agent needs both. Your test set covers what you anticipated; production shows you what you did not.

Log every run: the prompt, the tool calls, the arguments, and the outcome, tied together so you can trace one task end to end. Watch for drift, rising escalation rates, new failure patterns, and unexpected tool use. Feed every real failure back into the evaluation set.

This closes the loop, and it matches the evidence-first, auditable posture behind ModelRefs methodology.

Metrics that matter

Track behavior, cost, and safety together. No single number is enough.

Metric	What it measures	Watch out for
Task success (state-verified)	Did the agent actually complete the task	Do not grade the final text alone
Trajectory quality	Were the steps and tool calls right	Invisible in a single pass/fail
Tool-call accuracy	Right tool, right arguments	Small models fail here often
Reliability (pass^k)	Succeeds k times in a row	One green run hides flakiness
Cost per successful task	Efficiency of the loop	Attribute cost per step to catch looping
Unsafe-action / escalation rate	Safety and restraint	A success-only suite ignores it

Common mistakes

Most bad agent tests share the same roots.

The frequent errors:

• Grading the final message instead of the resulting state.
• Running once and trusting a green result.
• Testing against live systems.
• Skipping trajectory scoring, so looping and near-misses go unseen.
• Skipping red-teaming, so unsafe behavior ships untested.

Fixing these is mostly rigor. Verify state, run repeatedly, sandbox everything, and measure safety as explicitly as you measure success.

Sources

• Anthropic, Building Effective Agents (engineering) — sandboxed testing, guardrails, human oversight, and starting simple.
• Sierra Research, tau-bench — state-verified success over final-text matching, separate policy-adherence scoring, and pass^k reliability.

Methodology: evaluation practices are asserted in ModelRefs' own voice as the reference layer, cross-checked against Anthropic's agent guidance and the tau-bench benchmark on 17 Jul 2026. This article is defensive: all adversarial testing described assumes a sandboxed environment.