Guide

AI Agent Evaluation: What to Test Before Launch

Turn a convincing demo into a repeatable release decision with realistic cases, explicit grading and checks on business outcomes.

Two software engineers reviewing test results together at a monitor

Key takeaways

  • Grade the completed task and the actions taken to reach it.
  • Include missing information, refusals and service failures.
  • Keep a fixed regression set and repeat variable cases.

AI agent evaluation checks whether an agent completes a defined task correctly, within its permissions and under realistic conditions. A fluent answer is only one part of that result. If an agent promises that a refund was issued but never updates the order system, the task failed. Design the evaluation around the business state you expect after the interaction.

Start AI agent evaluation with an acceptance rule

Choose one workflow and write its success condition in ordinary language. For a hypothetical returns assistant: identify the authorized customer’s order, apply the current policy, prepare the permitted resolution and escalate exceptions. Specify which steps the agent may perform and which require a person. This prevents the test from rewarding persuasive answers to the wrong task.

Turn that statement into observable checks. Does the order belong to the requesting customer? Was the correct policy version used? Did the assistant ask for a missing identifier instead of guessing? Did it stop before an action requiring approval? These checks are more useful than asking whether the conversation looked helpful.

Build agent test datasets from decision boundaries

Use sanitized examples of actual work when authorized, and add clearly synthetic cases for situations the team expects but has not encountered. Keep customer details out unless necessary and appropriately controlled. Preserve the property that makes a case difficult, such as two orders with similar identifiers or a policy change during a conversation.

  • Ordinary cases with complete information and an allowed outcome.
  • Ambiguous requests that require a clarifying question.
  • Requests exceeding the user’s or agent’s permissions.
  • Unavailable tools, stale results and interrupted operations.
  • External content attempting to redirect the workflow.

For each case, save the input, starting system state, allowed actions and expected outcome. A successful refusal needs an explicit expected result just as much as a successful lookup. Otherwise the evaluator may reward an agent for acting when it should have stopped. Keep realistic difficult cases rather than smoothing every input into a tidy demonstration.

Grade actions separately from the final answer

A grading plan for a returns assistant
DimensionEvidenceFailure example
Task outcomeFinal state in a test order systemClaims a refund without a recorded refund
Tool-call accuracyTool name, arguments and authorization resultLooks up the wrong customer’s order
Policy adherencePolicy version and permitted actionApproves an exception without review
CommunicationClear explanation of actual progressSays complete while approval is pending

Use deterministic checks for facts that can be tested directly, including identifiers and state changes. Reserve judgment-based grading for qualities such as clarity or whether an explanation addresses the request. If a model assists with grading, compare its judgments with human review before relying on its scores.

Anthropic’s agent evaluation guide distinguishes tasks, trials and graders. The practical implication is to keep the scenario separate from the mechanism that scores it. That makes disagreements easier to investigate without rewriting the test case.

Repeat cases and read the failures

An agent can take different paths through the same task. Repeat important cases and record how often each required condition is met. One successful run does not establish reliability. Version the model, prompt, tool definitions and starting data so a later regression has a meaningful comparison point.

Read failed transcripts alongside tool logs. A wrong result may come from retrieval, a misleading tool description, a missing permission check or an unclear expected answer. Changing the prompt alone can hide the underlying issue. Inspect a sample of passing cases too: a weak grader can mistake a plausible response for a correct outcome.

Agree on AI agent release criteria before reviewing scores

Separate release-blocking failures from quality improvements. Unauthorized access, repeated writes and false claims of completed actions deserve explicit gates. Set acceptable ordinary-task performance with the business owner, accounting for consequences and available human review. An average score should not conceal a severe failure in a small subgroup.

Keep a fixed regression set and a separate development set. When a production incident reveals a new failure, add a representative case after resolving it. The observability plan supplies live evidence; evaluation turns it into a repeatable release check.

Request an evaluation pack containing the task definition, representative cases, grading rules, known failures and release recommendation. Include it in the product engineering scope. If the team cannot define a correct outcome, use a discovery session to settle that question before expanding the agent’s authority.

Frequently asked questions

How many evaluation cases are enough?

There is no universal count. Cover important decisions, permission boundaries and failure modes, then expand as evidence reveals gaps. A representative set is more useful than many nearly identical easy cases.

Can another AI model grade every result?

A model can assist with subjective judgments, but direct checks should verify measurable facts and system changes. Human calibration helps identify unreliable grading.

Have a workflow or customer experience to improve?

Tell us where work gets stuck and what a better result would look like. We'll help you define a practical next step.