AWS’s nine-stage agent evaluation loop catches failures that uptime cannot see

AWS’s [October 16 AgentCore article](https://aws.amazon.com/blogs/machine‑learning/building‑trust‑in‑ai‑agents‑a‑production‑ready‑evaluation‑framework‑with‑amazon‑bedrock‑agentcore/) describes nine stages: define quality, compare experiments, test at merge, gate deployment, score a canary, run adversarial tests, detect drift, monitor live health and feed failures back into improvement. The sequence addresses a problem ordinary uptime checks miss. An agent can return HTTP 200 quickly while selecting the wrong tool, using the wrong period or inventing a figure. The system is available, but the decision is unsafe. The practical contribution is the separation of evaluation moments. A pre‑merge regression test answers a different question from a production drift alarm, so one passing test suite should not stand in for the whole lifecycle.

AWS says its framework evaluates individual tools, each sub‑agent and the multi‑agent orchestration layer. Each level has its own golden set, evaluators and promotion criteria. That hierarchy prevents a common blind spot. A sub‑agent may answer correctly in isolation while the orchestrator routes the wrong request to it, or a retrieval tool may return the wrong customer even though the final prose is fluent. Teams can map every production incident to one of those levels, then add a retained case to the smallest test set that would have caught it.

AWS reports that an internal pre‑production mismatch among a data change, tool change and prompt update reduced pass rate from 78% to 38%. The team also found examples including a $410.08 figure labelled as $410.08 million and an unsupported annual recurring revenue estimate. These are company‑reported case‑study results, not an independent benchmark. AWS does not establish that every AgentCore deployment will see the same failure rate or improvement. The useful lesson is narrower: versioned components that are meant to ship together need an atomic acceptance contract, and business‑unit and period errors deserve explicit evaluators instead of generic “helpfulness” scores.

AWS organizes evaluation around safety, quality, performance, cost and operational control. Its example translates business outcomes into explicit signals such as entity resolution accuracy, grounding rate, scope adherence and occurrences of unauthorized calculations. A threshold must include a response when it fails. A merge gate can block a change; a canary alarm can pause rollout; a drift signal can route a sampled failure for review. A dashboard without an executable response only describes the problem. Model‑graded evaluators also need calibration against human‑reviewed cases. Otherwise a fluent judge can repeat the same blind spot as the agent it scores.

Teams do not need nine new services on day one. Start with one workflow whose wrong answer has a clear cost. Define the users, allowed data, correct entities and units, permitted calculations, latency budget and maximum cost. Build a small golden set that includes routine cases and known failures, then run it before merge and before deployment. Add a canary sample in production, preserve the exact model, prompt, tools and data revision, and turn every confirmed failure into a regression case. AWS’s framework is useful because it connects those steps. Its internal metrics remain vendor evidence, while the architecture can be tested against a team’s own incidents and thresholds. *Figure by OpenTools Team; no third‑party expressive material used. It condenses the nine stages into build, release and production control groups.*