Most agent projects that die in production died in a conference room first, and nobody in the room noticed. The demo answered the question. The slide looked convincing. The pilot got funded. Six months later the thing is writing eight hundred duplicate records into a CRM table and the team is arguing about whether the model or the prompt is at fault.
The survey data is blunt about how often this happens. A March 2026 study of 650 enterprise technology leaders found that 78% had at least one agent pilot running, and 14% had scaled a single agent to organization-wide use. Of the pilots that produced meaningful results, roughly two in three, only about 10% reached production. Forrester's 2026 estimate is harsher: around 88% of agent pilots never graduate at all. Gartner expects task-specific agents inside 40% of enterprise applications by the end of 2026, up from less than 5% in 2025.
Read those numbers together and the problem stops looking like a technology gap. Hundreds of agents are heading for production carrying the same preparation the last cohort carried, and the last cohort mostly stalled. Current models are good enough for a large share of back-office work. The difficulty is that an agent is a non-deterministic program with side effects, and most teams still verify it the way they verify a slide deck. They look at it.
An evaluation harness is what turns an agent into software you can defend in front of a risk owner. Four parts, none of them exotic: a fixed set of cases, a runner that records what actually happened, scorers that put a number on the result, and a gate in CI that stops a bad change from shipping. The first version takes about two weeks. A single production incident review usually takes longer than that and teaches you less.
The demo only tests the path you wrote last week
A demo is a script. Someone typed the three questions the agent handles well, in an order that keeps the context clean, against a model version that may not exist next month. That is a smoke test. Smoke tests do not catch the failures that cost money: the tool call that fires twice on a network retry, the refund issued before the order lookup finishes, the customer who switches to French halfway through, the record lookup that returns two matches and gets resolved by guessing.
Every one of those is an ordinary distributed-systems failure wearing a new hat. The difference is that a traditional service fails loudly with a stack trace, and an agent fails with a confident, well-formatted paragraph that happens to be wrong.
Start with twenty cases you have already seen fail
The dataset comes before the runner code. Twenty real cases beat two hundred synthetic ones, because synthetic cases encode what you assume about how the system breaks, and those assumptions are exactly why you have bugs. Pull the cases from support tickets, from the Slack thread where someone wrote "that's weird, watch this", from the run where the agent booked a meeting in 1997.
What each case has to pin down
- The input, copied verbatim from the system rather than cleaned up for readability.
- The outcome you would accept, written in a form a machine can check.
- The tools the agent is allowed to touch while getting there.
- The model version, prompt version, and tool list the run used.
That last one matters more than it looks. A case that cannot be reproduced is not a test, it is an anecdote. Provider model updates ship without warning, and a prompt that lives in a Google Doc changes without a commit. Both count as releases, and neither shows up in a changelog.
Record the trajectory, not just the final answer
Two agents can return the same correct reply while one of them reads a customer's entire order history to do it. Final-answer scoring cannot tell them apart. Capture the full sequence of steps: every tool call, its arguments, what came back, token counts, wall time, and step count.
Trajectory capture has to happen from the first version, not after the scoring discussion. Retrofitting a scorer onto traces you never stored means re-running everything, and the re-runs are where evaluation budgets go to die.
Trajectory scoring also catches the failure that hurts most at scale: arriving at the right answer by a path that would be wrong for a different user. An agent that answers a pricing question by pulling the internal price sheet is helpful for a sales rep and a data leak when the account manager for a competitor asks the same question. Both runs score 100 on accuracy. Only one of them is a finding.
Score with assertions first and a model judge second
Most of what you need to check is deterministic, and checking it with a language model is a waste of money and a source of noise. Did the agent call the right tool. Did the computed total match to the cent. Does the returned JSON validate against the schema. Did the date land inside the allowed window. Did the agent write to any table outside its list. These are assertions, they are fast, and they fail for one reason at a time.
Judges earn their place on the softer dimensions: tone, relevance, whether a summary is faithful to the source document. Calibrate a judge against your own labels on fifty cases before you trust its output on five thousand, and keep a human review queue for the handful of cases where the business cost of a wrong answer is high. A judge that mostly agrees with itself measures agreement, not quality.
The gate needs four checks, not one
Accuracy alone hides the failures that matter. A release gate that only measures correctness will cheerfully ship a change that doubles cost and triples latency because the score went up two points. The checks that hold up in practice are:
- A floor per environment. Below a set score, the build does not ship, regardless of trend. Floors stop a slow slide that individual deltas hide.
- A regression delta. More than a few points below the last release stops the build and sends someone looking for the cause.
- Cost per task. Tracked as a first-class number next to the score, since an agent that gets there in twelve tool calls is a different product from one that gets there in three.
- Latency and step count at p95. The average is a comfortable lie in agent workloads, because the slow tail is where users abandon the task and where retry loops start compounding.
Write the override path down while you are building the gate: a named approver, a reason, an expiry date. A gate that can be overridden deliberately, twice a year, stays alive. A gate with no override gets routed around within a month, and then nobody is measuring anything.
Put security in the same suite
AGAT's 2026 security survey found 82% of executives confident their policies protect against unauthorized agent actions, while only 14.4% of organizations send agents to production with full security or IT approval. The gap between those two numbers is where incident reports come from.
Permission behavior is testable, so test it in the same harness instead of writing a separate policy document nobody reads.
Three cases worth writing first
- A request outside the agent's scope, where the suite asserts that it declines and routes the request to a human.
- A tool result carrying instructions the agent should ignore, where the suite asserts that it does ignore them.
- A request that would write to production, where the suite asserts that it stops at the staging boundary.
These cases are cheap to write and they fail loudly, which is the point. They also give a security reviewer something to read other than a diagram.
Cost per task and step count belong in the test output
An agent that takes twelve steps where three would do costs money, and the cost hides inside a passing test suite. At fifty thousand tasks a month, the difference between four calls and twelve is real money, plus three times the blast radius when a dependency degrades. Reporting cost and steps next to the score makes the tradeoff visible at review time instead of at the quarterly invoice.
It also changes design conversations. Teams that see per-task cost in the CI output start caching lookups, batching reads, and tightening tool descriptions, because the number is sitting there next to the pass rate.
Production traces feed the set every week
A golden dataset only covers the failures someone already knew about. Once real users arrive, input drifts, new intents appear, and edge cases turn up outside the original twenty. The maintenance loop is simple: sample production traffic, flag the runs that look wrong, promote them into the dataset with an expected outcome, and let the gate catch the repeat. Half an hour a week keeps the suite honest.
Do this before you buy tooling, not after. The commercial options in 2026 are solid, from hosted evaluation platforms to built-in evaluators in the major cloud AI stacks, and any of them will run the loop once the loop exists. None of them will tell you which twenty cases matter for your business. That part is the work.
Two weeks, roughly, in this order
Week one: cases, then a single runner
Days one and two go to collecting cases from tickets, logs, and the people who use the thing. Days three to five go to the runner and trace capture, with one case running end to end and a score attached to it. Resist building all four parts in parallel. One case that runs and reports beats a framework that runs nothing.
Week two: scorers, gate, budgets
Add the deterministic scorers, wire the gate, set the floors and the cost and latency budgets, and print those numbers in the output. Then point the same harness at a sample of production traffic, because the runner you built for release gating is most of what continuous evaluation needs.
The first week usually turns up a failure nobody knew about. That finding is worth as much as the harness, and it is the cheapest one the team will get.
An agent without a harness is a prototype with a launch date on it. You do not control the model. You control the cases, the traces, the thresholds, and the gate, and those are what let you tell a risk owner, a customer, or an auditor what the system does when it goes wrong.
If your team has an agent in a pilot and no eval set behind it, that is the piece to build before the production date gets any closer. Talk to Tech Hub Services about running the first harness against your own cases.
