The agent looked right. The path was broken.

Traditional software gives you a promise: the same input returns the same output. You write a unit test, it passes, you move on. That model has worked for decades and it's deep in our muscle memory.

Multi-step agents make no such promise. An agent reads context, picks a tool, interprets what comes back, adjusts its plan, and produces an answer through a chain of decisions that can look different on every run. So the question stops being "did it get the right answer?" and becomes something harder: "can I trust the path it took to get there?"

A unit test can't answer that. It can tell you a function returned the expected value. It can't tell you whether the agent chose the right tool, passed sensible arguments, handled an empty result instead of swallowing it, or looped three times before landing on something plausible. The most dangerous failure mode isn't a crash. It's a confident answer sitting on top of a broken trajectory.

This isn't a theoretical concern anymore. Industry surveys in 2026 put well over half of organisations already running agents in production, with output quality cited as the top barrier to getting there. At the same time, Gartner has predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027, pointing to escalating costs, unclear business value, and inadequate risk controls.

When I read those two facts together, I don't read "slow down." I read "the teams that win will be the ones who can actually see what their agents are doing." Speed without visibility is how you end up in that 40%.

The reframe I keep coming back to is simple to say and harder to live by. For an agent, the trajectory is the product. It's every decision, tool call, argument, result, retry, and backtrack along the way. If you only score the end state, you miss the behaviours that decide whether the thing holds up at scale.

In practice, I think about evaluation across five layers:

• Tool correctness. Did it call the right tool with valid, sensible arguments? Catches misuse hiding behind a passing answer.
• Step quality. Was each individual step accurate and useful? Shows you where the error entered the chain.
• Trajectory coherence. Was the sequence logical and efficient, or did it loop and backtrack? Catches the silent process failures.
• Task completion. Did it actually satisfy the user's goal? Your headline pass/fail.
• Business outcome. Did completing the task create real value?

That last layer is the one most teams skip, and it's the one I care about most. An agent can pass every technical check and still fail to move the number that justified building it. Tying evaluation back to a business outcome is what keeps you honest about whether the agent is working or merely passing.

A golden dataset you actually maintain. A curated set of real test cases, each with the input, the tool sequence you expect, the criteria for an acceptable answer, and a note on the specific failure it's meant to catch. The best cases I've used come straight from production logs (with private data stripped), not from a whiteboard. 50 cases is enough to learn from. 100 to 200 is a real baseline for production. Anything touching finance, health, legal, or sensitive data should sit well above that. And these decay, so review them quarterly and feed new failures back in.

Calibrated LLM-as-judge, not magic. Letting a model grade open-ended output scales beautifully, but only with a rubric. A good rubric doesn't ask "is this good?" It defines what good means, with concrete anchors: did it use the right tool, stay within the user's intent, flag uncertainty, handle missing data? Then let automated judges run across the full set and escalate the low-confidence, high-risk, and outlier cases to a human. The cost gap is enormous, but human review still earns its keep on the novel and the high-stakes calls.

Evaluation as a deployment gate, not a quarterly ritual. If your evals don't run on every pull request and block the merge when quality drops, they're decoration. For Python teams, the common pattern is pytest plus an eval framework such as DeepEval, with a pass-rate gate. 90% is a reasonable default, lower for experiments, higher for anything regulated or handling sensitive data. This isn't about slowing teams down. It's about making quality visible before it reaches users.

Tracing that closes the loop. Pre-launch testing catches the failures you anticipated; production monitoring catches the ones you didn't. Every production request should emit a structured trace (tool called, arguments, raw response, next action, final output), and standards like OpenTelemetry keep that data portable across tools. Then the loop that matters most: every real failure becomes a future test case. Review the trace, confirm it, strip the private data, annotate the failure mode, add it to the dataset. Your suite should get stronger every single time the system breaks.

I don't think "agent evaluation" stays a specialism for long. In a year or two, it'll look the way automated testing looks today: not impressive, not optional, just the baseline cost of being allowed to ship.

The teams that internalise that early get a real edge. Not because their models are smarter, but because they can change a prompt, swap a tool, or upgrade a model and know within minutes whether they made things better or worse. Everyone else is shipping on vibes and finding out from their users.

There's an honesty dividend in here too, which is the part I find most useful. Rigorous evaluation forces you to define what "good" actually means, in numbers, before you ship. That conversation alone kills a surprising number of bad ideas and sharpens the good ones.

So if I could leave a team-building with agents with one thing, it's this: if you can't see the path your agent took, you don't know whether it works. You just haven't been surprised yet.

At Seen Ventures, we spend a lot of our time helping teams move agents from prototype to production, designing the evaluation, tracing, and feedback loops that make reliability visible. If you're wrestling with any of this, I'd genuinely like to compare notes.

• LangChain. State of Agent Engineering. 2026. https://www.langchain.com/state-of-agent-engineering
• Gartner. Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Press release, 25 June 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
• Cemri, M., et al. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657, 2025. https://arxiv.org/abs/2503.13657
• Braintrust. LLM-as-a-Judge vs Human-in-the-Loop Evals. 2026. https://www.braintrust.dev/articles/llm-as-a-judge-vs-human-in-the-loop-evals