Teams now ship agents: software where a model runs a multi-step task on its own: reads a support ticket, looks up the account, decides, replies. Unlike normal software, it doesn't fail loudly. An engineer opens a run that went wrong and finds a 60-step trace where every step looks correct in isolation. The actual error was that step 4 retrieved the wrong customer's record, and steps 5 through 60 reasoned flawlessly about the wrong person. Nothing in existing tooling points at step 4.

I designed Verdict around three constraints that kill the obvious solutions. The same input legitimately produces different outputs, so any screen showing one run as the result is lying. "Correct" is a judgement call, not a boolean, which means graders, and graders disagree with each other. And traces are tens of megabytes, so nothing can be rendered in full. The product that comes out of those three is not a dashboard: it's a comparison tool that names which specific cases changed direction, shows every case as a distribution rather than a result, and treats grader disagreement as the most valuable signal in the system instead of averaging it away.

Key Decisions

No single quality score: Every stakeholder wants one number for standup. A composite score destroys the most valuable signal in the system: when a human grader and a model grader disagree, it means the judge is miscalibrated, the human is wrong, or the rule is ambiguous: all three are urgent and actionable. Averaging them produces a number that is precisely wrong and looks authoritative. What I gave PMs instead was grader agreement coverage, which is the honest version of the number they were asking for.

Comparison instead of a dashboard: Score-over-time answers "is it worse" and stops exactly where the user's problem starts. Worse, it hides the failure that costs money: the average improves two points while eleven previously-passing cases now fail. The home surface leads with regressions and fixes as counts of named cases, and the aggregate is demoted to a secondary line. The cost is a genuinely worse first-run experience, because comparison needs a baseline and week one has none - so cold start got designed as its own screen.

A suggestion that admits it's a suggestion: The trace opens collapsed except for a heuristic path to the first step whose inputs differ from comparable successful runs. The heuristic is deliberately dumb and sometimes wrong, so it is marked as a suggestion, dismissible in one click, and it never hides steps - only collapses them. A confident wrong answer here would be worse than no answer, because the engineer would stop looking.

Masking that preserves shape: Traces contain customer data, so values are masked. But a uniform-width block would have destroyed the clue that mattered: in the failing run, the retrieved record was 4.2 KB where successful runs retrieved 1.1 KB. Masked values keep their real length and carry a type label, so the engineer can still see that the wrong thing came back without seeing whose it was.