Agent Quality Has to Be Measured at the Run Level

After the August 17 brief, Digg AI and Digg Tech both surfaced Prime Intellect’s large autonomous-research experiment as a stronger signal than another model-launch cycle. The primary source matters because it did not just rank models on a static prompt set: it ran 153 autonomous research attempts across 18 frontier models, with runs lasting up to eight days and publishing the resulting traces. That makes the current shift more practical for builders. Simon Willison’s recent LLM release shows the tooling layer catching up with the same need through visible reasoning traces and content-addressable logs, while Addy Osmani’s loop-engineering guidance keeps landing on the management lesson: your real leverage is outside the model, in the goals, checks, and artifacts that let you inspect a run instead of merely watching a score. The useful synthesis is that agent quality is becoming a harness question, not just a model question.

A lot of teams still compare models as if a better benchmark score will automatically produce a better product workflow. That breaks down once agents can search, edit, retry, call tools, and burn hours of compute without producing a trustworthy outcome. If you only capture final outputs, you cannot see repeated dead ends, fake convergence, budget blowups, or the exact move that made one run reusable. Builders who log traces, score accepted runs, and promote behaviors with evidence will improve faster than teams still shopping for a single magic model.

Turn one recurring agent workflow into a run-level evaluation harness. Trigger: pick a repeated task where the team currently argues about which model or prompt is best, such as bug reproduction, PR repair, support drafting, research synthesis, or growth analysis. Context: define the task packet, input data, allowed tools, expected artifact, and the success rule before the run starts. Tools: give the worker a fixed harness with model choice, tool permissions, retry policy, and optional scratchpad or search tools, then log every important event instead of only the final answer. Verifier: use an external acceptance check such as tests, schema validation, rubric scoring, downstream state confirmation, or human review against a narrow checklist. Budget: cap attempts, wall-clock time, spend, and tool calls per run so bad trajectories fail cheaply. Artifacts: save the trace, tool calls, intermediate files, verifier outputs, accepted artifact, rejected variants, and a short failure classification. Stop condition: end the run when the verifier passes, when the run repeats without new evidence, when the budget is exhausted, or when the task requests a forbidden action.

Pick one workflow your team describes with model names instead of outcomes and rewrite the review around accepted runs: what passed, what failed, what the trace showed, and what policy change you will test next. If you cannot inspect the run, you are not really improving the workflow.