The Experimental Loop Is Becoming The Product
The durable product-builder skill is no longer asking a model for one better answer. It is building a closed experimental loop that can generate options, execute trials, evaluate outcomes, and decide what to try next with an auditable trail.
The strongest August 5 signal came from Digg Tech surfacing Jeff Dean and colleagues launching Discovery Loop around the explicit goal of automating experimental loops in science and engineering. The primary claim on the company’s launch materials is not just “better AI for researchers.” It is that the bottleneck is the loop itself: propose an experiment, run it, inspect the result, refine the hypothesis, and repeat at a scale humans alone cannot sustain. That is important beyond science startups. Boris Cherny and other operators have been making the parallel product-builder point for months in software: leverage is shifting from handcrafted prompts toward repeatable loops with goals, tools, verification, and iteration. The useful synthesis is that the loop is becoming the actual product surface.
Many teams still deploy AI as a one-shot assistant and then wonder why gains plateau. One-shot generation is fine when the task is obvious and the output can be eyeballed. It is weak when value comes from searching a space, comparing alternatives, running trials, or learning from failures. Closed loops turn “AI assistance” into an operating system for progress. They also make product economics clearer, because the work can be budgeted, instrumented, and improved instead of disappearing into chat transcripts and intuition.
Turn one repeated discovery or optimization problem into a closed-loop system. Trigger: a workflow depends on trial, feedback, and revision, such as prompt improvement, ad creative testing, pricing experiments, bug fixing, research synthesis, or sales-message iteration. Context: define the objective function, the search space, available tools, risk boundary, and how outcomes will be scored before the loop starts. Tools: give the system the minimum actions needed to generate options, execute or simulate trials, record outcomes, and queue the next attempt. Verifier: require an external scorer such as tests, conversion metrics, evaluator judgments, or human review on edge cases. Budget: set caps on runs, token spend, experiment cost, and review time, plus a trigger for escalation when the loop stalls. Artifacts: save the objective, experiment log, traces, verifier scores, winning candidates, and stop reasons. Stop condition: the loop ends when it reaches a predeclared threshold, exhausts its budget, or stops improving for a defined number of iterations.
Take one workflow currently handled as “ask the model again” and rewrite it as a loop with an objective, verifier, budget, and stop condition. If the next attempt is not informed by the prior result, you still have retries, not a learning loop.
For tasks where the answer is already obvious and cheap to inspect, a closed loop may add unnecessary overhead compared with a simple single-pass generation plus review.
The Discovery Loop launch makes automated experimental cycles the core product thesis, and that direction aligns with broader operator movement toward loop engineering over one-shot prompting.
Did converting the workflow from retries to a scored loop increase output quality without hiding runaway cost?
Watch: improvement per iteration · cost per accepted outcome · stall rate · human interventions per loopApply it now
Knowledge only counts when it changes the build.
Take one workflow currently handled as “ask the model again” and rewrite it as a loop with an objective, verifier, budget, and stop condition. If the next attempt is not informed by the prior result, you still have retries, not a learning loop.
- Stage
- build
- Produce
- Closed-loop design for one repeated product or engineering problem
Full context at Discovery Loop. Bring back one decision, test, or workflow change.
Read the original ↗Keep Going