AI Products Need a User-Specific Theory of Good

The strongest July 11 signal came from Digg’s AI and Tech clusters surfacing Thinking Machines Lab’s new manifesto, “The Future Worth Building Is Human,” and the operator discussion around it. The essay argues that useful AI should extend human will and judgment, not flatten organizations into one centrally trained worldview. John Schulman made the timing explicit in his follow-up: after the recent jump in agent capability, the important open problem is not just more autonomy, but better ways for humans and models to work together. Miles Brundage’s thread is a useful external read on the same moment: he frames the post as an attempt to operationalize augmentation, not merely gesture at it.

Many teams still treat “human in the loop” as a safety checkbox at the end of a run. That misses the product opportunity. The real moat is often the tacit judgment inside a team: what counts as on-brand, what tradeoff wins in a gray area, which failure is acceptable, which evidence is decisive, and when to escalate instead of guess. If the model cannot absorb and update against that local theory of good, autonomy tends to average the team’s edge away. Products that preserve user judgment as first-class state will feel more trustworthy, adapt faster, and compound distinctiveness instead of eroding it.

Turn one repeated AI-assisted workflow into a judgment-capture loop. Trigger: a recurring decision such as ranking leads, editing copy, triaging bugs, drafting support replies, or reviewing pull requests. Context: define the objective, the house style, the non-negotiable constraints, and 10 to 20 examples of accepted versus rejected outputs. Tools: use an editable spec, a rubric or pairwise preference interface, memory for stable preferences, and optional fine-tuning or profile-level adaptation when the workflow is frequent enough. Verifier: every run must be scored against the team’s rubric, with overrides captured as new preference data instead of disappearing into chat history. Budget: set the maximum review minutes, error budget, and retraining or recalibration cadence up front. Artifacts: maintain a spec, gold set, evaluator, preference changelog, and escalation rules for ambiguous cases. Stop condition: the system acts within the encoded policy on routine cases and reliably hands novel or high-stakes cases back to a human.