Write the AI Incident Playbook Before You Need It
The more an AI feature feels like core product behavior, the more it needs normal product incident discipline.
Fable going offline exposed a gap many AI teams still have: they can demo the happy path, but they cannot explain what changed when the model route, policy layer, or output quality shifts. That is fine for prototypes and unacceptable for paid workflows.
Customers do not care whether a failure came from model access, provider policy, retrieval, orchestration, or your app. They care that yesterday the workflow worked and today it does not. Product builders need enough logging and language to diagnose the change without turning every incident into a mystery.
Add three things to any serious AI feature: a run log that shows model, tools, policy interventions, and retrieval context; an eval baseline for the fallback path; and a plain-English incident template that explains degraded AI behavior without overpromising certainty.
Measurement decision record
Write a one-page AI incident playbook for one feature: symptoms, likely causes, checks to run, fallback behavior, user message, and the metric that proves recovery.
Open the related Product Analytics exercise →Keep Going