Put a 30-Case Gate on Every Fast Decision Call
A faster, cheaper decision call is only a win if a small set of real cases shows the answers did not get worse.
- For the Decisions API, OpenAI says developers define a question and its allowed answers, then supply text, images or native JSON as context.
- OpenAI lists the price as $0.10 per million input tokens and $0.01 per million cached input tokens.
- Box says that in its own evaluation the API picked routes about 5x faster on 30 insurance-certificate cases, with the same accuracy as the Responses API.
- OpenAI’s own starting advice is to take one recurring decision, define its possible answers, and test it on examples that reflect real usage.
OpenAI published the general-availability announcement on 22 October 2026. The customer figures in it, including Box’s 30-case comparison, are claims from those teams’ own tests as quoted by OpenAI, not independent benchmarks.
Most products make lots of small routing and classification calls that nobody reviews one by one. Moving them to a faster, cheaper endpoint is tempting, and a quiet accuracy drop there shows up later as wrong routes, bad escalations or miscategorised data.
Pick one recurring yes/no or pick-one decision in your product. Pull 30 real inputs with known correct answers, run them through your current call and the candidate, and compare accuracy, p95 latency and cost per call before you switch.
Put it to work 30 minutes
A 30-case switch gate for one recurring product decision
Write a one-page switch gate for one decision: the question, the allowed answers, 30 labelled cases, current versus candidate accuracy, p95 latency and cost, and the accuracy floor below which you keep the old call.
See a worked example
Use one decision your product already makes, such as routing a support request or tagging an upload. If you do not know the correct answer for a case, leave it out rather than guessing.
- Decision and allowed answers
- Which queue should this ticket go to? billing / bug / account / other
- 30-case test set
- 30 tickets from last week, routed and confirmed by the support lead.
- Switch rule
- Switch only if the candidate gets at least as many of the 30 right and p95 latency drops.
Your turn
Write the question and the closed list of answers.
Say where the real inputs and known answers come from.
State the accuracy floor and what you will compare.
Your draft stays in this browser. No account needed.
Thirty cases will not catch rare failures. If a wrong answer is costly or unsafe, such as refunds, account access or regulated routing, use a larger labelled set and keep a human review path for low-confidence answers.
Facts were checked against OpenAI’s announcement. Customer speed and accuracy figures are vendor-published claims; the switch gate is an editorial suggestion to test in your own product.
Did the candidate call match your current accuracy on the 30 cases, and did it hold after a week of real traffic?
Watch: Accuracy on the labelled 30-case set, current versus candidate · p95 latency and cost per call after the switch · Decisions that users or reviewers override or report as wrong