A new model arrives. The launch examples look good, but your application has different inputs, deadlines and failure costs. Start the comparison with the job your product needs done.
Use this worksheet to compare candidates against your own acceptance rule. The examples illustrate a ticket router; apply the checks to your actual inputs and record the results.
Write the acceptance rule before running the models
Choose one concrete job: route a support ticket, extract fields from a receipt, or summarize a source article without adding claims. Describe the output a downstream system can accept. Include what the system should do when the input is ambiguous or incomplete.
For an illustrative ticket router, a valid result might contain a supported queue name and an explicit unknown outcome. A well-formed answer sent to the wrong queue still fails. Agree on the expected result before seeing the candidate's answer.
Use the same cases, then inspect the disagreements
Keep a representative set of ordinary requests and difficult cases. Include malformed input, missing context, unsupported requests and the languages your users actually use. Remove personal or confidential data from shared fixtures. Hold the task instructions and available evidence constant for the first comparison.
Keep a separate set for the final decision after tuning prompts. Review cases where the models disagree rather than judging only a polished demo. Repeat important cases to inspect output variation. A single successful response is useful evidence about that response, not a reliability estimate.
Price the complete accepted result
Count the work around the model: retries, tool calls, validation and any review needed to produce an accepted result. Record response time across the cases instead of comparing only the fastest answer. Use current provider pricing and your observed usage; this guide supplies neither a price table nor a savings claim.
For example, a cheaper call that needs another model to repair an invalid output may have a higher total workflow cost. Whether that happens in your application is something to measure. Keep quality failures visible rather than burying them inside an average cost number.
Test the fallback as its own candidate
Run the same acceptance checks on the fallback. Deliberately simulate a timeout, rate limit and invalid response in a test environment. Decide which failures can retry and which need a clear unavailable state or a person.
Start with a limited rollout and a rollback path. Expand only when the candidate meets your written acceptance rule. A model that is excellent at another task can still be the wrong choice here.