A model migration looks like a one-line change. Swap the identifier, run the tests, ship. Then production starts behaving strangely in ways nobody connects to the migration for a week.
The failures cluster. After enough of these, four categories cover almost everything.
1. Output format drift
Your parser was written against one model's habits. The new one produces the same information in a slightly different shape: markdown fences around JSON, a preamble sentence, keys in a different order, an array where you expected a single object.
If you are parsing free-form output, this will bite you. Schema-enforced structured output removes the entire category:
response = client.responses.create(
model=MODEL,
input=prompt,
response_format={"type": "json_schema", "schema": ORDER_SCHEMA},
)
Anything you cannot put behind a schema needs a parser test in your eval set.
2. Refusal boundaries move
Different models draw safety lines in different places. A prompt that worked fine may now get refused, or — more awkwardly — content that used to be refused now goes through.
This is why refusal rate belongs in your eval set as a first-class metric, not as a note in a doc. It is the one number that moves most between model families and the one nobody thinks to measure.
3. Prompt sensitivity is not transferable
Few-shot examples, ordering tricks, "think step by step," specific phrasings that you tuned over months — these are calibrated to a particular model. Some transfer. Some do nothing. A few actively hurt.
The cheapest test is ablation: run the eval set with the tuned prompt and again with a plain version. If the plain version is within noise, delete the tuning. You will be surprised how often it is.
4. Latency and cost profiles invert
A model that is cheaper per token can be more expensive per request if it is more verbose. A faster model can feel slower if its time-to-first-token is worse. Compare on the metric your users experience:
old model new model
$ per request 0.0041 0.0038
tokens out (p50) 180 310
time to first token 310ms 620ms ← users feel this
p95 total 1.8s 2.1s
Cheaper on paper, worse in the product.
Migrate behind a flag, run both models on the same traffic sample, and compare on your own eval set. A vendor benchmark tells you nothing about your prompts.
The checklist
Before flipping the default: schema validation on every structured output, refusal rate in the eval set, an ablation on your prompt tuning, and a side-by-side on time-to-first-token. Four checks, roughly a day. Considerably less than the week you would otherwise spend guessing.