Model Upgrades Are Deployment Events

Swapping a model takes a minute. You change one string in a config file, restart the service, and a different system starts answering your users. Nothing in your code changed. A good deal of your system's behaviour just did.

That gap is where migrations go wrong. Over months, a production system quietly grows around one model's habits. Prompts are tuned against its phrasing. Parsers expect its formatting quirks. Tool calls follow its rhythm, retry logic assumes its latency, cost forecasts assume its tokenizer. The replacement may top every public benchmark and still be worse at your specific job, in specific places, for reasons no leaderboard will show you.

So we treat a model change the way we treat any other deployment: an event with a before and an after, evidence gathered on both sides, and a way back. The model is a dependency. Upgrading it deserves at least the suspicion you would give a major-version bump of a library your product cannot run without. More, actually. A library does not rewrite its own behaviour between releases.

Pinning buys time. It does not buy permanence.

You have two ways to name a model, and each fails differently. Call an alias, the short undated name, and the provider can move it to a newer snapshot underneath you. Nothing errors. Your system just drifts, and the first symptom is a slow slide in output quality that nobody can date. Pin a dated snapshot instead and behaviour stays put, but you have scheduled your own migration: snapshots retire.

The clocks are public and they are not generous. On Friday, OpenAI shuts down a batch of legacy snapshots (GPT-4, GPT-4 Turbo, the first GPT-4o snapshot, o1, o3-mini and o4-mini among them), with replacements named on its deprecations page. The named replacements are GPT-5.6 models, Sol, Terra and Luna by tier, and a second wave is already scheduled behind this one: GPT-4o's later snapshots, GPT-4o mini, GPT-4.1 and GPT-4.1 mini retire on January 14, 2027. OpenAI commits to at least six months' notice for generally available models, three months for specialised variants, and possibly as little as two weeks for previews. Six months sounds comfortable until the work lands in the quarter when everything else ships too.

For generally available models, Anthropic's floor is shorter. Its model deprecations page promises customers with active deployments at least 60 days' notice before a publicly released model retires, and requests to a retired model simply fail. Claude Sonnet 4.5 was deprecated on September 30, 2026 and retires on November 30. Anthropic names Claude Sonnet 5.5 as the replacement. The same page carries a quieter trap: on Claude 4.7 and later, setting temperature, top_p or top_k to a non-default value returns a 400 error. Code that never changed models can still break on the day it is forced to move.

And if you reach the same model through Amazon Bedrock or Google Cloud, check that platform's schedule too. The provider's clock is not always your platform's clock.

A better model can still regress your workload

The evidence that upgrades hurt real prompts is not folklore. In a case study of toxicity detection on social media text, 58.8% of prompt and model combinations lost accuracy after an API update, and among those that dropped, 70.2% dropped by more than 5%. The update was newer. The workload was worse.

Slices are the operative word. An aggregate eval score can hold steady while one segment of your traffic falls off a cliff. If your evals only report the average, you will promote a regression and call it an upgrade.

Run the migration like a release

Start with an inventory, because the model ID never lives in one place. It sits in application config, in worker env vars, in scheduled jobs, in the eval harness, in the docs nobody updated. List every caller before you touch anything.

Then freeze a baseline. Run your eval set against the current model and keep the outputs, scored by slice. The slice is the unit that matters: document type, language, request shape, whatever partitions your traffic.

Replay the same set against the candidate. Compare slice by slice rather than by the total, and compare cost and latency alongside quality. A candidate that is two points better and three times slower is not obviously a win.

Only then let the candidate touch real work. Shadow first: it answers live requests, its outputs are logged and scored, nobody acts on them. Canary second: a small share of traffic, watched slice by slice. Promotion is a config change, and so is rollback, flipping back to the pinned ID. That is exactly why the whole sequence has to finish while the old model is still being served. A rollback plan with an expiry date is a deadline wearing a costume.

One migration, traced

Say the workload is invoice extraction: an agent reads incoming invoices and posts structured totals to an accounting system. The old snapshot retires in six weeks. Here is the shape of a migration that respects the clock.

The baseline is frozen by document type: standard invoices, credit notes, multi-page scans, invoices in a second language. The candidate scores two points higher overall and drops sharply on credit notes, where it returns the gross figure where the old model returned the net. Nothing in the aggregate score objected. The slice did.

The repair is narrow: an explicit instruction about credit notes, tested against that slice first, then the full set to confirm nothing else moved. The candidate shadows for several days of real traffic, takes a 5% canary share, and is promoted with two weeks to spare. Every step is ordinary release engineering. The only unusual part is remembering that a model needs it.

Where the gate leaks

An eval set measures the workload you had when you built it. Traffic drifts, new document types arrive, and a gate that passed in spring can certify a migration in autumn against a world that no longer exists. Refresh the set from recent production failures, or it becomes a museum.

Outputs are also non-deterministic, even at settings that should pin them down. A single replay run will show differences that are noise wearing the costume of signal. Repeat runs and thresholds, not exact-match comparisons, are what separate a regression from a coin flip.

And the rollback really does expire. Once the retirement date passes, the old behaviour is gone. Not deprecated, gone. Any problem you find afterwards gets fixed forward, on the new model, under time pressure. That is the argument for starting at the deprecation notice rather than the shutdown reminder. It is also the argument for not jumping on day one: a replacement that has been serving other people's traffic for a few months has a track record your evals can learn from. Use the notice window. It is the one part of this the providers actually give you for free.

Work with us

If your systems are pinned to snapshots with retirement dates on them, or worse, riding aliases nobody has scheduled, we run the migration like the release it is. Start at Trida AI's intake.