We built a small internal tool on a hosted language model. It worked consistently for several months and then started producing noticeably different output for the same inputs.
Nothing on our side had changed. Understanding what had is worth describing, because anybody building on these systems will encounter it.
The models are updated continuously
The central fact and the one that is easy to forget.
A hosted model behind a stable name is not a fixed artefact. Providers update models, adjust safety behaviour, change system-level instructions, and occasionally migrate a name to a different underlying model.
Some of this is announced. A good deal of it is not, because from the provider's perspective it is an improvement rather than a change.
Which means output that you tested and relied on can shift without any notification, and the first indication is frequently a user complaint.
Why identical inputs give different outputs anyway
The other source of variation, which is inherent rather than a change.
These systems generate text probabilistically. Unless sampling is set to be fully deterministic, the same prompt produces different completions.
Even with deterministic settings, results are frequently not perfectly reproducible, because of the way computation is batched and executed across hardware.
Which means anybody expecting the reliability of a conventional function is starting from the wrong mental model.
What this means for building on them
Several practices that we adopted after being caught out.
Pin to a specific model version where the provider offers one. Most do, and the versioned name is more stable than the general one. It is also deprecated eventually, which requires attention.
Build an evaluation set — a collection of inputs with expected characteristics in the output — and run it regularly, so that drift is detected by a test rather than by a user.
Treat the output as untrusted input to the rest of the system. Validate structure, check ranges, handle the case where it returns something unexpected, because it will.
And keep a human in the loop for anything consequential, which is the recommendation everybody makes and which is frequently dropped once something appears to work.
Prompt fragility
A related problem that surprised us.
Prompts that had been refined against one model version performed noticeably worse against another, because the phrasings that work are partly specific to how a given model was trained.
Which means prompt engineering effort does not fully transfer, and a model upgrade can require rework that looks disproportionate to the change.
The mitigation is to keep prompts as simple and explicit as possible rather than relying on discovered phrasings that happen to work, since simple instructions transfer better.
The deprecation timeline
A commercial consideration worth planning around.
Pinned versions are retired, generally with notice, on timelines set by the provider rather than by you.
Which means anything built on a specific version has a maintenance obligation on somebody else's schedule.
For a small internal tool that is an inconvenience. For something in a product, it is a real dependency that should be understood before committing.
Open-weight models running on your own infrastructure remove that dependency, at the cost of running the infrastructure and generally with lower capability at a given size.
What we do now
Pinned versions, with an evaluation suite run weekly and on any provider announcement.
A documented expectation that output is probabilistic, communicated to the people using the tool, which reduced complaints substantially on its own.
Validation of everything the model returns before it is used.
And a fallback path for when the service is unavailable, since hosted services have outages and a tool that simply stops is worse than one that degrades.
The general point
The useful framing is that you are integrating with a service that changes, not calling a library that does not.
All the practices that apply to any external dependency apply here — version pinning, testing, validation, monitoring, fallbacks — and they apply more rather than less, because the interface is natural language and the failure modes are subtle rather than obvious.
A dependency that fails loudly is easy. One that quietly starts giving slightly different answers is the difficult case, and that is what this is.
Logging what you sent
A practical habit that made diagnosis possible.
We log the exact prompt, the model version, the parameters and the response for every call, with a retention period.
When behaviour changes, that log is the only way to establish whether the input changed, the model changed, or the sampling simply produced something different.
Without it we were arguing from memory about what the system used to do, which is not a productive way to diagnose anything.
The obvious caution is that such logs may contain personal data and need to be treated accordingly.
The cost dimension
A related instability worth planning for.
Pricing for these services has moved substantially and generally downward, which is welcome, and it has also moved in structure — different rates for different context lengths, caching discounts, batch tiers.
Which means a cost model built on today's pricing is unlikely to hold, in either direction.
For anything where per-request cost matters, measuring actual token usage rather than estimating it is worth doing early, since the difference between the estimate and the reality is frequently large.