An automatic update to a tool we depend on changed a behaviour we relied on, without a note in the changelog, and cost us three days of work to diagnose and route around.
That is an ordinary experience rather than an unusual one, and it is worth understanding why it keeps happening.
Why regressions happen
Not carelessness, mostly. The structural reasons are more interesting.
Software has enormous state spaces. The number of combinations of settings, data, platforms and usage patterns exceeds what can be tested exhaustively by orders of magnitude.
Testing therefore covers the paths that are known and expected. Anything depending on an undocumented behaviour, an unusual configuration or a rare sequence is outside that coverage.
And a great deal of real-world use depends on undocumented behaviour, because users discover what works rather than reading specifications.
When a vendor changes something they consider an implementation detail, they have not broken a documented contract, and they have broken a workflow.
The release cadence pressure
The factor that has made this worse over the last decade.
Continuous deployment means changes ship constantly rather than in discrete releases. That is genuinely good for fixing problems quickly.
It also means less time between change and release, more changes in flight simultaneously, and users receiving updates they did not request at times they did not choose.
The old model of a version you installed deliberately, having read about it, is largely gone for consumer and cloud software.
What is actually within your control
Less than one would like and not nothing.
Deferring updates where the option exists. Many operating systems and applications allow a delay, and waiting a week means somebody else finds the problem first.
Not updating immediately before anything important. This sounds obvious and is the discipline most frequently broken, usually by an automatic update.
Keeping a known-good state. For anything critical, having a way back — a previous version, a backup, a snapshot — converts a disaster into an inconvenience.
And avoiding dependence on undocumented behaviour where you can identify it, which is hard because you frequently cannot tell.
The version pinning trade
For anything with a technical component, the question of whether to freeze versions.
Pinning gives stability and predictability, and accumulates security debt, because unpatched software accumulates known vulnerabilities.
Never pinning gives you the latest fixes and exposes you to every regression.
The workable middle is pinning with a scheduled review — updating deliberately, on a cadence you choose, having read what changed, with a rollback available.
That requires effort that most individuals and small organisations do not have, which is why most people run whatever arrives.
Reporting it
Worth doing and worth understanding what it achieves.
A well-written report with reproduction steps is genuinely valuable to a vendor and does get acted on more often than people assume.
A report saying it does not work is not actionable and will be closed.
What makes a report useful: what you did, what happened, what you expected, the version, the platform, and whether it worked previously. Five lines.
The realistic expectation is that a widely affecting regression gets fixed quickly and a narrow one may never be, because the vendor is weighing the number of affected users against the cost.
The dependency question underneath
The wider lesson from our three days.
We had built a workflow on a specific behaviour of a specific tool, with no alternative, no documentation of the dependency, and no test that would have caught the change.
Nobody had decided to do that. It accumulated, as these things do.
What we changed afterwards was to write down what we depend on and why, which took an afternoon and immediately identified three other single points of failure we had not thought about.
That document is the most useful thing to come out of the incident, and it would have been just as useful written before.
Reading changelogs
A habit worth building for anything you depend on.
Most vendors publish release notes, and they are more informative than they look once you know what to scan for — anything described as a behaviour change, a default change, a deprecation, or a removal.
Bug fixes are rarely the risk. Changes to defaults are, because they alter behaviour for people who never configured anything.
Ten minutes reading before an update on a critical system is a considerably better use of time than three days diagnosing afterwards, which is a lesson I have now learned twice.
The support window afterwards
A related point about what happens when you decline to update.
Vendors support a limited number of versions, and staying on an old one eventually means running something that receives no fixes at all.
Which converts a stability decision into a security decision over time, usually without any announcement.
Knowing the vendor's stated support policy for previous versions tells you how long deferring is viable, and that policy is published far more often than people check.
Testing before rolling out
For anything used by more than one person, a single machine updated first, a week ahead of the rest, catches most of this at almost no cost.