Our monthly infrastructure bill roughly doubled over two years. Traffic grew by a fraction of that. Nobody could account for the difference, and finding out required going through it line by line.
What the audit found
The largest single item was resources nobody was using.
Servers spun up for a project and never shut down. Storage volumes detached from anything and still charged for. Snapshots accumulated automatically with no expiry policy, which had grown to a substantial monthly cost on their own.
Test environments running continuously, including at weekends and overnight, when they were used during working hours by three people.
Load balancers and addresses reserved for services that no longer existed.
Together these accounted for a sizeable share of the bill, and none of them was a decision anybody had made. They were the residue of decisions that had never been unwound.
The second category was over-provisioning
Instances sized for a peak that had been estimated rather than measured, and never revisited.
When we looked at actual utilisation, most were running at a fraction of their allocated capacity continuously.
The reason is straightforward and human. Nobody is blamed for a system that is too large. Somebody is blamed for a system that runs out of capacity during an incident.
So the incentive on every individual sizing decision is to round up, and the aggregate of a hundred conservative decisions is a large amount of unused capacity.
Data transfer, which surprised us
The line item that was largest relative to expectation.
Moving data out of a provider's network is charged, frequently at rates that are substantial relative to storage and compute.
Moving data between regions is charged. Moving between availability zones within a region is charged on some services.
Which meant an architecture that had been designed for resilience, spreading components across zones, was generating continuous cross-zone transfer costs that nobody had modelled.
This is well documented and it is not intuitive, and it is one of the more common sources of unexpected cost.
Managed services and the convenience premium
The category where the trade is real and worth examining.
Managed databases, queues and search services cost substantially more than running the equivalent yourself on plain compute.
What you buy is patching, backups, failover, monitoring and not having to employ somebody who knows how to operate it.
For a small team that is frequently excellent value, since the alternative cost is a person.
For a larger organisation with the expertise already on staff, the premium can be substantial and worth reconsidering, particularly for services that are stable and undemanding.
We moved two things and left the rest, which was the right split for us and would be different for somebody else.
What we changed
Tagging everything with an owner and a project, enforced, so that every line on the bill has somebody responsible for it.
Automatic shutdown of non-production environments outside working hours, which was the single largest saving and took an afternoon to implement.
Expiry policies on snapshots and logs, which had been accumulating indefinitely by default.
Right-sizing based on measured utilisation rather than estimates, with a review cadence.
Commitment-based pricing for the baseline load, which offers substantial discounts in exchange for a term commitment and which we had been ignoring because it required predicting our usage.
And a monthly review of the bill by somebody who cares, which sounds trivial and is the mechanism that keeps all the rest of it working.
The organisational cause
Underneath all of it.
Engineers could provision resources instantly and were not shown the cost of doing so. Finance saw a single large invoice with no breakdown they could interpret.
Nobody was doing anything wrong and nobody had the information to do anything right.
Showing teams their own costs, monthly, changed behaviour more than any policy. People are reasonable about spending when they can see it, and blind when they cannot.
What I would tell somebody starting
Set up tagging and cost allocation before anything else, because retrofitting it across an existing estate is considerably harder.
Set budget alerts, which every provider supports and which almost nobody configures until after a surprise.
Assume that anything created will never be deleted unless something deletes it automatically.
And read the pricing page for data transfer specifically, because it is the cost that is least visible in the architecture and most visible on the invoice.
The tooling
What we used, since the providers supply most of it free.
Native cost analysis tools, which break spend down by service and by tag once tagging exists, and which almost nobody opens.
Automated recommendation services that identify idle and over-provisioned resources, which are reasonably good and worth reviewing rather than accepting wholesale.
Budget alerts at several thresholds, which turn a quarterly surprise into a same-week notification.
Third-party cost management platforms exist and are worth it above a certain scale, and below that the native tooling is sufficient if somebody actually looks at it.