Training a large model is expensive and widely reported. Serving it is less visible and, for anything that becomes popular, becomes the larger number over time.
Training is a fixed cost, serving is not
Training runs once and produces a set of weights. Whatever it cost is spent regardless of whether the model is used afterwards.
Serving costs scale with usage. Every request consumes computation, and a model handling continuous traffic accrues cost every second it operates.
For a model with substantial adoption, the cumulative serving cost passes the training cost and continues rising for as long as the model remains available.
Generation is sequential
Text is produced one token at a time, and each token requires a pass through the model that depends on the tokens already generated.
This limits how much the work can be parallelised within a single response, and it means long outputs cost proportionally more than short ones.
Processing the input is comparatively cheap because it can be handled in parallel, which is why input and output are priced differently.
Memory bandwidth is the constraint
Generating each token requires reading the model's weights from memory, and for large models that transfer dominates the time taken rather than the arithmetic.
Serving efficiently therefore means keeping the hardware busy by batching many requests together so a single pass over the weights serves several users.
Batching improves cost per request while adding latency, which is the central trade in how these services are engineered.
Idle capacity is expensive
Accelerators must be provisioned for peak demand, and demand varies through the day. Hardware sitting idle still costs the same to hold.
Providers respond by moving lower-priority work into quiet periods, which is why batch processing is offered at a substantial discount.
Users who can tolerate delayed results are effectively paid to smooth demand, and the discount reflects the value of that smoothing.
Smaller models change the arithmetic
Techniques that reduce precision, prune unnecessary weights or train a compact model to imitate a larger one all cut the memory traffic per token.
The result is a model that is cheaper to serve at some cost in capability, which is acceptable for the many tasks that never needed the largest model.
This is why routing simple requests to small models and reserving large ones for difficult work has become standard practice in production systems.