The Economics of Model Serving
Model and serving choices get treated as a quality decision and discovered to be a pricing decision once the bill arrives. Batching, quantization, tiering, and the self-host bet, as cost decisions.
A team picks a model the way they'd pick a library: whichever one scores best on the benchmarks that look closest to their use case. Then the feature ships, the usage numbers come in, and the bill arrives, and only at that point does anyone go back and treat model choice as what it actually was the whole time: a pricing decision wearing a quality decision's clothes.
Three choices decide the economics of a served model, and none of them show up on a capability leaderboard: how the serving layer batches requests, how aggressively the model is quantized, and whether inference runs on rented per-token API calls or self-hosted GPUs.
Batching decides how busy the GPU actually is
A naive serving setup batches requests statically: collect a group, run them through the model together, wait for the whole batch to finish before starting the next one. If one request in the batch needs a long response and the rest are short, the GPU sits idle waiting for the slow one, because the batch can't move on until every sequence in it is done.
Continuous batching, the approach behind vLLM and similar serving engines, breaks that constraint by scheduling at the level of individual decode steps instead of whole requests. A finished sequence exits the batch immediately and a new one takes its slot on the next step, so the GPU stays busy processing tokens instead of waiting on the slowest member of a fixed group. The mechanism doesn't change what the model computes. It changes how much of the GPU's time is spent computing versus waiting, which is most of what determines cost per token at any real request volume.
Quantization is a risk curve, not a free discount
Running a model at lower numerical precision, INT8 or INT4 instead of the original weights, cuts memory footprint and increases throughput, because smaller weights move faster through memory and more of the model fits on a single GPU. The tradeoff is a quality cost that isn't uniform across models. Larger models tend to tolerate aggressive quantization with modest degradation, because they have enough redundant capacity that compressing precision doesn't remove information the model was relying on. Smaller or more specialized models degrade faster, because there's less redundancy to absorb the loss.
The practical implication is that quantization level is a per-model decision to validate against your own eval set, not a global setting to copy from someone else's blog post. What holds for a large general-purpose model at INT4 does not automatically hold for a small fine-tuned model doing a narrow task.
Weight-only quantization, compressing the stored parameters but computing in higher precision, is the safer default: it recovers most of the memory savings with a smaller quality hit, because the arithmetic itself still happens at full precision even though the weights are stored compactly. Quantizing the activations as well, not just the weights, buys more throughput but introduces more of the error that shows up as the model losing coherence on longer or harder generations. The extra throughput is real. So is the extra risk, and skipping the validation step to get to production faster is how a quantization choice becomes a silent quality regression instead of a cost win.
Tiering turns a flat cost into a shaped one
Most systems answer every request with the same model, which means paying frontier-model prices for questions a much cheaper model would answer just as well. A router pattern fixes this: estimate how hard a request is, send the easy majority to a small, fast, inexpensive model, and reserve the expensive model for the fraction that actually needs it. The difficulty estimate can come from the same kind of confidence signal that decides whether an agent's action needs human escalation: cheap model handles it, unless the confidence score says otherwise, in which case the request escalates to a stronger model instead of a human.
def route(request, cheap_model, expensive_model, difficulty_estimator, threshold):
difficulty = difficulty_estimator(request)
if difficulty < threshold:
return cheap_model.generate(request)
return expensive_model.generate(request)
The threshold here is the same kind of dial as an evaluation gate's execute threshold: move it and the tradeoff between average cost and worst-case quality moves with it, and it should be owned as a cost decision rather than left at whatever the router template shipped with.
A two-way split, cheap or expensive, is usually a simplification of what production traffic actually needs. A three-tier version, a small model for the bulk of routine requests, a mid-size model for the ones the small model flags as uncertain, and a frontier model reserved for the fraction the mid-size model can't resolve either, spreads the cost curve further and keeps the most expensive model off the critical path for the majority of traffic. The number of tiers is itself a cost decision: each additional tier buys a finer-grained cost curve at the price of another routing boundary to get wrong.
None of this is visible without tracing what actually happened to a given request, which model handled it, at what cost, and whether the difficulty estimate was right. Observability tooling built for LLM traffic, LangSmith or Langfuse being the common examples, is what turns "the average cost per request went up" into "the router sent too many requests to the expensive tier last Tuesday," and without that trace, a cost regression in a tiered system is nearly impossible to diagnose after the fact.
Self-hosting is a volume bet
A hosted API, Bedrock or an equivalent, charges per token and requires no infrastructure: no GPUs to provision, no serving engine to tune, no capacity planning. Self-hosting requires all of that, in exchange for a cost structure that stops scaling with usage once the GPUs are paid for. The breakeven is a bet on volume and its shape: bursty, unpredictable traffic favors the hosted API, because idle self-hosted GPUs cost the same as busy ones, while sustained high volume favors self-hosting, because the fixed GPU cost gets amortized across enough tokens that the per-token cost drops below what any hosted API charges.
Teams get this backward more often in the self-hosting direction than the other way: standing up a GPU fleet to save money on a workload that turns out to be spiky enough that the fleet sits idle most of the time, at which point the hosted API would have been cheaper and required no one on call for it.
The model was never the whole decision
Two teams can deploy the identical open-weight model and end up with completely different unit economics, because the model was only ever one input to the cost, and the smaller decisions around it, how requests get batched, how aggressively weights get compressed, which requests even reach the expensive model, decided the rest. The team that treated serving as an afterthought finds out what it costs when the invoice arrives. The team that treated it as the actual decision built the invoice on purpose.