The Economics of Model Serving

Most teams treat model choice as a quality decision. You pick the leaderboard winner that looks closest to the use case, ship the feature, and discover the bill.

Three choices decide the economics of a served model, and none of them show up on a capability leaderboard. They are how the serving layer batches requests, how aggressively the model is quantized, and whether inference runs on rented per-token API calls or self-hosted GPUs. Tiering sits next to those three; it decides which requests even reach the expensive model.

Batching decides how busy the GPU is

A naive serving setup batches requests statically: collect a group, run them through the model together, wait for the whole batch to finish before starting the next one. If one request in the batch needs a long response and the rest are short, the GPU sits idle waiting for the slow one. The batch cannot move on until every sequence in it is done.

Continuous batching, the approach behind vLLM and similar serving engines, schedules at the level of individual decode steps instead of whole requests. A finished sequence exits the batch immediately and a new one takes its slot on the next step. The GPU stays busy processing tokens instead of waiting on the slowest member of a fixed group.

The mechanism does not change what the model computes. It changes how much of the GPU's time is spent computing versus waiting. That ratio is most of what determines cost per token at any real request volume.

This is table stakes in any serving engine you would pick. A first setup still ships the static batch. The invoice then looks like a model-price problem until someone measures wait time against compute time.

Quantization is a risk curve

A drop to INT8 or INT4 instead of the original weights cuts memory footprint and increases throughput. Smaller weights move faster through memory; more of the model fits on a single GPU. The tradeoff is a quality cost that is not uniform across models.

The larger models tend to tolerate aggressive quantization with modest degradation. They have enough redundant capacity that compressing precision does not remove information the model was relying on. Smaller or more specialized models degrade faster. There is less redundancy to absorb the loss.

The quantization level is a per-model decision to validate against your own eval set. It is not a global setting to copy from someone else's blog post. What holds for a large general-purpose model at INT4 does not automatically hold for a small fine-tuned model doing a narrow task. INT4 is a precision. You live with the quality number on your eval set or you do not ship it.

The safer default is weight-only quantization. You compress the stored parameters and compute in higher precision. It recovers most of the memory savings with a smaller quality hit, because the arithmetic itself still happens at full precision even though the weights are stored compactly. Quantizing the activations as well, not just the weights, buys more throughput. It also introduces more of the error that shows up as the model losing coherence on longer or harder generations.

The extra throughput is real. So is the extra risk. Skipping the validation step to get to production faster is how a quantization choice becomes a silent quality regression instead of a cost win.

Tiering turns a flat cost into a shaped one

Most systems answer every request with the same model, which means paying frontier-model prices for questions a much cheaper model would answer just as well. A router pattern fixes this. You estimate how hard a request is, send the easy majority to a small, fast, inexpensive model, and reserve the expensive model for the fraction that needs it.

The difficulty estimate can come from the same kind of confidence signal that decides whether an agent's action needs human escalation. Cheap model handles it, unless the confidence score says otherwise. Then the request escalates to a stronger model instead of a human.

def route(request, cheap_model, expensive_model, difficulty_estimator, threshold):
    difficulty = difficulty_estimator(request)
    if difficulty < threshold:
        return cheap_model.generate(request)
    return expensive_model.generate(request)

The threshold here is the same kind of dial as an evaluation gate's execute threshold — move it and the tradeoff between average cost and worst-case quality moves with it. You should own it as a cost decision rather than leave it at whatever the router template shipped with.

The function invents no costs. A reader who plugs a fake threshold gets a fake curve. The inputs are a difficulty estimator you have measured and a threshold you have priced; filling those in from a blog post produces a number that looks like a plan and is not one. If the estimator is wrong, the threshold does not select the right tier. The cheap tier eats requests it cannot handle, or the expensive tier eats requests it should not have seen.

A two-way split, cheap or expensive, is usually a simplification of what production traffic needs. A three-tier version puts a small model on the bulk of routine requests, a mid-size model on the ones the small model flags as uncertain, and a frontier model on the fraction the mid-size model cannot resolve either. That spreads the cost curve further. It keeps the most expensive model off the critical path for the majority of traffic.

The number of tiers is itself a cost decision. Each additional tier buys a finer-grained cost curve at the price of another routing boundary to get wrong.

The wrong tier stays invisible without a trace of what happened to a given request: which model handled it, at what cost, and whether the difficulty estimate was right. Observability tooling built for LLM traffic, LangSmith or Langfuse being the common examples, is what turns "the average cost per request went up" into "the router sent too many requests to the expensive tier last Tuesday." Without that trace, a cost regression in a tiered system is nearly impossible to diagnose after the fact.

Self-hosting is a volume bet

A hosted API, Bedrock or an equivalent, charges per token and requires no infrastructure. No GPUs to provision, no serving engine to tune, no capacity planning. Self-hosting requires all of that, in exchange for a cost structure that stops scaling with usage once the GPUs are paid for.

The breakeven is a bet on volume and its shape. Bursty, unpredictable traffic favors the hosted API, because idle self-hosted GPUs cost the same as busy ones. Sustained high volume favors self-hosting, because the fixed GPU cost gets amortized across enough tokens that the per-token cost drops below what any hosted API charges.

Most teams get this backward more often in the self-hosting direction than the other way. You stand up a GPU fleet to save money on a workload that turns out to be spiky enough that the fleet sits idle most of the time. At that point the hosted API would have been cheaper and required no one on call for it.

The model was never the whole decision

Two teams can deploy the identical open-weight model and end up with completely different unit economics. The model was only ever one input to the cost. The smaller decisions around it — how requests get batched, how aggressively weights get compressed, which requests even reach the expensive model — decided the rest.

Write down how you batch, how far you quantize, where the route threshold sits, and whether the volume is bursty enough that Bedrock still wins. Do that before you pick the leaderboard winner.