Token budgets are a product decision, not an infra one

The first time an LLM bill surprises a team, the reaction is almost always technical: cache more, switch to a cheaper model, trim the system prompt. Those help. But the number was largely fixed the moment someone wrote "the assistant should remember the whole conversation."

Cost per request is a product decision wearing an infrastructure costume.

Write the budget down before you build

Every feature that calls a model should carry three numbers in its spec:

• Target cost per request
• Target p95 latency
• What happens when either is exceeded

That last one is the interesting column. Most teams have no answer, which means the fallback is "be slow and expensive forever."

Where the tokens actually go

Run the arithmetic before writing code. A chat feature with full history looks harmless until you count:

system prompt          900 tokens   (every turn)
retrieved context    3,200 tokens   (every turn)
conversation history   +400 tokens  per turn, cumulative

By turn ten you are sending roughly 8,000 tokens to produce maybe 300. Ninety-six percent of what you pay for on that request is re-reading things the model already saw.

Three levers move that number, in descending order of impact:

1. Cache the stable prefix. System prompt and tool definitions do not change between turns. Prompt caching turns them into a fraction of their cost.
2. Summarise history instead of replaying it. Keep the last few turns verbatim and compress everything before that. Users rarely notice; the token count halves.
3. Retrieve less, better. Four well-ranked documents beat twenty mediocre ones on both cost and accuracy.

Latency is the same conversation

Perceived latency is not total latency. A response that streams its first token in 400 ms feels faster than one that completes in 2 s with nothing on screen. Budget for time-to-first-token separately from time-to-completion, because the fixes are different: the first is about prompt size and model choice, the second is about output length.

If a feature has no cost ceiling, it has an infinite one.

The habit worth building

Put a cost and latency line in every AI feature spec, the same way you would put an error budget on a service. It costs nothing at design time and saves the rewrite six months later.