How to Set a Latency Budget for AI Features
A latency budget forces you to decide how slow an AI feature is allowed to be before you build it. How to set one, spend it across the pipeline, and design around it.
Decide how slow an AI feature is allowed to be before you build it, not after users complain. A latency budget is a number you commit to up front: this feature must respond within X, or the design has failed. It sounds obvious and almost nobody does it. They wire up the model call, string together retrieval and a couple of chained prompts, and discover in production that the thing takes eleven seconds because nobody was tracking the running total. A budget turns latency from an accident you measure afterward into a constraint you design against.
The thesis: latency in an AI-native feature is a product decision made at design time, and the budget is how you make it before the architecture locks it in.
Why AI features need an explicit budget
A normal API call is fast and predictable, so you rarely budget for it. Model calls are neither. They are slow, variable, and they stack. A retrieval step, a first model call to plan, a second to generate, a validation pass, maybe a re-rank. Each is a second or more, and they add up fast because they usually run in sequence. Without a budget, that sequence grows one reasonable-sounding step at a time until the feature is unusably slow, and by then the architecture is baked.
Setting the number first inverts the process. Instead of measuring what you happened to build, you decide what is acceptable and force the design to fit. This is treating latency as a product decision, not a metric you passively report.
How to set the number
Anchor the budget to the interaction, not to a vanity target. Different jobs tolerate different waits.
An inline assist the user is actively waiting on, like an autocomplete or a live suggestion, needs to feel immediate. Budget it in the hundreds of milliseconds, and if you cannot hit that, it is the wrong pattern.
A generated result the user requested and expects to take a moment, like a summary or a draft, can spend a few seconds, especially if you stream it instead of spinning so the wait feels productive.
A background job the user is not watching, like an overnight batch or an async agent run, can take minutes. There the budget is throughput and cost, not perceived speed.
Set the number against the human, then hold the design to it.
How to spend the budget across the pipeline
Once you have the total, allocate it across the steps like a cost budget. Each stage gets a slice, and if one stage blows its slice, something else has to give.
Cut sequential calls. The fastest way to blow a budget is chaining model calls that could run in parallel or be collapsed into one. Every extra round trip spends latency you may not have. Ask whether two prompts can become one.
Cache the stable parts. If retrieval or parsing produces the same result across requests, cache it so you only pay for the volatile generation step. This is caching model responses to cut costs doing double duty as a latency tool.
Route by difficulty. Send the easy, high-confidence cases down a cheaper and faster deterministic path and reserve the slow, expensive model for the genuinely hard ones. Not every request needs to spend the full budget.
Pick the model to fit the slice. A smaller, faster model that hits the budget and passes your eval set beats a bigger one that blows it. Capability you cannot afford to wait for is not capability, it is a demo.
When to loosen or ignore the budget
The caveat. Not every feature should chase low latency. If the job is genuinely a background task the user does not wait on, forcing a tight budget just raises cost for no perceived benefit, and you should optimize throughput and dollars instead. And if hitting the budget means shipping answers so bad users reject them, the budget was set wrong, quality has its own floor. The budget is a design constraint, not a suicide pact.
We set a latency budget per model-backed feature across the Girard AI products before writing the pipeline, because a feature that is correct and unusably slow is a feature nobody uses. Decide the number first. Spend it deliberately. Let it kill the designs that cannot hit it, early, when killing them is cheap.