What the fast path actually skips
The roughly 60 ms target is not a faster language model. It is a route that avoids calling one.
Most latency advice for tool calling starts with the model: use a smaller one, cache the prompt, stream the response. All of that helps, and none of it gets you to tens of milliseconds. A generation step is measured in hundreds of milliseconds at best, and often in seconds.
Bluwing's fast path is fast because it does not generate. It resolves.
Two kinds of turn
A request can be handled in one of two ways.
Resolve. The engine matches the request against the tree, checks that the required arguments are grounded, and emits the call. No model writes a sentence. The work is ranking, comparison and validation over short strings.
Reason. The request is ambiguous, spans several steps, or needs a value that is only available after another call. The engine falls back to a reasoning path, and that path has the latency you would expect.
The fast path is the common case by design. Bookings, lookups, status checks and simple sends are resolves. Multi-step tasks that need a plan are not, and pretending otherwise would trade correctness for a stopwatch.
What the fast path skips
- No generation of the tool name. The name comes from the leaf that won the ranking.
- No generation of the arguments. Values are transcribed and normalised from their sources.
- No full schema in context. The working set is twelve tools or fewer.
- No round trip to an external model. The decision is made inside the engine.
What it does not skip is validation. A fast answer that calls the wrong tool is worse than a slow one that calls the right tool, so the checks run on every path.
The budget, stage by stage
The target is a budget you can spend, not a single number to hit:
| Stage | What it does | Share of the budget |
|---|---|---|
| Normalise | Clean the message, split clauses | Small |
| Narrow | Walk the tree to a branch | Largest |
| Ground | Match values to arguments | Medium |
| Validate | Check required fields and types | Small |
| Serialise | Build the tool_calls response | Tiny |
When a request misses the target, the table tells you where it went. That is more useful than a single end-to-end percentile, because it points at the stage that needs work.
Why not just use a smaller model?
A small model is still a model. It still needs the tool definitions, still generates tokens, and still has a quality floor that drops as the list grows. You can get to a few hundred milliseconds that way, and a few hundred milliseconds is fine for a chat reply and slow for a booking button.
The other problem is determinism. A smaller model is still sampling. The same request can choose a different tool on a different day. The fast path is a lookup and a check, so the same request and the same version produce the same call.
When the engine slows down on purpose
The fast path is not allowed to guess. If the ranking is close, or a required value is missing, or a clause needs another tool's result first, the engine takes the slower route. That is the correct trade: the user waits a little longer instead of getting a confident wrong answer.
Measuring it honestly
Report the path, not just the average. A service that answers 90% of turns in 60 ms and 10% in 900 ms has a very different feel from one that answers everything in 150 ms, even though the averages are close.
Track at least:
- the share of turns resolved on the fast path;
- latency by path, not blended;
- the reasons requests fall back to reasoning.
The last one is the most actionable. A spike in fallbacks usually means the tree labels drifted, not that the engine got slower.
Frequently asked questions
Is 60 ms measured or targeted?
It is a design target for the fast path. Real numbers depend on region, tree size and request shape, and should be measured on your own workload.
Do reasoning turns cost more?
A turn costs one credit either way. The path affects latency, not price. See one credit per turn.