Semantic Routing
Score every prompt by difficulty and send the easy ones to the cheap model: classifier scores, thresholds, misroutes and the eval that sets them.
An interactive AI Infrastructure lesson: 19 steps, about 30 minutes, on a live simulation in your browser.
The shop's support bot takes everything from password resets to refund disputes. Behind the edge gateway stand two services: chat on Llama 3.1 8B at $0.50 a million tokens, and code on Llama 3.1 70B at $3.00. Six times the price, and the stronger reasoner.
Today every request is routed by model name and the support app asks for the big one, always. Watch both services start, then one password reset answered on 141 GB of weights. It works. It is also the most expensive possible way to answer it.
What you will learn
The dear model answers everything
- One door, two prices: Routing by model name alone sends every question to whatever the client asked for — including the dear model for cheap questions.
- A password reset on 70B: A request has a difficulty, and the model it needs should match it. Name routing cannot see difficulty.
Score first, route second
- A tiny judge at the door: Semantic routing is two steps: score the prompt, then route the score. The classifier is part of the request path, so its cost counts too.
- Two scores, two services: A scored request matches the first rule its score fits. The route table is a decision list over difficulty, not names.
- Where does 0.7 go?: Above every ceiling sits the default. Design the default as carefully as the rules — it catches everything you did not imagine.
- What the score is made of: Difficulty scores come from cheap observable signals. They drift with the product, so thresholds are maintained, not set once.
The threshold is the product
- Rush hour at 0.4: A threshold concentrates the common case on the cheap model. Its value is a product decision about which questions deserve the dear weights.
- Someone sets 0.9: Raising the threshold moves hard questions onto cheap weights. Every point of threshold is bought with someone's wrong answer.
- Drill: price the split: Blended spend = share × tokens × price, summed per band. The threshold sets the shares, so the threshold sets the bill.
- Read the meter: The spend counter is the shared truth between engineering and finance. Restore the threshold the evals support, then guard it.
When routing goes wrong
- Everything goes cheap: A misroute is invisible to the router: counters climb, spend falls, and only graded answers reveal it. Watch quality, not traffic.
- The scores stop arriving: Unscored requests skip gated rules and take the default. The default is your failure mode — make it correct first, cheap second.
- Scores resume, ceiling back: Recovery is the threshold back where the evals put it, with scores flowing again. Nothing else moves.
Evals set the threshold
- Grade both models by band: Evals grade models per difficulty band. The threshold sits where the cheap model's accuracy falls below what the product tolerates.
- Drill: read the ceiling: The ceiling sits under the first band the cheap model fails. Tolerance first, threshold second.
- The judge sends a bill too: The classifier is request-path cost. Its budget is a fraction of the price gap it exploits — measure both in the same dollars.
- Budget the misroutes: Route unknown traffic dear and promote on evidence. A misroute budget turns routing quality into an SLO like any other.
Recap & playground
- Cheat sheet
- Playground