Routing as a constrained decision, not a model ranking

The cheapest model call is not necessarily the cheapest completed task. An agent may make several calls, retry tools, request human review, or produce an answer that someone must correct. Begin with a unit such as accepted report, resolved support case, or verified patch. Account for all work needed to reach that outcome. Keep failed tasks in the denominator when calculating cost per attempted task, and report cost per successful task separately.

Use hypothetical prices for reasoning exercises and a dated rate table for real accounting. If 100 tasks cost 12 currency units in total and 80 meet acceptance criteria, cost per attempted task is 0.12 and cost per accepted task is 0.15. Reporting only the first value conceals the economic effect of failures. Neither figure includes human remediation unless that cost was explicitly recorded.

A router chooses which execution path receives a request. It may use task type, input length, required tools, measured difficulty, or a learned score. RouteLLM is one research example of learning routing decisions from preference data; its reported results do not establish the right threshold for your workload. Evaluate the router on the actual task distribution and model versions you intend to use.

Suppose a small route costs one cent and a larger route costs five cents. An unacceptable result carries an estimated remediation cost of 50 cents. If the larger route increases success probability by 0.20, the expected avoided remediation cost is ten cents, which exceeds the four-cent premium. For a 0.02 improvement, the benefit is one cent. These are decision-model assumptions, not measured facts, and the cost estimate should reflect the actual consequence of failure.

Another design tries a smaller route first and escalates selected results. If the small route costs 0.002 units, the larger route costs 0.020, and 30 percent of requests escalate, expected model cost is 0.002 plus 0.30 times 0.020, or 0.008 units per request. Add the gate's own cost, repeated context, tool calls, and retries before claiming savings.

The gate is part of the system being evaluated. A weak verifier can accept confident mistakes, while an overcautious verifier sends nearly everything to the expensive path. Cascading also increases latency for escalated tasks because the attempts are sequential. A direct large-model route may be preferable for cases that are both difficult and time-sensitive. Route by evidence about the task and acceptance conditions, not by the model's unsupported claim that its answer is certainly correct.

A practical routing objective can minimize expected cost subject to minimum task success, a latency percentile target, and an upper bound on consequential errors. A single weighted score is convenient, but it can obscure an unacceptable tradeoff. If the system must not send unauthorized messages, a cheaper average cannot compensate for violating that condition. Report constraint failures directly.

Tune thresholds on development data and evaluate once on held-out data before rollout. If 90 percent of your requests are easy, an overall score can hide very poor performance on the remaining tenth. Segment by task family, language, input length, and tool dependency where those distinctions matter. Retain a small, authorized exploration sample to detect routing drift, or periodically re-evaluate alternatives offline. Otherwise a router can stop collecting evidence about paths it rarely selects.

Track which route was selected, why, what it cost, whether it escalated, and whether the final task passed an external acceptance test. The route identifier should include model, prompt, tool configuration, and verifier versions. A model change can alter both quality and token use even when the router code remains identical.

The accompanying arithmetic chooses between two hypothetical routes using estimated success improvement and remediation cost. It is a decision simulator, not a trained routing model or a benchmark result. Its value is making assumptions visible enough to challenge. Change the failure cost and observe which cases switch routes. Then ask whether that economic preference respects the application's hard constraints. A good routing report explains where savings came from, where quality changed, and which task groups still need more evidence.

Costs in the routing comparison are hypothetical cents, and gain is an assumed absolute improvement in success probability. The cascade calculation uses separate hypothetical currency units. Equality chooses the cheaper route. No model is called, and none of the figures is a provider price or benchmark result.