The Hidden Complexity of AI Model Routing

Model routing in AI seems simple: pick the best model for each task. But our experience at Hugging Face reveals it's a deceptively complex challenge that goes far beyond model selection. Here's why routing is harder than you think, and how we're tackling it.

Cost: Caching Changes Everything

Intuition says cheaper models cost less. Reality disagrees. In our tests across 417 tasks, Claude Sonnet 4.6 cost $79 total while GPT-4.1 cost$155, nearly double, despite GPT-4.1's lower token pricing.

The culprit? Caching. Agent workloads reuse large chunks of context. Sonnet's lower cache-read pricing gave it a massive advantage, overcoming both its higher base price and longer reasoning chains.

The lesson: actual cost depends on the interplay of model, workload, and infrastructure. A router that only looks at pricing sheets is solving the wrong problem.

Complexity: More Than Task Difficulty

Many routers try to match task difficulty to model strength. This fails in two ways:

1. Difficulty is often invisible upfront. "Summarize this contract" looks simple but might trigger retrieval, compliance checks, and multiple refinements.
2. Difficulty is just one factor. Production systems juggle cost, latency, specialization, reliability, compliance, and data residency simultaneously.

Routers aren't solving one problem. They're constantly balancing competing priorities in real-time.

Latency: It's the System, Not Just the Model

Bigger models are slower, but user-experienced latency depends on much more:

• Routing itself adds overhead
• Infrastructure (hardware, cache warmth, endpoint load) often dominates response times
• Routing granularity (per-task vs. per-step) impacts both latency and complexity

A router that ignores the serving system is optimizing against the wrong reality.

Our Solution: Optimization, Not Classification

We stopped asking "which model is best?" and started optimizing across cost, quality, and latency simultaneously. The result:

Each blue point is a different router configuration. The key isn't any single point, it's the range of options. Config1 (latency-optimized) hits 84% accuracy for $93, cutting cost 21% and latency 9% vs. Opus alone, with only a 4% accuracy drop. Config2 pushes cost even lower.

Notice the difficulty-based router (teal) lands at similar accuracy but higher cost. It can't explore the full tradeoff space like our optimization approach.

Crucially, our optimization is lightweight (6ms, 2kB per task), avoiding becoming a new bottleneck.

The Bigger Picture

Effective routing isn't about finding the "best" model. It's about finding the best operating point for the entire system, models, caching, infrastructure, compliance, and workload patterns included.

When routing works well, it's rarely because it found the perfect model. It's because it found the sweet spot for the whole system. That's a harder problem than classification, but it's the one worth solving.

If you're building routing into your own AI systems, we'd love to hear about the tradeoffs you're encountering. The landscape is complex, but by sharing our experiences, we can build more efficient and effective AI infrastructure together.