Claude Sonnet 5: Anthropic's Clever Tokenizer Trick Masks a Price Hike

Anthropic just released Claude Sonnet 5, claiming Opus-level performance at lower prices. But a closer look reveals a clever sleight of hand that could leave users paying more, not less.

The headline pricing remains unchanged: $3 per million input tokens,$15 per million output tokens, with a temporary discount to $2/$10 until August 31st. So where's the catch?

It's in the tokenizer. Sonnet 5 introduces a new one that, according to Anthropic, produces "approximately 30% more tokens than on Claude Sonnet 4.6" for the same input text. This isn't just a technical footnote; it's an effective price hike hiding in plain sight.

Let's look at the real-world impact:

The effect varies dramatically by language and content type. English users face the steepest increase at 42%. Spanish and Python code aren't far behind at 33% and 27%. Chinese, interestingly, is barely affected at 1%.

This tokenizer change is a masterclass in pricing strategy. Anthropic can truthfully claim they haven't raised prices while effectively increasing revenue per word processed. It's AI's version of shrinkflation: same price, less product.

Beyond the pricing sleight of hand, Sonnet 5 does bring some genuine improvements:

1. A massive 1 million token context window (with 128,000 token max output)
2. 'Adaptive thinking' enabled by default
3. Same tool set and platform features as Sonnet 4.6

Notably, Anthropic has removed fine-grained sampling controls (temperature, top_p, top_k). This simplification might frustrate power users who relied on these parameters.

Anthropic's system card offers a glimpse into their regulatory strategy:

Sonnet 5 is significantly less capable at cyber tasks than Mythos 5: its safeguards are thus similar to those we apply to Opus 4.7 and Opus 4.8 (models that are more capable than Sonnet 5 but much less capable than Mythos 5).

By positioning Sonnet 5 as powerful but constrained, Anthropic likely aims to sidestep the heightened scrutiny more capable systems might face.

So, is Sonnet 5 worth it? For English users, you're looking at up to a 42% effective price increase for Opus-level performance. That might still beat Opus pricing, but it's far from the straightforward upgrade Anthropic implies. For Chinese users, the value proposition is much clearer given the minimal tokenizer impact.

Ultimately, Sonnet 5 is a clever move by Anthropic. They've boosted capabilities, potentially increased revenue, and still positioned it as price-competitive. But users should look beyond the headline claims and do the math for their specific use cases. In the AI world, the devil is always in the details, or in this case, the tokenizer.