AI tokenizer tax: non-English text costs up to 5x more

Same answer, different price. Research reported today by Rest of World puts numbers on the AI tokenizer tax: among the 20 most-used languages online, the most expensive took 1.5 to 5 times more tokens than the cheapest to say the same thing. Every major model bills per token. So a support bot answering in one language can cost several times more than the same bot in another, with no change in quality.

What actually happened

The analysis comes from the Coalition for Language Equity in AI Research (CLEAR), which published it in September as "The Tokenizer Tax." Rest of World's report today covers the findings:

• Top 20 languages. The most token-hungry language used 1.5x to 5x the tokens of the cheapest.
• 200 languages. Across an expanded set, the gap in one model grew to 17x.
• Script matters. Latin-script languages do best. Scripts such as Burmese, Tibetan, Khmer, Lao and Ol Chiki were the most penalized.
• Nearly every model. The gap showed up across nearly all models in the analysis.

What the report does not give: a per-language table, the model list, or the test method. We could not reach CLEAR's own page to check them. So treat the range as directional, not as a quote for your language.

Why it matters for your business

We are a Phoenix studio. A lot of the businesses we work with answer customers in English and Spanish, and some in Vietnamese, Arabic or Chinese. Almost nobody budgets those languages separately. They take the English pilot cost, multiply by volume, and call it the forecast.

That forecast is wrong by whatever your language ratio is. You do not need a study to find it. You need an afternoon:

1. Take 50 real conversations per language. Pull them from your support inbox or chat logs.
2. Count tokens with your model's own tokenizer. OpenAI, Anthropic and Google each expose token counting. Run the same content through each model you are considering.
3. Price per resolved conversation, per language. That is the number that goes in the budget.
4. Then route. If one model is much cheaper for a language at the same quality, send that language there. A gateway makes this a routing rule, not a rebuild.

Token prices keep falling, which shrinks the gap in dollars. The ratio stays until the labs change their tokenizers.

Sources: Rest of World: the AI tokenizer tax, CLEAR: The Tokenizer Tax.