Arabic Token-Tax Index
How many more tokens does the same content take in Arabic than in English? Token counts for LLM tokenisers on parallel Arabic–English text, with 95% confidence intervals, and what a page costs at list prices.
The Arabic text has not yet been reviewed by a native Arabic writer; the English governs.
Arabic/English token ratio by tokeniser
Bar: Arabic tokens divided by English tokens for the same content. Whisker: 95% bootstrap interval. Vertical line at 1: the same number of tokens as English. Lower means Arabic costs less extra.
Leaderboard
Select a column heading to sort. Fewer tokens means lower cost and more text per context window; it is not a measure of model quality. On a narrow screen some columns are hidden; all of them are in the dataset.
Cost per page at list prices
Price not shown
Not measured in this version
Method and dates
Text sources and licences
Cite as
Corrections and requests for new tokenisers: hello@bayanatlabs.com.