LLM tokens measured in C#: why AI can't count the R's in strawberry
The model's failure on the "strawberry" question isn't a counting problem: it never sees letters, only numeric pieces called tokens. Using Microsoft.ML.Tokenizers 2.0.1 and the gpt-5 tokenizer, the author measures this from C# and shows that the word alone is a single token with ID 101830, so the r's inside it stay out of reach. The piece also covers how token cost varies by language, by numbers, and by random text such as GUIDs, plus how to count tokens in .NET.
Key points
- " strawberry" is one token (ID 101830) in the gpt-5 tokenizer, so the letters are inaccessible inside it.
- Spelling the word letter by letter turns each r into its own token and makes counting possible.
- The same meaning costs roughly 55% more tokens in Romanian than in English.
- A 36-character GUID takes 33 tokens; logs, hashes and base64 fill the context window fast.
- Microsoft.ML.Tokenizers 2.0.1 offers CountTokens() and GetIndexByTokenCount() for counting and budget-aware truncation.
- Tokenizers differ per model: ToListAsync is one token on GPT-4 and three on GPT-4o, and Claude exposes /v1/messages/count_tokens.
The model's wrong answer to the strawberry question comes down to the fact that it never sees letters. Before a prompt reaches the model, a tokenizer splits the text and maps each piece to an ID from a fixed list; " strawberry" on its own is a single token numbered 101830. The letters sit inside that number and the model cannot open it up. Write the word letter by letter and the r's become separate tokens that can actually be counted.
The author measures everything with Microsoft.ML.Tokenizers 2.0.1 against the gpt-5 tokenizer (o200k_base). A tokenizer created once via `TiktokenTokenizer.CreateForModel("gpt-5")` exposes `CountTokens()` for counting and `GetIndexByTokenCount()` for cutting text to a budget. The same meaning costs about 55% more tokens in Romanian than in English, and random hex such as a GUID burns nearly one token per character. Since price, context window and rate limits are all token-based, counting characters instead is misleading.
Real-world use
- An LLM-powered log analysis tool measuring how quickly GUID- and hash-heavy lines fill the context window.
- A multilingual support assistant estimating the token cost of Turkish or Romanian answers versus English to price usage.
- Truncating a long file to a token budget with GetIndexByTokenCount() before it goes into the prompt.
- A .NET service calling an API with token-per-minute rate limits that needs to size requests up front.
Why it matters
Because price, context window and rate limits are all measured in tokens, counting characters or words produces misleading estimates; measuring with the right tokenizer directly affects prompt budgets and bills.