Tokens and tokenisers: the units your model actually reads (and bills)

Every limit and every invoice in the LLM world is denominated in tokens, not words. Context windows are measured in tokens, pricing is per token, and rate limits are tokens per minute. So it's worth a few minutes to understand what a token is and where it comes from.

What a token is

A token is a chunk of text — often a whole short word, sometimes a word-piece, sometimes a single character or a space. A tokeniser is the component that splits your text into these chunks and maps each to an integer id, because the model operates on numbers, not letters. A rough rule of thumb for English: one token ≈ 4 characters, or about three-quarters of a word.

Why this matters in practice

• Cost — you pay per input and output token, so prompt length and verbosity are line items, not just style.
• Limits — the context window is a token budget; a '128K context' is 128,000 tokens, not characters or words.
• Language equity — tokenisers are trained mostly on English, so the same sentence in another language (or lots of code, or emoji) can cost noticeably more tokens.

Count before you're surprised

Because the mapping isn't one-to-one with words, guessing is unreliable. Every provider ships a tokeniser you can run locally to count exactly — do it when you're budgeting a prompt or checking whether a document will fit.

import tiktoken

enc = tiktoken.get_encoding('o200k_base')
tokens = enc.encode('Tokenisers are sneakily important.')
print(len(tokens))     # exact token count
print(tokens[:5])       # the integer ids the model sees

You think in words; the model thinks in tokens; your bill is in tokens. Learn to count in the currency you're charged in.