What is a token? The most important idea in AI, explained plainly

If you read about one concept in AI, read about this one. Almost everything that seems strange about language models at first is explained here.

A machine does not see words#

A language model does not work with letters or words. It works with numbers. To turn your sentence into numbers, it first cuts it into pieces — and those pieces are tokens.

One token is not one word. A common, short word usually fits in a single token. A long or rare word breaks into several. Each piece gets its own number.

The word "token" is itself one token#

The word token encodes to exactly one token — number 10346. The Estonian word tehisintellekt takes five. That is where this site's name comes from: the smallest unit a machine works with, and simultaneously the most useful thing you can learn about AI.

Why it is worth knowing#

Tokens measure two things.

Price. API bills are counted in tokens, for both input and output. If you use AI in a company, the token count is your cost forecast.

Memory. The context window — how much of a conversation the model holds at once — is also measured in tokens. When a conversation runs long, the beginning slides out of the window and the AI "forgets".

Why AI cannot count letters#

The classic example: ask a model how many rs there are in "strawberry" and it gets it wrong surprisingly often. The reason is exactly this — it does not see letters. It sees tokens, whole chunks. Counting letters is, for it, roughly what counting the pixels in a letter would be for you.

What to do with this#

Three practical conclusions:

1. In a long conversation, restate the important things — they may have slid out of the window.
2. If you pay per API call, a shorter prompt is a cheaper prompt. But not always a better one: a prompt too short produces an unusable answer you will have to ask again anyway.
3. Text written in Estonian burns more tokens than the same content in English. If cost matters and language does not, that is worth knowing.

Try it yourself: type something into the tokenizer and watch how many pieces your sentence becomes.