Glossary — Fundamentals
Token
A token is a piece of text — usually a word, part of a word, or a punctuation mark — that a language model turns into a number so it can compute with it.
Simply put#
A machine does not see words. It sees numbers. To turn text into numbers, it first cuts the text into pieces — and those pieces are tokens.
One token is not one word. A short, common word often fits into a single token; a long or rare one breaks into several.
Example#
The word "token" is itself exactly one token, number 10346. The Estonian word "tehisintellekt" takes five. That is precisely why Estonian is more expensive for a machine: the same meaning costs more tokens.
Why it matters to you#
Tokens measure two things: price and memory. API bills are counted in tokens, and how much of a conversation a model can hold at once is also measured in tokens.