Glossary — Language models
Tokenization
Tokenization is the splitting of text into tokens — the first thing that happens to every message you send an AI.
2 tokensTokenization
Simply put#
Before a model can do anything with your question, it cuts it into pieces and replaces each piece with a number. The rules for this were learned during training: frequent letter sequences get their own token, rare ones get chopped finely.
Example#
"How are you?" becomes four tokens. The Estonian equivalent takes five or six. The difference comes from the training data containing vastly more English.
Why it matters to you#
It explains why AI sometimes "sees" words wrongly — why it cannot reliably count the letters in a word, for instance. It does not see letters. It sees tokens.