GLOSSARY
Tokenizer
The component that chops text into tokens — the small units a model actually reads, writes and bills by.
Models do not see words or characters — they see tokens, numeric IDs produced by a tokenizer. An English word is often one token, a common word fragment a few; Chinese characters are frequently one token each, while rare words break into odd fragments. This is why the same prompt costs a different number of tokens in different languages.
Token counts define API pricing, context-window limits and generation speed, so understanding tokenization explains many practical surprises: why JSON output is expensive, why code burns tokens faster than prose, and why a 128K-context model still loses the thread of very long conversations.