Token

The subword chunk a model actually reads and writes; text is split into these before the model sees it, and context limits and pricing are counted in them.

Tokens are the pieces a language model actually reads and writes. The model never sees letters or whole words the way you do; before any text reaches it, a tokenizer chops that text into chunks drawn from a fixed vocabulary (each chunk is a token) and hands the model the ID number of each one. Every input and every output happens in this unit.

The chunks are usually subwords, not words. A common word like “cat” is a single token; a rarer one like “unbelievable” might split into three (“un”, “believ”, “able”); a typo, an emoji, or a snippet of code breaks into whatever fragments the vocabulary already has. This is a deliberate compromise: keep the vocabulary small enough to be workable (typically 30k-200k tokens) while still being able to spell out any word, name, or garbage string by combining pieces. A rough rule of thumb for English is that one token runs about four characters, or roughly three-quarters of a word.

Tokenizers read postpostmodern as pieces, not one word.

Token3323izers12230·read1349·post1736post2203modern49789·as438·pieces9666,11·not537·one825·word3409.13

Thirteen tokens, split by a local Qwen tokenizer. The word Tokenizers alone breaks in two and postpostmodern into three; the comma and period are tokens of their own; and the faint dot marks the leading space that rides along inside a token. The number under each piece is the ID the model actually receives: to it, this sentence is the list [3323, 12230, 1349, ...], and the words are only here for us.

Tokens are the through-line of the whole pipeline. Each token is mapped to an embeddings vector on the way in; attention is the mechanism by which tokens weigh each other; the transformer chews on the whole sequence of them at once; and generation is just predicting the next token over and over, with temperature deciding how boldly the model gambles among the likely candidates. It is also why the practical limits you hear quoted are counted in tokens rather than words: a model’s context window (how much it can hold in view at once) and, for hosted models, the bill are both measured per token.