Strawberry in pieces

About this video

For a long time a famous way to trip up an AI model was to ask how many r's there are in the word strawberry. Models that could write essays often said two. The mistake shows how every language model reads: not letter by letter, and not word by word, but in tokens.

A token is a piece of text: a common word, part of a word, a punctuation mark or a chunk of a number. Before the model reads anything, a program called a tokenizer cuts the text into tokens and turns each one into a number. The model reads the numbers and writes numbers back, which the tokenizer turns into text again. Strawberry arrives as three pieces, st, raw and berry, so the model never sees its letters.

Why not letters? A model does one step of work per piece, and a sentence of seventy letters would take seventy steps. Why not whole words? There are millions of them, in every language, with new ones every day. Tokens sit in between: a fixed vocabulary of about 100,000 to 200,000 pieces, chosen because they come up often. Common words are one token; rare words such as unbelievably split into a few pieces, and long numbers split into chunks.

For English, a token is about four characters, or three quarters of a word, so 1,500 words is about 2,000 tokens. Other languages often need more: the same short paragraph took 40 tokens in English and 62 in Persian with one widely used tokenizer, and 127 with an older one. Speed, limits and price are all counted in tokens, which is why an agent's work makes sense once you count in tokens instead of messages.

Transcript

1. 0:00How many r's are in strawberry? Older models often said two. Why?
2. 0:08The model never sees the letters. It sees three pieces, and each piece is just a number.
3. 0:20Why not letters? That would be seventy steps for one sentence. Whole words? There are millions, and always a new one.
4. 0:42A tokenizer cuts the text into tokens, pieces from a fixed vocabulary, and turns each one into a number. The model reads numbers and writes numbers back.
5. 1:06Common words are one token. Rare words and long numbers split into a few common pieces.
6. 1:28In English a token is about four characters, three quarters of a word. Fifteen hundred words is about two thousand tokens.
7. 1:50Other languages often need more. The same paragraph took forty tokens in English and sixty-two in Persian, or a hundred and twenty-seven with an older tokenizer.
8. 2:14Speed, limits and price are all counted in tokens. So count in tokens, not in messages.
9. 2:32A model doesn't read letters or words. It reads tokens.