Tokens·5 of 37

Split Into Tokens

Optional, for readers who know a little Python. In [Text Into Pieces](text-into-pieces) you saw a tokenizer cut text into pieces from a fixed vocabulary. Here you write a small one. Real tokenizers use a cleverer method, but the simplest way that works is greedy longest match: start at the beginning of the text, take the longest piece from the vocabulary that the text starts with, then carry on from just after it. If no piece in the vocabulary fits, take a single character on its own, so that any text can always be cut up. For example, with the vocabulary st, straw, raw, berry, the word strawberry becomes straw|berry: at the start both st and straw fit, and straw is longer. ## Input The vocabulary comes first, one piece per line, then a line ---, then one text per line. In the vocabulary an underscore _ stands for a space, so _cat is the piece " cat" with its space in front, the way real tokenizers keep spaces. ``text The _cat _sat --- The cat sat ` ## Output The starter code already reads the input and prints, for each text, its pieces joined by | (spaces shown as _) and the number of tokens: `text The|_cat|_sat (3 tokens) ` Your job is the function tokenize(text, vocab), where vocab is a set of strings (with real spaces). Return the list of pieces. **Hints.** Keep a position i. At each step, try lengths from the longest possible down to 1 and check whether text[i:i + n] is in vocab`. Watch what happens with an empty vocabulary: every character is its own token.