# SimpleTokenizerV3

This bonus notebook implements a whitespace-preserving variation of the chapter's `SimpleTokenizerV2`. In hindsight, this `SimpleTokenizerV3` here may be a more intuitive simple tokenizer for the main chapter. However, it is kept separate from the chapter notebook so the book's main code to avoid diverging from the actual book.

- [simple-tokenizer-v3.ipynb](simple-tokenizer-v3.ipynb): a simple tokenizer that preserves delimiter and whitespace round trips