Chapter 2: Working with Text Data

• compare-bpe-tiktoken.ipynb benchmarks various byte pair encoding implementations
• bpe_openai_gpt2.py is the original bytepair encoder code used by OpenAI