LLMs from Scratch

This repository contains the code for developing, pretraining, and finetuning a GPT-like LLM and is the official code repository for the book Build a Large Language Model (From Scratch).

课时

• README.md
• Troubleshooting Guide
• list of exercise solutions
• Bonus Material
  • Python Setup Tips
  • Installing Python Packages and Libraries Used in This Book
  • Docker Environment Setup Guide
  • Comparing Various Byte Pair Encoding (BPE) Implementations
  • Understanding the Difference Between Embedding Layers and Linear Layers
  • Dataloader Intuition With Simple Numbers
  • BPE From Scratch
  • SimpleTokenizerV3 variant
  • KV Cache
  • Attention Alternatives
  • Grouped-Query Attention
  • Multi-Head Latent Attention
  • Sliding Window Attention
  • Gated DeltaNet
  • DeepSeek Sparse Attention (DSA)
  • Cross-Layer KV Sharing
  • Mixture-of-Experts (MoE)
  • Alternative Weight Loading Methods
  • Pretraining GPT on the Project Gutenberg Dataset
  • Adding Bells and Whistles to the Training Loop
  • Optimizing Hyperparameters for Pretraining
  • Building a User Interface to Interact With the Pretrained LLM
  • Converting GPT to Llama
  • PyTorch Performance Tips for Faster LLM Training
  • LLM Architectures
  • Qwen3 Dense and Mixture-of-Experts (MoE) From Scratch
  • Gemma 3 From Scratch
  • Olmo 3 From Scratch
  • Tiny Aya From Scratch
  • Qwen3.5 From Scratch
  • Gemma 4 E2B and E4B From Scratch
  • Chapter 5 with other LLMs as Drop-In Replacement (e.g., Llama 3, Qwen 3)
  • Additional Experiments Finetuning Different Layers and Using Larger Models
  • Finetuning Different Models on the 50k IMDb Movie Review Dataset
  • Building a User Interface to Interact With the GPT-based Spam Classifier
  • Dataset Utilities for Finding Near Duplicates and Creating Passive Voice Entries
  • Evaluating Instruction Responses Using the OpenAI API and Ollama
  • Building a User Interface to Interact With the Instruction-Finetuned GPT Model
• Appendix A: Introduction to PyTorch
  • Appendix A: Introduction to PyTorch
  • LLMs from Scratch
• Appendix D: Adding Bells and Whistles to the Training Loop
• Appendix E: Parameter-efficient Finetuning with LoRA
• Chapter 1: Understanding Large Language Models
  • Recommendations for Getting the Most Out of a Technical Book
• Chapter 2: Working with Text Data
  • Chapter 2: Working with Text Data
• Chapter 3: Coding Attention Mechanisms
  • Chapter 3: Coding Attention Mechanisms
  • More Efficient Multi-Head Attention Implementations
  • Understanding PyTorch Buffers
• ch04
  • Chapter 4: Implementing a GPT Model from Scratch To Generate Text
  • Chapter 4: Implementing a GPT Model from Scratch To Generate Text
• ch05
  • Chapter 5: Pretraining on Unlabeled Data
  • Memory-efficient Model Weight Loading
  • Extending the Tiktoken BPE Tokenizer with New Tokens
  • 11_qwen3
    • Qwen3 From Scratch with Chat Interface
  • Muon Optimizer
• Chapter 6: Finetuning for Classification
  • Chapter 6: Finetuning for Classification
• Chapter 7: Finetuning to Follow Instructions
  • Chapter 7: Finetuning to Follow Instructions
  • Chapter 7: Finetuning to Follow Instructions
  • Generating Datasets for Instruction Finetuning
• pkg
  • llms-from-scratch PyPI Package
• setup
  • 01_optional-python-setup-preferences
    • Native pixi Python and package management
    • Native uv Python and package management
  • AWS CloudFormation Template: Jupyter Notebook with LLMs-from-scratch Repo

开始学习 → Appendix A: Introduction to PyTorch