Exercises

1. In natural language processing, language models have evolved from statistical to neural network models.

   • Please use the mini corpus provided in this chapter (datawhale agent learns, datawhale agent works) to calculate the probability of the sentence agent works under the Bigram model
   • The core assumption of N-gram models is the Markov assumption. Please explain the meaning of this assumption and what fundamental limitations N-gram models have?
   • How do neural network language models (RNN/LSTM) and Transformer overcome N-gram model limitations respectively? What are their respective advantages?

2. The Transformer architecture^[4] is the foundation of modern large language models. Among them:

   Hint: Can combine code implementation in Section 3.1.2 of this chapter to aid understanding

   • What is the core idea of the Self-Attention mechanism?
   • Why can Transformer process sequences in parallel while RNN must process serially? What role does Positional Encoding play?
   • What is the difference between Decoder-Only architecture and complete Encoder-Decoder architecture? Why do current mainstream large language models all adopt Decoder-Only architecture?

3. Text subword tokenization algorithms are a key technology for large language models, responsible for converting text into token sequences the model can process. Why can't we directly use "characters" or "words" as model input units? What problem does the BPE (Byte Pair Encoding) algorithm solve?

4. Section 3.2.3 of this chapter introduced how to deploy open-source large language models locally. Please complete the following practice and analysis:

   Hint: This is a hands-on practice question; actual operation is recommended

   • Following this chapter's guidance, deploy a lightweight open-source model locally (recommend Qwen3-0.6B), try adjusting sampling parameters and observe their impact on output
   • Choose a specific task (such as text classification, information extraction, code generation, etc.), design and compare different prompt strategies (such as Zero-shot, Few-shot, Chain-of-Thought) and their effect differences on output results
   • Compare closed-source models and open-source models from dimensions of performance, cost, controllability, privacy, etc.
   • If you want to build an enterprise-level customer service agent, which type of model would you choose? What factors need to be considered?

5. Model Hallucination^[11] is one of the key limitations of current large language models. This chapter introduced methods to mitigate hallucinations (such as retrieval-augmented generation, multi-step reasoning, external tool invocation)

   • Please choose one and explain its working principle and applicable scenarios
   • Research cutting-edge studies and papers—are there other methods to mitigate model hallucinations, and what improvements and advantages do they have?

6. Suppose you want to design a paper-assisted reading agent that can help researchers quickly read and understand academic papers, including: summarizing core content of paper research, answering questions about papers, extracting key information, comparing viewpoints of different papers, etc. Please answer:

   • Which model would you choose as the base model when designing the agent? What factors need to be considered when choosing?
   • How to design prompts to guide the model to better understand academic papers? Academic papers are usually very long and may exceed the model's context window limit—how would you solve this problem?
   • Academic research is rigorous, meaning we need to ensure information generated by the agent is accurate, objective, and faithful to the original text. What designs do you think should be added to the system to better achieve this requirement?