11.7 Chapter Summary

In this chapter, we systematically learned the theory and practice of Agentic RL, from basic concepts to complete training pipeline, from data preparation to model deployment. Let's review the main content of this chapter.

(1) Essence of Agentic RL

Agentic RL treats LLM as a learnable policy, embedding it into the agent's perception-decision-execution loop, optimizing agent performance in multi-step tasks through reinforcement learning. Its core difference from traditional PBRFT (Preference-Based Reinforcement Fine-Tuning) lies in:

• Task Nature: From single-turn dialogue optimization to multi-step sequential decision-making
• State Space: From static prompts to dynamically evolving environment states
• Action Space: From pure text generation to text + tools + environment operations
• Reward Design: From single-step quality assessment to long-term cumulative returns
• Optimization Objective: From short-term response quality to long-term task success

(2) Six Core Capabilities

Agentic RL aims to enhance six core capabilities of agents:

1. Reasoning: Multi-step logical deduction, learning reasoning strategies
2. Tool Use: API/tool invocation, learning when and how to use
3. Memory: Long-term information retention, learning memory management
4. Planning: Action sequence planning, learning dynamic planning
5. Self-Improvement: Self-reflection optimization, learning from mistakes
6. Perception: Multimodal understanding, visual reasoning and tool use

(3) Training Pipeline

Complete Agentic RL training pipeline includes:

1. Pretraining: Learning language knowledge on large-scale text (usually using existing pretrained models)
2. Supervised Fine-Tuning (SFT): Learning task format and basic reasoning ability
3. Reinforcement Learning (RL): Optimizing reasoning strategies through trial and error, surpassing training data quality

Among these, SFT is the foundation, RL is the enhancement. Without SFT foundation, RL is difficult to succeed; without RL optimization, models can only imitate training data.

If you want to deeply learn Agentic RL, recommend following this path:

Foundation Stage

1. Reinforcement Learning Basics: Learn basic concepts like MDP, policy gradient, PPO
2. LLM Basics: Understand technologies like Transformer, pretraining, fine-tuning
3. Practice HelloAgents: Run example code from this chapter, understand complete pipeline

Advanced Stage

1. Deep Dive into TRL: Learn TRL library implementation, understand details of algorithms like SFT and GRPO
2. Custom Datasets: Train models using your own datasets
3. Custom Reward Functions: Design reward functions suitable for your tasks
4. Parameter Tuning: Systematically tune hyperparameters, improve model performance

Expert Stage

1. Multi-Step Reasoning: Research long-sequence reasoning tasks
2. Tool Learning: Enable agents to learn tool use
3. Multi-Agent: Research multi-agent collaboration
4. Cutting-Edge Papers: Read latest research papers, follow frontier progress

We hope this chapter helps you understand and master Agentic RL technology, apply this knowledge in your own projects, and build more intelligent Agent systems!