11.1 From LLM Training to Agentic RL

In previous chapters, we implemented various agent paradigms and communication protocols. However, when agents handle more complex tasks, they perform poorly, naturally raising questions: How can we make agents have stronger reasoning capabilities? How can we make agents learn to use tools better? How can we make agents capable of self-improvement?

This is precisely the core problem that Agentic RL (agent training based on reinforcement learning) aims to solve. This chapter will introduce reinforcement learning training capabilities to the HelloAgents framework, enabling you to train agents with advanced capabilities such as reasoning and tool use. We will start from the basics of LLM training and gradually delve into practical techniques such as Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO), ultimately building a complete agent training pipeline.

11.1.1 From Reinforcement Learning to Agentic RL

In Section 2.4.2 of Chapter 2, we introduced agents based on reinforcement learning. Reinforcement Learning (RL) is a learning paradigm focused on solving sequential decision-making problems. It learns how to maximize long-term rewards through direct interaction between agents and the environment, learning through "trial and error".

Now, let's apply this framework to LLM agents. Consider a mathematical problem-solving agent that needs to answer questions like this:

Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast
every morning and bakes muffins for her friends every day with four.
She sells the remainder at the farmers' market daily for $2 per fresh
duck egg. How much in dollars does she make every day at the farmers' market?

This problem requires multi-step reasoning: first calculate the number of eggs Janet has left each day (16 - 3 - 4 = 9), then calculate her income (9 × 2 = 18). We can map this task to the reinforcement learning framework:

• Agent: LLM-based reasoning system
• Environment: Mathematical problems and verification system
• State: Current problem description and existing reasoning steps
• Action: Generate next reasoning step or final answer
• Reward: Whether the answer is correct (correct +1, incorrect 0)

Traditional supervised learning methods have three core limitations: first, data quality completely determines training quality, and models can only imitate training data, making it difficult to surpass; second, lack of exploration ability, only passively learning paths provided by humans; third, difficulty optimizing long-term goals, unable to precisely optimize intermediate processes of multi-step reasoning.

Reinforcement learning provides new possibilities. By allowing agents to autonomously generate multiple candidate answers and receive rewards based on correctness, they can learn which reasoning paths are better, which steps are critical, and even discover better problem-solving methods than human annotations^[8]. This is the core idea of Agentic RL: treating LLM as a learnable policy, embedding it in the agent's perception-decision-execution loop, and optimizing multi-step task performance through reinforcement learning.

11.1.2 LLM Training Landscape

Before diving into Agentic RL, we need to first understand the complete process of LLM training. The birth of a powerful LLM (such as GPT, Claude, Qwen) typically goes through two main stages: Pretraining and Post-training. As shown in Figure 11.1, these two stages constitute the complete evolutionary path of LLM from "language model" to "conversational assistant".

Figure 11.1 LLM Training Landscape

Pretraining Stage is the first stage of LLM training, with the goal of making the model learn basic language patterns and world knowledge. This stage uses massive amounts of text data (usually TB-level) and trains the model through self-supervised learning, where the training signal is constructed from the text itself, such as predicting the next word from the previous context. The most common pretraining task is Causal Language Modeling, also known as Next Token Prediction.

Given a text sequence $x_1, x_2, ..., x_t$, the model needs to predict the next word $x_{t+1}$:

$$ \mathcal{L}{\text{pretrain}} = -\sum^{T} \log P(x_t | x_1, x_2, ..., x_{t-1}; \theta) $$

Where $\theta$ is the model parameters, $P(x_t | x_1, ..., x_{t-1}; \theta)$ is the probability distribution of the next word predicted by the model, and the goal is to minimize negative log-likelihood, i.e., maximize the probability of predicting the correct word. For example, given the text "The cat sat on the", the model needs to predict that the next word is most likely "mat". Through training on massive amounts of text, the model gradually learns grammar rules (what word sequences are legal), semantic knowledge (relationships between words), world knowledge (factual information about the world), and basic reasoning abilities.

The characteristics of the pretraining stage are: massive data volume, high computational cost, learning general language understanding and generation capabilities, and using self-supervised objectives constructed from unlabeled text rather than manually labeled task data.

Post-training Stage aims to address the shortcomings of pretrained models. Although pretrained models have powerful language capabilities, they are just "next word prediction" models and don't know how to follow human instructions, generate helpful, harmless, and honest answers, refuse inappropriate requests, and interact with humans in a conversational manner. The post-training stage aims to solve these problems and align the model with human preferences and values.

Post-training typically includes three steps. The first step is Supervised Fine-Tuning (SFT)^[15], with the goal of making the model learn to follow instructions and dialogue formats. Training data consists of (prompt, completion) pairs, and the training objective is similar to pretraining, still maximizing the probability of correct output:

$$ \mathcal{L}{\text{SFT}} = -\sum^{N} \log P(y_i | x_i; \theta) $$

Where $x_i$ is the input prompt, $y_i$ is the expected output, and $N$ is the number of training samples. SFT characteristics are: smaller data volume, requires manual annotation, quick results, mainly learning task formats and basic capabilities.

The second step is Reward Modeling (RM). Although SFT models can follow instructions, the quality of generated answers varies. We need a way to evaluate answer quality, which is the role of the reward model^[13,14]. Reward model training data consists of preference comparison data, containing two answers to the same question, one better (chosen) and one worse (rejected). The reward model training objective is to learn human preferences:

$$ \mathcal{L}{\text{RM}} = -\mathbb{E} [\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))] $$

Where $r_\phi(x, y)$ is the reward model, input is (prompt, answer) pair, output is quality score; $y_w$ is the better answer (chosen), $y_l$ is the worse answer (rejected), $\sigma$ is the sigmoid function, and the goal is to make the reward model give higher scores to better answers.

The third step is Reinforcement Learning Fine-tuning. With the reward model, we can use reinforcement learning to optimize the language model to generate higher quality answers. The most classic algorithm is PPO (Proximal Policy Optimization)^[1], with the training objective:

$$ J_{\text{PPO}} = \mathbb{E}{x, y \sim \pi\theta} [r_\phi(x, y)] - \beta \cdot D_{KL}(\pi_\theta || \pi_{\text{ref}}) $$

Where $\pi_\theta$ is the current policy, i.e., the language model, $\pi_{\text{ref}}$ is the reference policy, which in this scenario can be the SFT model, $r_\phi(x, y)$ is the reward model score, $D_{KL}$ is KL divergence, aimed at preventing the model from deviating too far, and $\beta$ is the balance coefficient. The meaning of this objective function is: maximize reward while not deviating too far from the original model.

Traditional RLHF (Reinforcement Learning from Human Feedback)^[5] requires a large amount of manual preference data annotation, which is costly. To reduce costs, researchers proposed RLAIF (Reinforcement Learning from AI Feedback)^[7], using powerful AI models (such as GPT-4) to replace human annotators. The RLAIF workflow is: use SFT model to generate multiple candidate answers, use powerful AI model to score and rank answers, use AI scores to train reward model, use reward model for reinforcement learning. Experiments show that RLAIF's effectiveness is close to or even exceeds RLHF, while costs are significantly reduced^[11].

11.1.3 Core Philosophy of Agentic RL

After understanding the basic training process of LLM, let's look at the difference between Agentic RL and traditional training methods. Traditional post-training (which we call PBRFT: Preference-Based Reinforcement Fine-Tuning) mainly focuses on optimizing single-turn dialogue quality: given a user question, the model generates an answer, then receives a reward based on answer quality. This approach is suitable for optimizing conversational assistants, but for agent tasks requiring multi-step reasoning, tool use, and long-term planning, it falls short.

Agentic RL is a new paradigm that treats LLM as a learnable policy embedded in a sequential decision-making loop. In this framework, agents need to interact with the external world in dynamic environments, execute multi-step actions to complete complex tasks, obtain intermediate feedback to guide subsequent decisions, and optimize long-term cumulative rewards rather than single-step rewards.

Let's understand this difference through a specific example. In the PBRFT scenario, a user asks "Please explain what reinforcement learning is", the model generates a complete answer, then scores directly based on answer quality. In the Agentic RL scenario, a user requests "Help me analyze the code quality of this GitHub repository", the agent needs to go through multiple steps: first call GitHub API to get repository information, successfully obtain repository structure and file list, get +0.1 reward; then read main code files, successfully obtain code content, get +0.1 reward; then analyze code quality reasonably, get +0.2 reward; finally generate analysis report with high quality, get +0.6 reward. Total reward is the accumulation of all steps: 1.0.

As can be seen, key features of Agentic RL are multi-step interaction, each action changes environment state, each step can receive feedback, and optimizing overall task completion quality.

Reinforcement learning is commonly formalized with the Markov Decision Process (MDP) framework. An MDP is defined by a five-tuple $(S, A, P, R, \gamma)$: state space $S$, action space $A$, state transition function $P(s'|s,a)$, reward function $R(s,a)$, and discount factor $\gamma$. The table below compares PBRFT and Agentic RL from the MDP perspective.

Table 11.1 Comparison of PBRFT and Agentic RL

In terms of state, PBRFT's state $s_0$ consists only of user prompts, time span $T=1$ (single step), state doesn't change, can be represented as $s_0 = \text{prompt}$. While Agentic RL's state $s_t$ contains historical observations and context, time span $T \gg 1$ (multi-step), state evolves with actions, can be represented as $s_t = (\text{prompt}, o_1, o_2, ..., o_t)$, where $o_t$ is the observation at step $t$ (such as tool return results, environment feedback, etc.).

In terms of action, PBRFT's action space only has text generation, single action type, represented as $a = y \sim \pi_\theta(y|s_0)$. While Agentic RL's action space includes text generation, tool invocation, environment operations, and other types, represented as $a_t \in {a_t^{\text{text}}, a_t^{\text{tool}}}$, for example $a_t^{\text{text}}$ is generating thinking process or answer, $a_t^{\text{tool}}$ is calling calculator, search engine, and other tools.

In terms of transition function, PBRFT has no state transition, represented as $P(s'|s,a) = \delta(s' - s_{\text{terminal}})$. While Agentic RL's state changes dynamically based on actions and environment, represented as $s_{t+1} \sim P(s_{t+1}|s_t, a_t)$, for example after calling search tool, state will include search results.

In terms of reward, PBRFT only has single-step reward $r(s_0, a)$, only given at task end, represented as $R_{\text{PBRFT}} = r(s_0, y)$, usually given by reward model: $r(s_0, y) = r_\phi(s_0, y)$. While Agentic RL has multi-step rewards $r(s_t, a_t)$, can give partial rewards at intermediate steps, represented as:

$$ R_{\text{Agentic}} = \sum_{t=0}^{T} \gamma^t r(s_t, a_t) $$

Where $\gamma \in [0,1]$ is the discount factor, $r(s_t, a_t)$ can be sparse reward (only given at task completion, such as correct answer +1), dense reward (given at each step, such as successful tool call +0.1), or a combination of both.

In terms of objective function, PBRFT maximizes single-step expected reward:

$$ J_{\text{PBRFT}}(\theta) = \mathbb{E}{s_0, y \sim \pi\theta} [r(s_0, y)] $$

While Agentic RL maximizes cumulative discounted reward:

$$ J_{\text{Agentic}}(\theta) = \mathbb{E}{\tau \sim \pi\theta} \left[\sum_{t=0}^{T} \gamma^t r(s_t, a_t)\right] $$

Where $\tau = (s_0, a_0, s_1, a_1, ..., s_T)$ is the complete trajectory.

This transformation is not just a difference in technical details, but a fundamental shift in thinking. PBRFT thinking focuses on "how to make the model generate better single answers", optimizing answer quality, focusing on language expression, making single-step decisions. While Agentic RL thinking focuses on "how to make agents complete complex tasks", optimizing task completion, focusing on action strategies, making multi-step planning. This transformation enables LLM to evolve from "conversational assistant" to "autonomous agent", capable of actively seeking information, knowing when and how to use external tools, willing to execute seemingly "detour" intermediate steps for the ultimate goal, and learning from mistakes.

Agentic RL aims to endow LLM agents with six core capabilities, as shown in Figure 11.2.

Figure 11.2 Six Core Capabilities of Agentic RL

Reasoning refers to the process of logically deriving conclusions from given information, which is the core capability of agents. Traditional CoT prompting methods rely on few-shot examples with limited generalization ability; SFT can only imitate reasoning patterns in training data, making it difficult to innovate. The advantage of reinforcement learning is learning effective reasoning strategies through trial and error, discovering reasoning paths not in training data, learning when deep thinking is needed and when quick answers are possible. Reasoning tasks can be modeled as sequential decision problems. Given question $q$, the agent needs to generate reasoning chain $c = (c_1, c_2, ..., c_n)$ and final answer $a$. The reward function is typically designed as $r(q, c, a) = 1$ if $a = a^*$ else $0$, with training objective $\max_\theta \mathbb{E}{q, (c,a) \sim \pi\theta} [r(q, c, a)]$. Through this approach, the model learns to generate high-quality reasoning chains, not just memorize answers.

Tool Use refers to the agent's ability to call external tools to complete tasks. In tool use tasks, the action space expands to $a_t \in {a_t^{\text{think}}, a_t^{\text{tool}}}$, where $a_t^{\text{think}}$ is generating thinking process, $a_t^{\text{tool}} = (\text{tool_name}, \text{arguments})$ is calling tools. Reinforcement learning allows agents to learn when to use tools, which tool to choose, and how to combine multiple tools. For example, when solving math problems, agents need to learn when to use calculators, when to use code interpreters, and when to reason directly.

Memory refers to the agent's ability to retain and reuse past information, which is crucial for long-term tasks. LLM's context window is limited, and static retrieval strategies (such as RAG) cannot be optimized for tasks. Reinforcement learning allows agents to learn memory management strategies: deciding which information is worth remembering, when to update memory, and when to delete outdated information. This is similar to human working memory, where we actively manage information in our brains, retaining important information and forgetting irrelevant information.

Planning refers to the ability to formulate action sequences to achieve goals. Traditional CoT is linear thinking and cannot backtrack; prompt engineering uses static planning templates that are difficult to adapt to new situations. Reinforcement learning allows agents to learn dynamic planning: discovering effective action sequences through trial and error, learning to balance short-term and long-term benefits. For example, in multi-step tasks, agents may need to first execute some seemingly "detour" steps, such as collecting information, before ultimately completing the task.

Self-Improvement refers to the agent's ability to review its own output, correct errors, and optimize strategies. Reinforcement learning allows agents to learn self-reflection: identifying their own errors, analyzing failure causes, and adjusting strategies. This capability enables agents to continuously improve without human intervention, similar to human "learning from mistakes".

Perception refers to the ability to understand multimodal information. For example, reinforcement learning can enhance visual reasoning capabilities, allowing models to learn to use visual tools and learn visual planning. This enables agents to not only understand text but also understand and operate in the visual world.

11.1.4 HelloAgents' Agentic RL Design

After understanding the core philosophy of Agentic RL, let's see how to implement these capabilities in the HelloAgents framework.

In terms of technology selection, we integrated the TRL (Transformer Reinforcement Learning) framework^[9] and chose the Qwen3-0.6B model^[10]. TRL is Hugging Face's reinforcement learning library, mature and stable, feature-complete, and easy to integrate. Qwen3-0.6B is Alibaba Cloud's small language model, with 0.6B parameters suitable for ordinary GPU training, excellent performance, and open source and free.

HelloAgents' Agentic RL module adopts a four-layer architecture design, as shown in Figure 11.3.

Figure 11.3 HelloAgents Agentic RL Architecture

The bottom layer is the Dataset Layer, containing the GSM8KDataset class, create_sft_dataset() function, and create_rl_dataset() function, responsible for data loading and format conversion. The second layer is the Reward Function Layer, containing the MathRewardFunction base class, AccuracyReward accuracy reward, LengthPenaltyReward length penalty, StepReward step reward, and convenient creation functions create_*_reward(), responsible for defining what good behavior is. The third layer is the Trainer Layer, containing SFTTrainerWrapper and GRPOTrainerWrapper, responsible for specific training logic and LoRA support. The top layer is the Unified Interface Layer, providing RLTrainingTool unified training tool, supporting four operations: action="train" (train model), action="load_dataset" (load dataset), action="create_reward" (create reward function), action="evaluate" (evaluate model).

11.1.5 Quick Start Example

Before diving into learning, let's quickly experience the complete training process. Since this chapter has a lot of theoretical content and practical debugging is quite tedious, we focus on learning to apply rather than constructing tools. First install the HelloAgents framework:

# Install HelloAgents framework (Chapter 11 version)
pip install "hello-agents[rl]==0.2.5"

# Or install from source
cd HelloAgents
pip install -e ".[rl]"

Then run the quick training example:

import sys
import json

from hello_agents.tools import RLTrainingTool

# Create RL training tool
rl_tool = RLTrainingTool()

# 1. Quick test: SFT training (10 samples, 1 epoch)
sft_result_str = rl_tool.run({
    "action": "train",
    "algorithm": "sft",
    "model_name": "Qwen/Qwen3-0.6B",
    "output_dir": "./models/quick_test_sft",
    "max_samples": 10,      # Only use 10 samples for quick test
    "num_epochs": 1,        # Only train 1 epoch
    "batch_size": 2,
    "use_lora": True        # Use LoRA to accelerate training
})

sft_result = json.loads(sft_result_str)
print(f"\n✓ SFT training completed, model saved at: {sft_result['output_dir']}")

# 2. GRPO training (5 samples, 1 epoch)
grpo_result_str = rl_tool.run({
    "action": "train",
    "algorithm": "grpo",
    "model_name": "Qwen/Qwen3-0.6B",  # Use base model
    "output_dir": "./models/quick_test_grpo",
    "max_samples": 5,       # Only use 5 samples for quick test
    "num_epochs": 1,
    "batch_size": 2,        # Must be divisible by num_generations(8), use 2
    "use_lora": True
})

grpo_result = json.loads(grpo_result_str)
print(f"\n✓ GRPO training completed, model saved at: {grpo_result['output_dir']}")

# 3. Evaluate model
eval_result_str = rl_tool.run({
    "action": "evaluate",
    "model_path": "./models/quick_test_grpo",
    "max_samples": 10,      # Evaluate on 10 test samples
    "use_lora": True
})

eval_result = json.loads(eval_result_str)
print(f"\n✓ Evaluation completed:")
print(f"  - Accuracy: {eval_result['accuracy']}")
print(f"  - Average reward: {eval_result['average_reward']}")
print(f"  - Test samples: {eval_result['num_samples']}")

print("\n" + "=" * 50)
print("🎉 Congratulations! You have completed training your first Agentic RL model!")
print("=" * 50)
print(f"\nModel paths:")
print(f"  SFT model: {sft_result['output_dir']}")
print(f"  GRPO model: {grpo_result['output_dir']}")

This quick example demonstrates the complete training process: SFT training allows the model to learn basic reasoning formats and dialogue patterns, GRPO training optimizes reasoning strategies through reinforcement learning to improve accuracy, and model evaluation assesses training effectiveness on the test set. Also, it's normal for accuracy to be very low after running, because the model has only seen 0.7% of training samples and only ran for one epoch.