11.4 GRPO Training

After completing SFT training, we have obtained a model capable of generating structured answers. However, the SFT model has only learned to "imitate" the reasoning process in training data and hasn't truly learned to "think". Reinforcement learning can allow the model to optimize reasoning strategies through trial and error, thereby surpassing the quality of training data.

11.4.1 From PPO to GRPO

In the field of reinforcement learning, PPO (Proximal Policy Optimization)^[1] is one of the most classic algorithms. PPO ensures training stability by limiting the magnitude of policy updates. However, PPO has some problems in LLM training: it requires training a Value Model, increasing training complexity and memory usage; it requires maintaining four models simultaneously (Policy Model, Reference Model, Value Model, Reward Model), making engineering implementation complex; training is unstable, prone to reward collapse or policy degradation.

GRPO (Group Relative Policy Optimization)^[2] is a simplified PPO variant specifically designed for LLMs. GRPO's core idea is: no need for Value Model, using group-relative rewards instead of absolute rewards; simplified training process, only requiring Policy Model and Reference Model; improved training stability, reducing risk of reward collapse.

Let's understand GRPO's principles through mathematical formulas. PPO's objective function is:

$$ J_{\text{PPO}}(\theta) = \mathbb{E}{s,a \sim \pi\theta} \left[ \min\left( \frac{\pi_\theta(a|s)}{\pi_{\text{old}}(a|s)} A(s,a), \text{clip}\left(\frac{\pi_\theta(a|s)}{\pi_{\text{old}}(a|s)}, 1-\epsilon, 1+\epsilon\right) A(s,a) \right) \right] $$

Where $A(s,a)$ is the advantage function, requiring Value Model to estimate:

$$ A(s,a) = Q(s,a) - V(s) = r(s,a) + \gamma V(s') - V(s) $$

GRPO's objective function is simplified to:

$$ J_{\text{GRPO}}(\theta) = \mathbb{E}{s,a \sim \pi\theta} \left[ \frac{\pi_\theta(a|s)}{\pi_{\text{ref}}(a|s)} \cdot (r(s,a) - \bar{r}{\text{group}}) \right] - \beta \cdot D(\pi_\theta || \pi_{\text{ref}}) $$

Where $\bar{r}{\text{group}}$ is the group average reward and $\beta$ is the KL divergence penalty coefficient. Key differences are: GRPO uses $r(s,a) - \bar{r}{\text{group}}$ instead of advantage function $A(s,a)$, no need for Value Model; GRPO uses group-relative rewards, reducing reward variance; GRPO adds KL divergence penalty, preventing policy from deviating too far.

As shown in Figure 11.7, comparison of PPO and GRPO training processes.

Figure 11.7 PPO vs GRPO Training Process

As can be seen, GRPO eliminates Value Model training, greatly simplifying the process.

As shown in Table 11.6, detailed comparison of PPO and GRPO.

Table 11.6 PPO vs GRPO Comparison

For LLM training, GRPO is a better choice because it is simpler, more stable, and has lower memory usage.

11.4.2 GRPO Training Practice

Now let's conduct GRPO training using HelloAgents. The prerequisite for GRPO training is completing SFT training, because GRPO requires a reasonable initial policy.

Basic GRPO training example:

from hello_agents.tools import RLTrainingTool

# Create training tool
rl_tool = RLTrainingTool()

# GRPO training
result = rl_tool.run({
    # Training configuration
    "action": "train",
    "algorithm": "grpo",

    # Model configuration
    "model_name": "./models/sft_full",  # Start from SFT model
    "output_dir": "./models/grpo_model",

    # Data configuration
    "max_samples": 100,     # Use 100 samples for quick test

    # Training parameters
    "num_epochs": 3,
    "batch_size": 4,
    "learning_rate": 1e-5,  # GRPO learning rate usually smaller than SFT

    # GRPO-specific parameters
    "num_generations": 4,   # Generate 4 answers per question
    "kl_coef": 0.05,        # KL divergence penalty coefficient

    # LoRA configuration
    "use_lora": True,
    "lora_rank": 16,
    "lora_alpha": 32,

    # Reward function configuration
    "reward_type": "accuracy",  # Use accuracy reward
})

print(f"\n✓ Training completed!")
print(f"  - Model save path: {result['model_path']}")
print(f"  - Training samples: {result['num_samples']}")
print(f"  - Training epochs: {result['num_epochs']}")
print(f"  - Average reward: {result['average_reward']:.4f}")

If average reward gradually increases and KL divergence remains in a reasonable range during GRPO training, it indicates training is proceeding normally.

GRPO has some specific parameters that need to be understood and tuned.

Generation Parameters:

• num_generations: How many answers to generate per question. More is better, but computational cost is also higher. Typical values are 4-8. The purpose of generating multiple answers is to calculate group-relative rewards and increase diversity of training signals.
• max_new_tokens: Maximum number of tokens to generate per answer. Too few may truncate answers, too many wastes computation. Recommend 256-512.
• temperature: Generation temperature, controls randomness. 0 means greedy decoding, 1 means standard sampling. GRPO recommends 0.7-1.0, maintaining some exploration.

Optimization Parameters:

• learning_rate: GRPO's learning rate is usually smaller than SFT because we don't want to deviate too far from the SFT model. Recommend 1e-6 to 1e-5; too large a learning rate (e.g., 5e-5) on small models may cause policy collapse.
• kl_coef: KL divergence penalty coefficient, controls magnitude of policy updates. Too small (0.01) may cause policy to deviate too far, too large (0.5) may limit learning. Recommend 0.05-0.1.
• clip_range: Policy ratio clipping range, similar to PPO's epsilon. Recommend 0.2.

Reward Parameters:

• reward_type: Reward function type, can be "accuracy", "length_penalty", "step", or "combined".
• reward_config: Additional configuration for reward function, such as target length for length penalty, coefficient for step reward, etc.

Let's conduct a complete GRPO training using all data and best practices:

from hello_agents.tools import RLTrainingTool

rl_tool = RLTrainingTool()

# Complete GRPO training
result = rl_tool.run({
    "action": "train",
    "algorithm": "grpo",

    # Model configuration
    "model_name": "./models/sft_full",
    "output_dir": "./models/grpo_full",

    # Data configuration
    "max_samples": None,    # Use all data

    # Training parameters
    "num_epochs": 3,
    "batch_size": 4,
    "learning_rate": 1e-5,
    "warmup_ratio": 0.1,

    # GRPO-specific parameters
    "num_generations": 4,
    "max_new_tokens": 512,
    "temperature": 0.8,
    "kl_coef": 0.05,
    "clip_range": 0.2,

    # LoRA configuration
    "use_lora": True,
    "lora_rank": 16,
    "lora_alpha": 32,

    # Reward function configuration
    "reward_type": "combined",
    "reward_config": {
        "components": [
            {"type": "accuracy", "weight": 1.0},
            {"type": "length_penalty", "weight": 0.5, "target_length": 200},
            {"type": "step", "weight": 0.3, "step_bonus": 0.1}
        ]
    },

    # Other configurations
    "save_steps": 500,
    "logging_steps": 100,
})

print(f"Training completed! Model saved at: {result['model_path']}")

11.4.3 GRPO Training Process Analysis

Let's deeply understand GRPO's training process and see what happens at each step.

(1) Training Loop

GRPO's training loop includes the following steps:

1. Sampling Phase: For each question, use current policy to generate multiple answers (num_generations). These answers form a "group" for calculating relative rewards.

2. Reward Calculation: Calculate reward $r_i$ for each generated answer. Rewards can be accuracy, length penalty, step reward, or their combination.

3. Relative Reward: Calculate group average reward $\bar{r} = \frac{1}{N}\sum_{i=1}^{N} r_i$, then calculate relative reward $\hat{r}_i = r_i - \bar{r}$. The benefit of this is reducing reward variance and making training more stable.

4. Policy Update: Use relative rewards to update policy, while adding KL divergence penalty to prevent policy from deviating too far from reference model.

5. Repeat: Repeat above steps until all training epochs are complete.

Let's understand through a specific example:

# Assume we have a question
question = "What is 48 + 24?"

# Generate 4 answers
answers = [
    "48 + 24 = 72. Final Answer: 72",      # Correct
    "48 + 24 = 72. Final Answer: 72",      # Correct
    "48 + 24 = 70. Final Answer: 70",      # Incorrect
    "Let me think... 72. Final Answer: 72" # Correct but verbose
]

# Calculate rewards (assuming using accuracy + length penalty)
rewards = [1.0, 1.0, 0.0, 0.8]  # 4th answer penalized for verbosity

# Calculate group average reward
avg_reward = (1.0 + 1.0 + 0.0 + 0.8) / 4 = 0.7

# Calculate relative rewards
relative_rewards = [
    1.0 - 0.7 = 0.3,   # Correct and concise, positive relative reward
    1.0 - 0.7 = 0.3,   # Correct and concise, positive relative reward
    0.0 - 0.7 = -0.7,  # Incorrect, negative relative reward
    0.8 - 0.7 = 0.1    # Correct but verbose, smaller relative reward
]

# Policy update: increase probability of first two answers, decrease probability of third answer

As can be seen, the relative reward mechanism encourages the model to generate answers "better than average" rather than simply pursuing high rewards. This can reduce reward variance and improve training stability.

(2) KL Divergence Penalty

KL divergence penalty is a key component of GRPO, preventing policy from deviating too far from the reference model. KL divergence is defined as:

$$ D_{KL}(\pi_\theta || \pi_{\text{ref}}) = \mathbb{E}{s,a \sim \pi\theta} \left[ \log \frac{\pi_\theta(a|s)}{\pi_{\text{ref}}(a|s)} \right] $$

In practice, we calculate KL divergence for each token, then sum:

$$ D_{KL} = \sum_{t=1}^{T} \log \frac{\pi_\theta(a_t|s, a_{<t})}{\pi_{\text{ref}}(a_t|s, a_{<t})} $$

The larger the KL divergence, the greater the difference between current policy and reference model. By adding KL divergence penalty term $-\beta \cdot D_{KL}$, we limit the magnitude of policy updates, avoiding "forgetting" knowledge learned during SFT phase.

The choice of kl_coef ($\beta$) is important:

• Too small (0.01): Policy may deviate too far, causing output format confusion or quality degradation
• Too large (0.5): Policy updates are limited, learning is slow, difficult to surpass SFT model
• Recommended (0.05-0.1): Balance exploration and stability

(3) Training Monitoring

During GRPO training, we need to monitor the following metrics:

• Average Reward: Should gradually increase. If reward doesn't increase, learning rate may be too small, KL penalty too large, or reward function design unreasonable. If reward rises then falls, may be overfitting or reward collapse.

• KL Divergence: Should remain in reasonable range (0.01-0.1). If KL divergence is too large (>0.5), policy deviates too far, need to increase kl_coef or reduce learning rate. If KL divergence is too small (<0.001), policy is barely updating, need to reduce kl_coef or increase learning rate.

• Accuracy: Should gradually improve. This is the most intuitive metric, reflecting the model's actual capability.

• Generation Quality: Need manual inspection of generated answers to ensure correct format and clear reasoning.

HelloAgents integrates two mainstream training monitoring tools: Weights & Biases (wandb) and TensorBoard.

Method 1: Using Weights & Biases (Recommended)

Weights & Biases is currently the most popular machine learning experiment tracking platform, providing powerful visualization and experiment management features.

import os

# 1. Set up wandb (need to register account first: https://wandb.ai)
os.environ["WANDB_PROJECT"] = "hello-agents-grpo"  # Project name
os.environ["WANDB_LOG_MODEL"] = "false"            # Don't upload model files

# 2. Enable wandb in training configuration
result = rl_tool.run({
    "action": "train",
    "algorithm": "grpo",
    "model_name": "Qwen/Qwen3-0.6B",
    "output_dir": "./models/grpo_monitored",
    "num_epochs": 2,
    "batch_size": 2,
    "use_lora": True,
    # wandb will automatically log all training metrics
})

# After training completes, visit https://wandb.ai to view training curves

wandb will automatically log the following metrics:

• train/reward: Average reward
• train/kl: KL divergence
• train/loss: Training loss
• train/learning_rate: Learning rate
• train/epoch: Training epoch

Method 2: Using TensorBoard

TensorBoard is a visualization tool provided by TensorFlow, also supporting PyTorch training.

# 1. TensorBoard logs will be automatically created in output_dir during training
result = rl_tool.run({
    "action": "train",
    "algorithm": "grpo",
    "model_name": "Qwen/Qwen3-0.6B",
    "output_dir": "./models/grpo_tb",
    "num_epochs": 2,
    "batch_size": 2,
    "use_lora": True,
})

# 2. Launch TensorBoard to view training curves
# Run in command line:
# tensorboard --logdir=./models/grpo_tb
# Then visit http://localhost:6006

Method 3: Offline Monitoring (No External Tools Required)

If you don't want to use wandb or TensorBoard, you can also monitor through training logs:

# Training process will print detailed logs
result = rl_tool.run({
    "action": "train",
    "algorithm": "grpo",
    "model_name": "Qwen/Qwen3-0.6B",
    "output_dir": "./models/grpo_simple",
    "num_epochs": 2,
    "batch_size": 2,
    "use_lora": True,
})

# Log example:
# Epoch 1/2 | Step 100/500 | Reward: 0.45 | KL: 0.023 | Loss: 1.234
# Epoch 1/2 | Step 200/500 | Reward: 0.52 | KL: 0.031 | Loss: 1.156
# ...

In GRPO training, you may encounter some problems. When reward doesn't increase, it may be that learning rate is too small or KL penalty is too large limiting policy updates, or reward function design is unreasonable or SFT model quality is too poor. In this case, increase learning rate (from 1e-5 to 5e-5), reduce kl_coef (from 0.1 to 0.05), check reward function, or retrain SFT model.

When KL divergence explodes (exceeds 0.5 or even 1.0) causing generated answer format confusion, it's usually because learning rate is too large or KL penalty is too small, or reward function is too aggressive. You can reduce learning rate (from 5e-5 to 1e-5), increase kl_coef (from 0.05 to 0.1), adjust reward function, or use gradient clipping.

When generation quality degrades (accuracy improves but format is confused, reasoning unclear), it may be that reward function only focuses on accuracy ignoring other quality metrics, or KL penalty is too small causing model to deviate too far from SFT, or overfitting occurs. In this case, use combined reward function to optimize multiple metrics simultaneously, increase kl_coef to maintain consistency, reduce training epochs, or increase training data.

GRPO training has higher memory usage than SFT because it needs to generate multiple answers simultaneously and store reference model outputs, prone to OOM. You can reduce num_generations (from 8 to 4), batch_size (from 4 to 2), or max_new_tokens (from 512 to 256), or use gradient checkpointing and mixed precision training to alleviate.