GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

Kaichen Zhang; Yuzhong Hong; Junwei Bao; Hongfei Jiang; Yang Song; Dingqian Hong; Hui Xiong

cs.AIApr 28, 2025

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

Authors: Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong

Organizations: Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou) · Zuoyebang Education Technology · Department of Computer Science and Engineering, HKUST

Abstract

Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.

Figures & tables

Figure 1: Three equivalent loss functions of GVPO offer distinct interpretations : (1) The Negative log-Likelihood perspective (top) illustrates that GVPO accommodates broader sampling distributions compared to conventional policy gradient methods; (2) The Mean Squared Error interpretation (middle) reveals GVPO’s unique optimal solution, which simultaneously maximizes reward under a KL constraint; and (3) The Reinforcement Learning viewpoint (bottom) highlights GVPO’s implicit regularization terms that ensure stable policy optimization. We assume $\beta=1$ for simplicity.

Algorithm	AIME2024	AMC	MATH500	Minerva	Olympiadbench
Qwen2.5-Math-7B	14.68	38.55	64.00	27.20	30.66
+GRPO	14.79	55.42	80.00	41.17	42.07
+Dr.GRPO	16.56	48.19	81.20	44.48	43.40
+Remax	17.19	60.24	82.00	40.44	45.19
+Reinforce++	16.67	54.22	80.40	43.01	41.78
+ GVPO	20.72	62.65	83.80	45.95	46.96

Table 1: Algorithm Performance Comparison on Mathematical Datasets

Panel of Figure 2 — Figure 2: Ablation on $\beta$ . Each line represents a dataset.

Panel of Figure 3 — Figure 3: Ablation on $k$ . Blue line: GVPO ; Red line: GRPO .

Panel of Figure 4 — Figure 4: Ablation on $\pi_{s}$ . #(historical $y$ ) : #( $y$ from $\pi_{\theta_{\text{old}}}$ )

Algorithm	AIME24	AMC	MATH500	Minerva	Olympiadbench
GVPO	20.72	62.65	83.80	45.95	46.96
GVPO - Var	0.00	0.00	0.00	0.00	0.00
GVPO - Cov	0.00	0.00	0.00	0.00	0.00
GVPO - Var - Cov	7.19	39.76	73.00	34.93	35.70
GVPO - Var + Entropy	0.00	0.00	0.00	0.00	0.00
GVPO - Var + Entropy (LR=1e-6)	3.02	39.76	33.20	23.16	8.89

Table 2: Ablation on Regularization Terms

Algorithm	AIME2024	AMC	MATH500	Minerva	Olympiadbench
GRPO	13.59 $\pm$ 1.11	44.34 $\pm$ 2.19	76.30 $\pm$ 1.35	35.40 $\pm$ 1.04	38.65 $\pm$ 0.74
GVPO	15.56 $\pm$ 1.05	45.78 $\pm$ 2.05	77.36 $\pm$ 0.98	36.03 $\pm$ 1.15	39.58 $\pm$ 1.08

Table 3: Algorithm Performance Comparison with Different Random Seeds

Algorithm	AIME2024	AMC	MATH500	Minerva	Olympiadbench
Llama-3.1-8B-Instruct	5.00	19.27	50.40	22.42	17.04
+GRPO	8.54	19.27	52.60	25.37	16.15
+ GVPO	11.56	20.48	56.60	29.41	20.59

Table 4: Algorithm Performance Comparison with Llama-3.1-8B-Instruct

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Reward	Win Rate	Accuracy	Gpt-4o	Gemini-2.5-pro	Deepseek-R1	Human
SFT	2.84	41.85%	—	—	—	—	—
+DPO	4.83	68.28%	60.43%	31%	40%	25%	34%
+GVPO	5.75	79.49%	64.93%	69%	60%	75%	66%

Appendix

Table 5: Algorithm Performance Comparison on Summarization

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
Sep 18, 2026Kaichen Zhang, Yuzhong Hong, Junwei Bao +4Policy Optimization LLM Post-Training
GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
Aug 18, 2026Peizheng Guo, Jianqi Zhang, Xingyu Zhang +4RL Post-Training Group Relative Policy Optimization
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Key Laboratory of Space Integrated Information System +1
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Apr 2, 2026Gengsheng Li, Tianyu Yang, Junfeng Fang +6RL for Language Model Reasoning On-Policy Self-Distillation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore +2

Sep 18, 2026J/K move · Enter open · S save

GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation

Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang+3

Policy Optimization LLM Post-Training Policy Gradient Methods On-Policy Distillation

Open paper AskPDF arXiv

Post-training plays a pivotal role in enhancing the reasoning capabilities and task-specific expertise of large language models (LLMs). Despite recent advances in post-training methods, such as Group Relative Policy Optimization (GRPO), their practical deployment remains impeded by training instability arising from the reliance on importance sampling. We introduce Group Variance Policy Optimization (GVPO), a novel post-training method that integrates the analytical solution of KL-constrained reward maximization into its gradient weighting scheme. This formulation provides an intuitive interpretation: GVPO's gradient corresponds to the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly to the KL-constrained reward maximization objective, and (2) it enables flexible sampling distributions without requiring importance sampling. Beyond general post-training, we show that GVPO naturally extends to on-policy distillation (OPD). Furthermore, GVPO enables the optimization of a broad family of extended OPD objectives, providing a principled foundation for diverse objective design. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training and on-policy distillation.

Abstract

LLM Post-Training

Explore similar work

GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing