GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation

Kaichen Zhang; Yuzhong Hong; Junwei Bao; Hongfei Jiang; Yang Song; Dingqian Hong; Hui Xiong

cs.AISep 18, 2026

GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation

Authors: Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong

Abstract

Post-training plays a pivotal role in enhancing the reasoning capabilities and task-specific expertise of large language models (LLMs). Despite recent advances in post-training methods, such as Group Relative Policy Optimization (GRPO), their practical deployment remains impeded by training instability arising from the reliance on importance sampling. We introduce Group Variance Policy Optimization (GVPO), a novel post-training method that integrates the analytical solution of KL-constrained reward maximization into its gradient weighting scheme. This formulation provides an intuitive interpretation: GVPO's gradient corresponds to the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly to the KL-constrained reward maximization objective, and (2) it enables flexible sampling distributions without requiring importance sampling. Beyond general post-training, we show that GVPO naturally extends to on-policy distillation (OPD). Furthermore, GVPO enables the optimization of a broad family of extended OPD objectives, providing a principled foundation for diverse objective design. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training and on-policy distillation.

Figures & tables

Fig. 1 : Three equivalent formulations of the GVPO loss provide distinct interpretations : (A) the negative log-likelihood perspective highlights that GVPO can accommodate diverse sampling distributions without requiring importance sampling; (B) the mean squared error perspective reveals GVPO’s unique optimum, which simultaneously maximizes the reward subject to a KL-divergence constraint; and (C) the regularization perspective shows GVPO’s implicit regularization terms, which promote stable policy optimization. We set the hyperparameter $\beta=1$ for simplicity.

Fig. 2 : GVPO for extended on-policy distillation objectives. An extended OPD objective assigns a weighting scheme $f(x,y)$ to each $(x,y)$ and minimizes the reverse KL divergence between the induced distributions $P_{\theta}$ and $P_{t}$ . GVPO optimizes it without computing partition functions, enabling sampling from $\pi_{\theta}$ beyond $P_{\theta}$ while ensuring the desired optimum $\pi_{\theta}^{*}=\pi_{t}$ .

Fig. 3 : OPD’s response length continually decreases under policy-gradient training across different learning rates.

Algorithm	AIME2024	AMC	MATH500	Minerva	Olympiadbench	Average	Optimal Solution Guarantee	No IS Weight for Off-policy
Qwen2.5-Math-7B	14.68	38.55	64.00	27.20	30.66	35.02
+GRPO	14.79	55.42	80.00	41.17	42.07	46.69	$\times$	$\times$
+Dr.GRPO	16.56	48.19	81.20	44.48	43.40	46.77	$\times$	$\times$
+Remax	17.19	60.24	82.00	40.44	45.19	49.01	$\times$	$\times$
+Reinforce++	16.67	54.22	80.40	43.01	41.78	47.22	$\times$	$\times$
+GSPO	18.33	61.44	81.60	40.44	44.74	49.31	$\times$	$\times$

TABLE I : Performance comparison of post-training algorithms on Qwen2.5-Math-7B across mathematical reasoning benchmarks, along with key theoretical properties, including whether each method provides the optimal-solution guarantee and whether it avoids the use of importance sampling for off-policy training.

Algorithm	AIME24	AMC	MATH500	Minerva	Olympiadbench
GVPO	20.72	62.65	83.80	45.95	46.96
GVPO - Var	0.00	0.00	0.00	0.00	0.00
GVPO - Cov	0.00	0.00	0.00	0.00	0.00
GVPO - Var - Cov	7.19	39.76	73.00	34.93	35.70
GVPO - Var + Entropy	0.00	0.00	0.00	0.00	0.00
GVPO - Var + Entropy (LR=1e-6)	3.02	39.76	33.20	23.16	8.89

TABLE II : Ablation of GVPO Regularization Terms

Algorithm	AIME2024	AMC	MATH500	Minerva	Olympiadbench
GRPO	13.59 $\pm$ 1.11	44.34 $\pm$ 2.19	76.30 $\pm$ 1.35	35.40 $\pm$ 1.04	38.65 $\pm$ 0.74
GVPO	15.56 $\pm$ 1.05	45.78 $\pm$ 2.05	77.36 $\pm$ 0.98	36.03 $\pm$ 1.15	39.58 $\pm$ 1.08

TABLE III : Algorithm Performance Comparison across Different Random Seeds

Algorithm	Algorithmic Thinking	Constraint Satisfaction	Logical Reasoning	Mathematical Domains	Pattern Recognition
Qwen2.5-1.5B-Instruct	7.00	1.24	24.17	22.73	13.90
+GRPO	27.95	11.90	61.83	37.52	25.65
+ GVPO	38.38	17.83	70.72	63.90	30.07

TABLE IV : Algorithm Performance Comparison on Reasoning Gym across Multiple Domains

Algorithm	AIME24	AMC	MATH500	Minerva	Olympiad
Llama	5.00	19.27	50.40	22.42	17.04
+GRPO	8.54	19.27	52.60	25.37	16.15
+ GVPO	11.56	20.48	56.60	29.41	20.59

TABLE V : Performance Comparison on Llama-3.1-8B-Instruct

Teacher: DeepSeek-R1-Distill-Llama-8B
Algorithm	AIME2024	AMC	MATH500	Minerva	Olympiadbench	Average
SFT	1.66	15.66	49.40	15.44	15.70	19.57
Seqkd	1.45	16.86	45.40	15.44	16.14	19.06
PG	7.29	27.71	49.20	16.17	23.40	24.75
GVPO	9.06	39.75	68.40	29.41	33.33	35.99
Teacher: DeepSeek-R1-Distill-Qwen-7B

TABLE VI : Performance comparison of on-policy distillation algorithms across mathematical reasoning benchmarks.

Panel of Figure 11 — Fig. 7 : Ablation on $\alpha$ . Each line represents a dataset. The left and right figures show results for Llama and Qwen, respectively.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Reward	Win Rate	Accuracy	Gpt-4o	Gemini-2.5-pro	Deepseek-R1	Human
SFT	2.84	41.85%	—	—	—	—	—
+DPO	4.83	68.28%	60.43%	31%	40%	25%	34%
+GVPO	5.75	79.49%	64.93%	69%	60%	75%	66%

Appendix

TABLE VII : Algorithm Performance Comparison on Summarization

Algorithm	Algorithmic Thinking	Constraint Satisfaction	Logical Reasoning	Mathematical Domains	Pattern Recognition
Qwen2.5-1.5B-Instruct	7.00	1.24	24.17	22.73	13.90
+GRPO	27.95	11.90	61.83	37.52	25.65
+ GVPO	38.38	17.83	70.72	63.90	30.07

Appendix

TABLE VIII : Algorithm Performance Comparison on Reasoning Gym across Multiple Domains

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 221 papers

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Apr 28, 2025Kaichen Zhang, Yuzhong Hong, Junwei Bao +4LLM Post-Training LLM Alignment
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou) · Zuoyebang Education Technology · Department of Computer Science and Engineering, HKUST
GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
Aug 18, 2026Peizheng Guo, Jianqi Zhang, Xingyu Zhang +4RL Post-Training Group Relative Policy Optimization
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Key Laboratory of Space Integrated Information System +1
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Apr 2, 2026Gengsheng Li, Tianyu Yang, Junfeng Fang +6RL for Language Model Reasoning On-Policy Self-Distillation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore +2

Apr 28, 2025J/K move · Enter open · S save

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang+3

Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou) · Zuoyebang Education Technology · Department of Computer Science and Engineering, HKUST

LLM Post-Training LLM Alignment RL Post-Training Policy Gradient Methods KL-Regularized RL

Open paper AskPDF arXiv

Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.

Abstract

Policy Optimization

Explore similar work

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing