CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

Sijia Cui; Pengyu Cheng; Jiajun Song; Yongbo Gai; Guojun Zhang; Zhechao Yu; Jianhe Lin; Xiaoxi Jiang

cs.LGMar 10, 2026

CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

Authors: Sijia Cui, Pengyu Cheng, Jiajun Song, Yongbo Gai, Guojun Zhang, Zhechao Yu, Jianhe Lin, Xiaoxi Jiang,

Organizations: Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.

Figures & tables

Figure 1: Intuition of CLIPO. Standard RLVR only relies on outcomes, neglecting the quality of intermediate reasoning steps. CLIPO addresses this limitation by maximizing similarity between successful reasoning trajectories. By aligning multiple positive rollouts, CLIPO identifies the invariant reasoning structure, i.e. , the “overlap” of successful paths, implicitly eliminating incorrect and hallucinative reasoning steps.

Figure 2: Framework of CLIPO. For each input prompt

{\bm{x}}

, policy optimization methods generate a group of rollouts

\{{\bm{y}}_{1},{\bm{y}}_{2},\dots,{\bm{y}}_{G}\}

then calculate corresponding RLVR rewards

\{r_{1},r_{2},\dots,r_{G}\}

. CLIPO applies a contrastive head on the top of the last hidden states

\{{\bm{h}}_{1},{\bm{h}}_{2},\dots,{\bm{h}}_{G}\}

of the rollout group and outputs trajectory-level semantic embeddings

\{{\bm{e}}_{1},{\bm{e}}_{2},\dots,{\bm{e}}_{G}\}

. The contrastive rewards

\{r_{1}^{\text{CL}},r_{2}^{\text{CL}},\dots,r_{G}^{\text{CL}}\}

are computed across the semantic embedding group to provide the similarity of successful and failed trajectories. The final CLIPO reward for the

i

-th rollout is

r^{\prime}_{i}=r_{i}+r^{\text{CL}}_{i}

Methods	GSM8K	SYM	P1	P2	cQA	tQA	thrmQA	MMLU	M-Avg.	G-Avg.	Avg.
Base Model	$86.05$	$82.40$	$70.90$	$47.96$	$73.63$	$29.99$	$24.90$	$68.09$	$71.83$	$49.15$	$60.49$
GRPO	$87.79$	$84.04$	$73.12$	$50.80$	$74.94$	$30.48$	27.71	$68.19$	$73.94$	$50.33$	$62.13$
GSPO	$87.19$	$83.94$	$73.28$	$51.36$	$74.86$	$30.23$	$27.38$	$67.68$	$73.94$	$50.04$	$61.99$
DAPO	$87.41$	$84.56$	74.70	$52.32$	$76.00$	$31.58$	$27.11$	69.00	$74.75$	$50.92$	$62.84$
GMPO	$86.81$	$84.02$	$73.90$	$49.80$	$76.33$	$30.97$	$26.57$	$68.49$	$73.63$	$50.59$	$62.11$
GRPO+CLIPO	88.02	$84.62$	$74.60$	54.16	76.90	$31.52$	27.58	68.64	75.35	51.16	63.26

Table 1: Performance comparison of different RLVR methods on Track I(GSM8K and General Reasoning). ‘Avg.’ represents the average performance across all datasets, and ‘M-Avg.’ represents the average performance on math-related datasets. ‘G-Avg.’ represents the average performance on general and QA tasks. ‘

\Delta

’ represents the improvement after using our method, with light green indicating improvement and light red indicating decline. The best results are highlighted in bold , and the second-best results are underlined .

Methods	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
Base Model	$75.35$	$59.24$	$35.31$	$44.13$	$10.63$	$6.25$	$56.63$	$20.33$	$38.48$
GRPO	$76.46$	$64.83$	$37.86$	$46.84$	$17.92$	$9.58$	$59.72$	$24.78$	$42.25$
GSPO	77.58	$64.56$	$38.36$	50.08	$16.25$	$10.42$	$60.17$	$25.58$	$42.87$
DAPO	$76.03$	$66.08$	39.94	$47.36$	$19.79$	$7.92$	$60.68$	$25.02$	$42.85$
GMPO	78.47	$64.27$	$37.25$	$48.42$	$19.58$	$9.58$	$60.00$	$25.86$	$42.93$
GRPO+CLIPO	$77.49$	67.21	$38.97$	$49.17$	$18.13$	$10.63$	$61.22$	$25.97$	$43.60$

Table 2: Performance comparison on Track II(Competition-Level Reasoning). ‘M-Avg.’ represents the average performance on Math, Math_S, Math_H. ‘C-Avg.’ represents the average performance on competition-level datasets including AMC, AIME, and AIME25.

Track	Methods	Metric 1	Metric 2	Avg.
Track I	CLIPO	75.35	51.16	63.26
	CLIPO-fixed	74.73	50.24	62.48
	$\Delta$	-0.62	-0.92	-0.77
Track II	CLIPO	61.22	25.97	43.60
	CLIPO-fixed	60.19	25.06	42.63
	$\Delta$	-1.03	-0.91	-0.97

Table 3: Comparison between CLIPO with and without fixed head on Track I and Track II. Metric 1 and Metric 2 represent M-Avg., G-Avg. for Track I, and M-Avg., C-Avg. for Track II, respectively.

\Delta

indicates the performance drop when using fixed head.

$\tau$	Math-Avg.	Comp-Avg.	Avg.
$0.2$	59.90	24.36	42.13
$0.1$	59.76	26.81	43.28
$0.05$	61.28	24.85	43.07
$0.02$	61.22	25.97	43.60
$0.015$	60.64	25.19	42.91

Table 5: Effect of different contrastive learning temperature

\tau

on CLIPO performance. It can be observed that lower temperature coefficients lead to better performance improvements; however, excessively low temperature may result in degraded performance.

GroupSize	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
$\|G\|=8$	$77.41$	$64.90$	$38.49$	$47.67$	$15.42$	$8.54$	$60.27$	$23.87$	$42.07$
$\|G\|=16$	77.49	67.21	$38.97$	$49.17$	$18.13$	10.63	61.22	$25.97$	$43.60$
$\|G\|=32$	$77.26$	$66.03$	39.89	50.45	20.83	$8.33$	$61.06$	26.54	43.80

Table 6: Effect of different group sizes on CLIPO performance. Larger group sizes lead to better performance improvements. The best results are highlighted in bold. It can be observed that larger group sizes generally lead to better performance improvements across Avg. metrics.

Model	Method	Math-Avg.	Comp-Avg.	Avg.
DS-7B	GRPO	79.39	45.38	62.38
	+CLIPO	79.28	46.54	62.91
	$\Delta$	-0.11	1.17	0.53
Llama-8B	GRPO	34.84	11.42	23.13
	+CLIPO	37.10	11.79	24.44
	$\Delta$	2.25	0.37	1.31

Table 7: Performance comparison on diverse models.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

General Settings
Category	Parameter	Track I	Track II
Total epochs	total_epochs	6	9
Critic warmup epochs	critic_warmup	14 step(1 epoch)	58 step(1 epoch)
Data and Sequence Lengths
Training batch size	train_batch_size	512	128
PPO mini-batch size	ppo_mini_batch_size	256	128

Appendix

Table 8: Default Training Hyperparameters and Configuration on Track I and Track II.

Contrastive LM Head Settings
Parameter	Track I	Track II
Loss type	infonce_loss	infonce_loss
Distance measure	cosine_distance	cosine_distance
Head type	linear	linear
Output size	512	2048
Contrastive temperature	0.05	0.05

Appendix

Table 9: Default Contrastive Head Parameters on Track I and Track II.

Dataset	Instruction Template
GSM8K / GSM-P1 / GSM-P2 / GSM-Symbolic	Let’s think step by step and output the final answer within \boxed{}.
MATH / AMC / AIME / AIME2025	Let’s think step by step and output the final answer within \boxed{}.
TheoremQA	Let’s think step by step and output the final answer within \boxed{}.
MMLU	Let’s think step by step and output the final answer (eg, A, B, C, D) within \boxed{}.
TruthfulQA	Let’s think step by step and output the final answer (eg, A, B, C, D, …) within \boxed{}.
CommonsenseQA	Let’s think step by step and output the final answer (eg, A, B, C, D, E) within \boxed{}.

Appendix

Table 10: Prompt Construction and Instruction Templates

Figure 4: The t-SNE visualization of semantic embeddings produced by the contrastive head at the start of training (left) and after $3$ epochs of training (right). Green points represent embeddings from correct rollouts, while red points correspond to incorrect ones. After training, correct responses cluster closely together, forming more distinct group clusters. Within these clusters, correct and incorrect responses also exhibit some separation.

Methods	GSM8K	SYM	P1	P2	cQA	tQA	thrmQA	MMLU	M-Avg.	G-Avg.	Avg.
CLIPO	88.02	84.62	74.60	54.16	76.90	31.52	27.58	68.64	75.35	51.16	63.26
CLIPO-fixed	88.63	85.12	73.48	51.68	75.59	30.11	26.77	68.49	74.73	50.24	62.48
$\Delta$	0.61	0.50	-1.12	-2.48	-1.31	-1.41	-0.80	-0.15	-0.62	-0.92	-0.77

Appendix

Table 11: Performance comparison of fixed-head ablation on Track I. CLIPO-fixed shows performance drops compared to CLIPO across most benchmarks.

Methods	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
CLIPO	77.49	67.21	38.97	49.17	18.13	10.63	61.22	25.97	43.60
CLIPO-fixed	76.76	64.83	38.99	49.77	17.92	7.50	60.19	25.06	42.63
$\Delta$	-0.73	-2.38	0.02	0.60	-0.21	-3.13	-1.03	-0.91	-0.97

Appendix

Table 12: Performance comparison of fixed-head ablation on Track II. CLIPO-fixed shows performance drops compared to CLIPO across most benchmarks.

Methods	GSM8K	SYM	P1	P2	cQA	tQA	thrmQA	MMLU	M-Avg.	G-Avg.	Avg.
GRPO	$87.79$	$84.04$	$73.12$	$50.80$	$74.94$	$30.48$	$27.71$	$68.19$	$73.94$	$50.33$	$62.13$
CLIPO-InfoNCE	$88.02$	$84.62$	$74.60$	$54.16$	$76.90$	$31.52$	$27.58$	$68.64$	$75.35$	$51.16$	$63.26$
$\Delta$	0.23	0.58	1.48	3.36	1.97	1.04	-0.13	0.46	1.41	0.83	1.12
CLIPO-SupCon	$87.79$	$85.04$	$74.00$	$51.12$	$77.07$	$30.23$	$27.04$	$69.25$	$74.49$	$50.90$	$62.69$
$\Delta$	0.00	1.00	0.88	0.32	2.13	-0.24	-0.67	1.06	0.55	0.57	0.56
CLIPO-SoftNN	$87.11$	$84.70$	$74.06$	$53.92$	$75.27$	$29.99$	$27.31$	$68.59$	$74.95$	$50.29$	$62.62$

Appendix

Table 13: Performance comparison of Contrastive Loss Variants on Track I. The improvements (

\Delta

) over base method(GRPO) are indicated with positive (green) and negative (red) colors.

Methods	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
GRPO	$76.46$	$64.83$	$37.86$	$46.84$	$17.92$	$9.58$	$59.72$	$24.78$	$42.25$
CLIPO-InfoNCE	$77.49$	$67.21$	$38.97$	$49.17$	$18.13$	$10.63$	$61.22$	$25.97$	$43.60$
$\Delta$	1.03	2.38	1.11	2.33	0.21	1.04	1.50	1.19	1.35
CLIPO-SupCon	$78.18$	$65.90$	$38.97$	$48.42$	$17.71$	$6.88$	$61.01$	$24.33$	$42.67$
$\Delta$	1.72	1.06	1.11	1.58	-0.21	-2.71	1.30	-0.45	0.43
CLIPO-SoftNN	$78.17$	$65.78$	$38.02$	$49.85$	$20.21$	$9.58$	$60.66$	$26.55$	$43.60$

Appendix

Table 14: Performance comparison of Contrastive Loss Variants on Track II. The improvements (

\Delta

) over base method(GRPO) are indicated with positive (green) and negative (red) colors.

Methods	GSM	SYM	P1	P2	cQA	tQA	thrmQA	MMLU	M-Avg.	G-Avg.	Avg.
Base	$86.05$	$82.40$	$70.90$	$47.96$	$73.63$	$29.99$	$24.90$	$68.09$	$71.83$	$49.15$	$60.49$
GRPO	$87.79$	$84.04$	$73.12$	$50.80$	$74.94$	$30.48$	$27.71$	$68.19$	$73.94$	$50.33$	$62.13$
CLIPO-SoftNN	$87.11$	$84.70$	$74.06$	$53.92$	$75.27$	$29.99$	$27.31$	$68.59$	$74.95$	$50.29$	$62.62$
$\Delta$	-0.68	0.66	0.94	3.12	0.33	-0.49	-0.40	0.41	1.01	-0.04	0.48
GSPO	$87.19$	$83.94$	$73.28$	$51.36$	$74.86$	$30.23$	$27.38$	$67.68$	$73.94$	$50.04$	$61.99$
CLIPO-SoftNN	$87.87$	$83.92$	$75.14$	$51.60$	$76.33$	$30.97$	$27.64$	$68.44$	$74.63$	$50.85$	$62.74$

Appendix

Table 15: Performance comparison of different RLVR methods. CLIPO-SoftNN improves the average performance based on four baseline methods, with the main improvements in M-Avg. tasks, while the improvements in G-Avg. tasks are limited, and even some decreases.

$\tau$	GSM8K	SYM	P1	P2	cQA	tQA	thrmQA	MMLU	M-Avg.	G-Avg.	Avg.
0.2	$87.34$	$84.58$	$73.42$	$50.48$	76.33	32.74	29.38	$67.73$	$73.95$	51.55	$62.75$
0.1	88.02	$85.38$	$74.50$	$49.64$	$75.92$	$31.95$	$27.91$	$68.14$	$74.39$	$50.98$	$62.68$
0.05	$87.41$	85.44	$74.86$	$53.16$	$75.18$	$30.42$	$27.71$	68.79	$75.22$	$50.53$	62.87
0.02	$87.26$	$85.00$	75.64	54.00	$75.10$	$30.17$	$27.31$	$68.34$	75.48	$50.23$	$62.85$

Appendix

Table 16: Performance comparison of different temperature settings on Track I.

$\tau$	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
0.2	$77.13$	$66.12$	$36.44$	$47.67$	$17.92$	$7.50$	$59.90$	$24.36$	$42.13$
0.1	$76.90$	$65.01$	$37.36$	51.05	18.75	10.63	$59.76$	26.81	$43.28$
0.05	$77.18$	$67.14$	39.53	$48.72$	$16.04$	$9.79$	61.28	$24.85$	$43.07$
0.02	77.49	67.21	$38.97$	$49.17$	$18.13$	10.63	$61.22$	$25.97$	43.60
0.015	$76.79$	$66.42$	$38.72$	$50.15$	$17.71$	$7.71$	$60.64$	$25.19$	$42.91$

Appendix

Table 17: Performance comparison of different temperature settings on Track II.

Figure 5: The plot illustrates the average cosine similarity of positive pairs within the same group during training, under different temperature settings. We observe that higher temperature values lead to greater fluctuations in the similarity of positive pairs.

$\tau$	GroupSize	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
0.05	$\|G\|=8$	$76.56$	$64.92$	$38.09$	$46.69$	$17.50$	10.00	$59.86$	$24.73$	$42.29$
	$\|G\|=16$	$77.18$	67.14	39.53	$48.72$	$16.04$	$9.79$	61.28	$24.85$	$43.07$
	$\|G\|=32$	77.92	$64.90$	$39.37$	49.55	21.04	$7.08$	$60.73$	25.89	43.31
0.02	$\|G\|=8$	$77.41$	$64.90$	$38.49$	$47.67$	$15.42$	$8.54$	$60.27$	$23.87$	$42.07$
	$\|G\|=16$	77.49	67.21	$38.97$	$49.17$	$18.13$	10.63	61.22	$25.97$	$43.60$
	$\|G\|=32$	$77.26$	$66.03$	39.89	50.45	20.83	$8.33$	$61.06$	26.54	43.80

Appendix

Table 18: Performance comparison of different group size.

DeepSeek-R1-Distill-Qwen-7B
Methods	Math	Math_S	Math_H	AMC	AIME	AIME25	M-Avg.	C-Avg.	Avg.
GRPO	89.70	83.98	64.48	72.59	36.88	26.67	79.39	45.38	62.38
CLIPO	90.08	83.94	63.83	73.80	39.38	26.46	79.28	46.54	62.91
$\Delta$	0.38	-0.05	-0.65	1.20	2.50	-0.21	-0.11	1.17	0.53
Llama3.1-8B-Instruct
GRPO	52.04	40.10	12.39	23.64	10.63	0.00	34.84	11.42	23.13

Appendix

Table 19: Performance comparison on various base models.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR
Aug 4, 2026Yongshi Ye, Liang Zhang, Yidong Chen +2RL for Language Model Reasoning Group Relative Policy Optimization
Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism
Experience Augmented Policy Optimization for LLM Reasoning
Jun 29, 2026Jinda Lu, Kexin Huang, Junkang Wu +7Reinforcement Learning RL for Language Model Reasoning
University of Science and Technology of China · 2Peking University · 3Dartmouth College +1
P^2O: Joint Policy and Prompt Optimization
Mar 23, 2026Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2

Aug 4, 2026J/K move · Enter open · S save

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

Yongshi Ye, Liang Zhang, Yidong Chen, Xiaodong Shi+1

Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points.

Abstract

RL for Language Model Reasoning

Explore similar work

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

Experience Augmented Policy Optimization for LLM Reasoning

P^2O: Joint Policy and Prompt Optimization