Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence

Yiran Zhang; Hanbo Huang; Jielan Li; Zhen Zhao; Shiyu Liang; Hongxia Hao

cs.CLOct 15, 2026

Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence

Authors: Yiran Zhang, Hanbo Huang, Jielan Li, Zhen Zhao, Shiyu Liang, Hongxia Hao

Organizations: Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but group-relative policy optimization broadcasts a single sequence-level advantage to every token of a response, although errors usually localize to a few decisive steps. Recent self-distilled variants refine credit with a teacher--student likelihood gap. However, this signal cannot tell whether it relies on the image or on a language shortcut, and is applied uniformly even when the teacher is uninformative. To address this, we propose Pairwise token contrast and Adaptive trajectory ConfidencE (PACE). Pairwise token contrast scores each response under four teacher views, crossing a correct or incorrect answer hint with the image present or removed, and compares the two hints for every token, which cancels shifts shared by both hints and reveals whether the token's support relies on the image. Adaptive trajectory confidence measures how well the teacher predicts each response given the correct hint and the image, normalizes this score within the batch, and uses it to set how far the token contrast redistributes credit within that response. PACE achieves the best average accuracy on three backbones and ranks first on all evaluated multimodal reasoning benchmarks with Qwen3-VL-8B-Instruct. Experiments show that both components matter: replacing pairwise token contrast with a single-view signal underperforms our method, while adaptive trajectory confidence adds further improvement.

Figures & tables

Figure 1: Overview of PACE. A periodically refreshed teacher scores each sampled response under four counterfactual views (correct/incorrect hints $\times$ full/null vision). Pairwise token contrast yields the token contrast $c_{i,t}$ ; adaptive trajectory confidence uses the teacher’s batch-normalized confidence on each response to set the mixing rate $\lambda_{i}$ , i.e., how far $c_{i,t}$ redistributes credit among the tokens of each response.

Figure 2: The four teacher views of pairwise token contrast.

Model	Method	GSM8K-V	LogicVista	MMMU	MathVision	MathVista	Average
Qwen3-VL-8B Instruct	Base	22.97	54.36	61.33	45.79	76.00	52.09
	GRPO	26.61	58.84	65.89	47.11	75.70	54.83 ( $\uparrow$ 2.74)
	PAPO	25.47	59.73	65.44	50.03	76.10	55.36 ( $\uparrow$ 3.27)
	VPPO	26.99	61.52	67.67	52.80	77.30	57.25 ( $\uparrow$ 5.16)
	OPSD	23.05	54.36	61.67	42.86	75.20	51.43 ( $\downarrow$ 0.66)
	RLSD	26.38	57.72	66.22	49.90	76.70	55.38 ( $\uparrow$ 3.29)

Table 1: Accuracy (%) (mean@8) on five multimodal reasoning benchmarks. Parenthesized values denote percentage-point change from Base. Bold indicates best results; underline indicates second best.

Figure 3: Ablations on Qwen3-VL-8B-Instruct. (a) Contrast coefficients

(\alpha,\beta)

with the adaptive mixing rate

\lambda_{i}

. (b) Fixed mixing rate

\lambda

. (c) Per-benchmark accuracy of fixed

\lambda

and PACE.

(a) Pairwise token contrast	Fixed $\lambda$	Adaptive $\lambda_{i}$
Vision gap (no hint contrast)	51.05	–
Hint contrast, full vision	57.06	57.72
Hint contrast, full $-$ null vision	57.68	57.34
Hint contrast, full $+$ null vision	57.23	58.47
(b) Adaptive traj. confidence	$\lambda$	Avg.
$S_{\mathrm{mean}}$ (mean-based)	[0.10, 0.65]	57.04

Table 2: Ablation results (average accuracy, %).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Value
Training data	MMFineReason-123K
Optimization steps	200
Batch size	256 prompts
Rollouts per prompt	8
Actor global batch size	256
Rollout temperature	1.0

Appendix

Table 3: Training configuration of PACE and the methods trained under the shared setup.

Configuration	Value
Training data	MMFineReason-123K
Optimization steps	200
Batch size	32
Optimizer	AdamW
Learning rate	$5\times 10^{-6}$
Warmup ratio	0

Appendix

Table 4: OPSD baseline configuration with the ms-swift GKD trainer.

Figure 4: Training dynamics of PACE, RLSD, and GRPO on Qwen3-VL-8B-Instruct over the shared 200-step budget. Top: overall training reward, i.e. the fraction of sampled responses passing the verifier. Bottom: mean response length in tokens. Light traces show raw values and bold curves show smoothed trends.

Figure 5: Teacher signals of PACE training on Qwen3-VL-8B-Instruct. (a) Vision gap and hint contrasts (full-vision and null-vision). Inset: zoomed view of the hint contrasts. (b) Mixing rate

\lambda_{i}

(median and P10–P90 band) and raw confidence

S_{i}

Figure 6: Per-step training and generation time across RLVR methods on an 8

\times

NVIDIA H200 GPU cluster. (a) Total step time. (b) Generation time.

Model	Method	GSM8K-V	LogicVista	MMMU	MathVision	MathVista	Average
Qwen3-VL-8B Instruct	Base	22.97 $\pm$ 0.63	54.36 $\pm$ 1.11	61.33 $\pm$ 0.68	45.79 $\pm$ 0.44	76.00 $\pm$ 0.72	52.09
	GRPO	26.61 $\pm$ 0.70	58.84 $\pm$ 1.24	65.89 $\pm$ 1.18	47.11 $\pm$ 0.57	75.70 $\pm$ 0.96	54.83 ( $\uparrow$ 2.74)
	PAPO	25.47 $\pm$ 0.64	59.73 $\pm$ 1.10	65.44 $\pm$ 0.73	50.03 $\pm$ 0.47	76.10 $\pm$ 0.74	55.36 ( $\uparrow$ 3.27)
	VPPO	26.99 $\pm$ 0.46	61.52 $\pm$ 0.94	67.67 $\pm$ 0.66	52.80 $\pm$ 0.40	77.30 $\pm$ 0.69	57.25 ( $\uparrow$ 5.16)
	OPSD	23.05 $\pm$ 0.76	54.36 $\pm$ 1.04	61.67 $\pm$ 0.70	42.86 $\pm$ 0.60	75.20 $\pm$ 0.48	51.43 ( $\downarrow$ 0.66)
	RLSD	26.38 $\pm$ 0.27	57.72 $\pm$ 1.26	66.22 $\pm$ 0.98	49.90 $\pm$ 0.49	76.70 $\pm$ 0.73	55.38 ( $\uparrow$ 3.29)

Appendix

Table 5: Accuracy (%) (mean@8) on five multimodal reasoning benchmarks with standard deviation.

Variant	GSM8K-V	LogicVista	MMMU	MathVision	MathVista	Average
PACE	28.13 $\pm$ 0.52	63.31 $\pm$ 1.52	69.00 $\pm$ 0.87	54.01 $\pm$ 0.42	77.90 $\pm$ 0.83	58.47
Token contrast, fixed $\lambda=0.5$
Vision gap (no hint contrast)	24.03 $\pm$ 0.70	55.26 $\pm$ 1.53	61.22 $\pm$ 1.10	40.95 $\pm$ 0.41	73.80 $\pm$ 0.49	51.05
Hint contrast, full vision	26.91 $\pm$ 0.70	61.97 $\pm$ 1.33	65.56 $\pm$ 1.09	53.36 $\pm$ 0.71	77.50 $\pm$ 0.57	57.06
Hint contrast, full $-$ null vision	28.28 $\pm$ 0.58	60.63 $\pm$ 1.44	67.33 $\pm$ 1.07	53.36 $\pm$ 0.47	78.80 $\pm$ 0.62	57.68
Hint contrast, full $+$ null vision	26.23 $\pm$ 0.85	61.07 $\pm$ 1.33	66.67 $\pm$ 0.83	53.06 $\pm$ 0.73	79.10 $\pm$ 0.90	57.23

Appendix

Table 6: Per-benchmark accuracy (%) of all ablation configurations (mean@8) with standard deviation.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 472 papers

Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness
Apr 20, 2026Mengzhao Jia, Zhihan Zhang, Meng JiangMultimodal Reasoning Multimodal Reward Modeling
University of Notre Dame
Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
Mar 26, 2026Jinda Lu, Junkang Wu, Jinghan Li +6RL for VLMs VLM Reasoning
University of Science and Technology of China · Peking University · Independent Researcher
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
Sep 30, 2026Zhihan Zhang, Lizi LiaoToken-Level Credit Assignment Multimodal Large Language Models
Singapore Management University Singapore, Singapore

Apr 20, 2026J/K move · Enter open · S save

Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness

Mengzhao Jia, Zhihan Zhang, Meng Jiang

University of Notre Dame

Multimodal Reasoning Multimodal Reward Modeling Reinforcement Learning with Verifiable Rewards Process Supervision

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.

Abstract

RL for VLMs

Explore similar work

Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness

Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation