PEAR: Adaptive Advantage Reshaping for Perception and Exploration in Large Vision-Language Models

Junxin Wang; Yingfeng Luo; Xiangchen Song; Weiqiao Shan; Runsong Zhao; Xinyu Liu; Shunjie Xing; Yifu Huo

cs.AIOct 23, 2026

PEAR: Adaptive Advantage Reshaping for Perception and Exploration in Large Vision-Language Models

Authors: Junxin Wang, Yingfeng Luo, Xiangchen Song, Weiqiao Shan, Runsong Zhao, Xinyu Liu, Shunjie Xing, Yifu Huo,

Organizations: School of Computer Science and Engineering, Northeastern University, Shenyang, China · University of Michigan, Ann Arbor, US · Kunming University of Science and Technology, Kunming, China · NiuTrans Research, Shenyang, China

Abstract

Effective multimodal reasoning requires a model to reliably utilize visual information while sufficiently exploring diverse reasoning paths. To achieve this goal, prior work combines visual dependency and token entropy to reshape sequence-level advantages, but scaling the contribution of token entropy by visual dependency can leave high-entropy tokens with low visual dependency underweighted, despite their important role in reasoning exploration. To address this limitation, we propose PEAR, an adaptive advantage reshaping method for improving perception and exploration in multimodal reasoning. PEAR retains visual dependency as the base signal and adds an entropy-based exploration gain only at high-entropy positions, with lower visual dependency leading to a larger additional gain. The fused signal is then mapped to positive token-level weights that reshape the sequence-level advantage. Experiments on Qwen2.5-VL-3B/7B show that PEAR achieves the highest average performance among evaluated baselines across multimodal reasoning benchmarks, alongside leading results on textual reasoning and general multimodal benchmarks.

Figures & tables

Figure 1: Comparison of token-level credit assignment strategies. Left : GRPO assigns the same credit to all tokens within a response, despite their different contributions to reasoning. Prior work fuses visual dependency and token entropy multiplicatively, limiting the influence of high entropy when visual dependency is low ( Li et al., 2026b ) . PEAR retains visual dependency as the basis for credit assignment and adaptively incorporates an exploration gain at high-entropy positions. Right : In the geometry example, PEAR gives greater credit to the reasoning transitions “since”, “Therefore”, and “thus” while continuing to emphasize vision-related tokens such as “OAP” and “BAP”.

Figure 2: Overview of the proposed PEAR framework. Here, $o_{i,t}$ denotes the $t$ -th token in response $i$ , while $P_{i,t}$ and $P^{\prime}_{i,t}$ denote the predictive distributions with and without image input, respectively. The Fusion Module derives visual dependency by comparing these distributions and estimates token entropy from $P_{i,t}$ . The entropy signal is then filtered to produce exploration gains, which are adaptively modulated by visual dependency and added to the visual base to obtain normalized token weights. Advantage Reshape uses these weights to transform the shared sequence-level advantage into token-level advantages for policy update.

Model	Geo3K	DynaMath	We-Math	MathVision	MathVerse	LogicVista	MMMU-Pro	MathVerse V	Avg.
MM-Eureka-7B	40.77	49.53	63.78	28.49	66.44	46.92	23.42	61.76	47.64
VL-Rethinker-7B	39.45	52.38	67.96	27.91	67.86	45.41	37.05	64.58	50.33
NoisyRollout-7B	52.12	53.65	63.27	27.95	55.97	46.47	33.49	60.04	49.12
R1-ShareVL-7B	42.12	53.27	69.48	27.95	67.40	46.64	35.01	63.69	50.70
Vision-SR1-7B	36.71	51.45	63.05	26.62	60.33	44.85	35.63	56.06	46.84
Qwen2.5-VL-7B	37.52	48.06	46.08	25.25	39.21	41.30	25.56	34.50	37.19

Table 1: Performance comparison on eight multimodal reasoning benchmarks using the Avg@8 metric. The best and second-best results within each model scale are highlighted in bold and underlined, respectively.

Model	Textual Reasoning					General Multimodal
Model	AIME24	MATH-500	BBH	MMLU-Pro	Avg.	MMStar	MMMU	MME (%)	POPE(F1)	Avg.
MM-Eureka-7B	6.46	64.00	61.99	52.01	46.12	57.70	56.11	83.36	85.80	70.74
VL-Rethinker-7B	4.38	60.40	62.99	52.40	45.04	60.90	56.55	83.06	85.80	71.58
NoisyRollout-7B	6.04	61.40	59.13	52.58	44.79	61.50	56.00	83.12	86.50	71.78
R1-ShareVL-7B	3.75	56.40	64.21	52.41	44.19	61.30	55.11	83.46	86.60	71.62
Vision-SR1-7B	5.00	56.80	62.92	52.19	44.23	33.70	55.33	82.70	85.90	64.41

Table 2: Performance comparison on textual reasoning and general multimodal ability benchmarks. The best and second-best results within each model scale are highlighted in bold and underlined, respectively. MME scores are normalized by the maximum score of 2800 and reported as percentages. PEPO-3B results on these benchmarks are not reported in the original work ( Li et al., 2026b ) and are therefore not included in this table.

Setting	Geo3K	DynaMath	We-Math	MathVision	MathVerse	LogicVista	MMMU-Pro	MathVerse V	Avg.
GRPO	29.97	44.21	59.06	23.39	56.10	38.76	26.16	52.90	41.32
Individual Signals
Visual Base Only	33.46	46.83	58.69	25.51	54.40	41.08	28.11	52.05	42.52
High-Entropy Gain Only	34.94	47.80	63.74	25.46	60.36	41.92	27.69	57.23	44.89
Component Ablation
w/o Visual Modulation	36.29	47.93	64.35	26.36	61.26	41.81	27.78	58.57	45.54

Table 3: Ablation studies of PEAR on Qwen2.5-VL-3B. All experiments are conducted under the same training setting unless otherwise specified. The best and second-best results among all evaluated settings are highlighted in bold and underlined, respectively. For each hyperparameter sweep, the other hyperparameter is fixed at its default value, with

\tau_{h}=0.5

and

\beta=1.2

Figure 3: Token word clouds on MathVerse: (a) high visual dependency; (b) low visual dependency and high entropy. Font size indicates frequency.

Figure 4: Training dynamics of Qwen2.5-VL-7B during policy optimization. PEAR maintains its exploration capability while continuing to improve training accuracy.

Method	Time (h)	Multimodal	Textual	General
Method	Time (h)	Reasoning	Reasoning	Multimodal
GRPO	14.9 (+0.0%)	50.60	45.13	71.94
DAPO	21.5 (+44.3%)	46.77	41.42	71.61
PEAR	18.1 (+21.5%)	52.70	46.21	72.44

Table 4: Training time on eight NVIDIA A100 GPUs and average performance of Qwen2.5-VL-7B under the same experimental setup.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	3B	7B
Training data	ViRL39K	ViRL39K
Training epochs	2	2
Hardware	$8\times$ A100	$8\times$ A100
Optimizer / precision	AdamW / BF16	AdamW / BF16
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$
Weight decay	$1\times 10^{-2}$	$1\times 10^{-2}$

Appendix

Table 5: Training configurations

Figure 5: Training reward (left) and entropy loss (right) across optimization steps for the 7B stabilization settings in Table 6 . Faint curves show raw values; solid curves show smoothed trends.

Setting	Geo3K	We-Math	MathVerse	LogicVista	MMMU-Pro	MathVerse V	Avg.
PEPO	39.62	68.82	66.11	46.20	35.33	62.17	53.04
PEAR ( $\rho=1$ )	42.45	68.45	68.65	46.42	36.35	65.42	54.62
+ Cosine Annealing	43.24	69.65	70.01	47.37	36.72	65.99	55.50
+ Entropy Penalty	45.13	69.65	70.28	47.96	36.55	66.50	56.01

Appendix

Table 6: Comparison with PEPO and ablation of PEAR’s 7B stabilization designs. Best results are highlighted in bold.

Benchmark	Responses	Temp. / top- $p$ / top- $k$	Model length	Output length
MATH-500	1	$1.0/0.95/50$	16,384	16,384
AIME24	16	$1.0/0.95/50$	16,384	16,384
BBH	1	$0/1.0/-1$	8,192	2,048
MMLU-Pro	1	$0/1.0/-1$	8,192	2,048

Appendix

Table 7: Decoding settings for textual reasoning evaluation.

Setting	Value
Responses per example	1
Temperature / top- $p$	$0/1.0$
Maximum model / output length	$32{,}768/2{,}048$
Minimum / maximum image pixels	$1{,}003{,}520/12{,}845{,}056$
Prompt and evaluator	Dataset-specific VLMEvalKit defaults

Appendix

Table 8: Common settings for general multimodal evaluation.

Reasoning for VLMs

See all 64 papers

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
Sep 30, 2026Zhihan Zhang, Lizi LiaoToken-Level Credit Assignment Multimodal Large Language Models
Singapore Management University Singapore, Singapore
PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
Jun 7, 2026Qiming Li, Tianlun Li, Xiaolong Cheng +5Token-Level Credit Assignment VLM Reasoning
Amap CV Lab, Alibaba Group · Peking University
Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
Mar 24, 2026Yunheng Li, Hangyi Kuang, Hengrui Zhang +4VLM Reasoning Multimodal CoT Reasoning
VCIP, School of Computer Science, Nankai University · Kuaishou Technology

Sep 30, 2026J/K move · Enter open · S save

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation

Zhihan Zhang, Lizi Liao

Singapore Management University Singapore, Singapore

Token-Level Credit Assignment Multimodal Large Language Models Multimodal Reasoning Reinforcement Learning with Verifiable Rewards RL for VLMs

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet existing frameworks rely on coarse, sequence-level reward signals that lack the fine-grained supervision over the visually-grounded steps within a multimodal reasoning chain. We investigate this gap through the lens of two token-level metrics: visual dependency (i.e. how much a token's prediction relies on the input image features) and predictive entropy. Our empirical analysis reveals two key findings: (1) correct reasoning chains exhibit a markedly sharper entropy reduction as visual grounding intensifies, compared to incorrect ones; (2) pivotal tokens, those whose misprediction triggers reasoning collapse, are statistical outliers in the joint distribution of visual dependency and predictive entropy derived from correct chains. Motivated by these findings, we propose token-level perception-grounded advantage estimation (TPAE), which estimates token-level advantages by measuring each token's statistical consistency with the vision-entropy patterns of correct rollouts. TPAE leverages this granular score to modulate the sequence-level advantage, producing a fine-grained supervision signal that can be integrated into various RLVR frameworks. Extensive experiments on seven benchmarks show that TPAE consistently outperforms leading strong baselines, yielding more stable and efficient optimization for multimodal reasoning. The code is publicly available at https://github.com/Zhihan72/TPAE.

Abstract

Reasoning for VLMs

Explore similar work

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought