Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought

Yunheng Li; Hangyi Kuang; Hengrui Zhang; Jiangxia Cao; Zhaojie Liu; Qibin Hou; Ming-Ming Cheng

cs.CVMar 24, 2026

Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought

Authors: Yunheng Li, Hangyi Kuang, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou, Ming-Ming Cheng

Organizations: VCIP, School of Computer Science, Nankai University · Kuaishou Technology

Abstract

Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories that interleave perceptual grounding with multi-step inference. However, existing Reinforcement Learning with Verifiable Rewards (RLVR) methods typically optimize reasoning at a coarse granularity, treating CoT uniformly without distinguishing their varying degrees of visual grounding. In this work, we conduct a token-level analysis of multimodal reasoning trajectories and show that successful reasoning is characterized by structured token dynamics reflecting both perceptual grounding and exploratory inference. Building upon this analysis, we propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior from hidden state similarity and integrates it with token entropy through a smooth gating mechanism to produce token-level advantages. PEPO integrates seamlessly with existing RLVR frameworks such as GRPO and DAPO, requiring neither additional supervision nor auxiliary branches. Extensive experiments across diverse multimodal benchmarks demonstrate consistent and robust improvements over strong RL baselines, spanning geometry reasoning, visual grounding, visual puzzle solving, and few-shot classification, while maintaining stable training dynamics. Code: https://github.com/xzxxntxdy/PEPO

Figures & tables

Figure 1: Overview of PEPO. (a) Effective multimodal reasoning arises from the complementarity between perception and exploration. Abbr. Exp.: Exploration-only, Per.: Perception-only, P+E: Perception + Exploration. (b) Unlike traditional sequence-level optimization with uniform advantages, PEPO reweights tokens using a perception prior from visual similarity and entropy via a smooth gate, producing fine-grained token-level advantages. (c) When integrated with GRPO or DAPO, PEPO consistently improves performance across diverse benchmarks.

Panel of Figure 2 — Figure 2: Distributions of different visual similarity metrics comparing correct and incorrect responses. (a) The global similarity ( $M_{\text{glob}}$ ) across all tokens, where correct responses exhibit a clear rightward shift. (b) The top- $K$ similarity ( $M_{\text{high}}$ ), where the correct-response peak also moves right. (c) The bottom- $K$ similarity ( $M_{\text{low}}$ ), where the shift is negligible. Together, these results show that reasoning correctness is characterized by a subset of visual-grounded tokens.

Figure 3: Token-level analysis of visual similarity and entropy. (a) High visual similarity tokens exhibit larger hidden-state shifts under image removal than high entropy tokens. (b) Word cloud of high entropy tokens and (c) word cloud of high visual similarity tokens, illustrating reasoning-related and perceptual terms.

Figure 4: Framework of PEPO. During response generation, the layer-wise hidden states of response tokens and vision tokens are extracted, along with the output logits. For each response token, visual similarity and entropy are computed, and the centered sum of their normalized values is passed through a smooth gating function to produce token-wise weights that modulate the advantages for PEPO updates.

Qwen2.5-VL-3B-Instruct
Method	Geometry3K val	Geometry3K test	MathVista mini	MathVerse mini	LogicVista	Avg
Base (zero-shot)	11.58	14.25	50.19	37.66	32.22	29.18
High-Entropy RL	15.58	18.26	50.69	39.84	32.59	31.39
GRPO	19.00	23.79	51.56	40.54	28.30	32.64
PEPO G (Ours)	21.91	27.27	54.45	45.42	34.45	36.70 (+4.06)
DAPO	22.63	27.00	53.68	44.44	33.13	36.18

Table 1: Results on Geometry3K validation/test splits and out-of-domain benchmarks, including MathVista, MathVerse, and LogicVista.

Table 6

Method	Qwen2.5-VL-3B-Instruct			InternVL3-2B-Instruct
Method	PuzzleVQA	AlgoPuzzleVQA	Avg	PuzzleVQA	AlgoPuzzleVQA	Avg
Base (zero-shot)	31.80	21.72	26.76	31.80	23.00	27.40
High-Entropy RL	35.00	24.61	29.81	32.60	25.22	28.91
GRPO	43.20	25.44	34.32	43.20	26.17	34.69
PEPO G (Ours)	45.00	26.94	35.97 (+1.65)	45.20	27.22	36.21 (+1.52)
DAPO	44.60	25.33	34.97	44.80	27.72	36.26

Table 4: Accuracy (%) on PuzzleVQA and out-of-domain AlgoPuzzleVQA datasets.

Method	Geometry3K test	MathVista	We-Math	MathVerse	LogicVista	Counting	MMMU-Pro	Avg
GRPO	28.72	59.34	58.90	55.25	38.14	55.81	25.66	45.97
PAPO G	30.95	61.38	60.09	57.14	38.67	62.56	27.11	48.27
PEPO G (Ours)	29.85	63.48	60.98	56.49	39.85	70.75	27.47	49.84 (+3.87)

Table 5: Comparison of scaling performance among GRPO, PAPO, and PEPO on Qwen2.5-VL-3B-Instruct trained with ViRL39K, evaluated across multiple reasoning benchmarks using the avg@8 metric. GRPO and PAPO results are taken directly from [ 67 ] .

Metric	Method	Geometry3K	PuzzleVQA	RefCOCO	FGVC Aircraft
Iter/s	GRPO	0.022	0.066	0.080	0.071
Iter/s	PEPO G	0.022	0.074	0.084	0.083
Mean length	GRPO	273	104	77	92
Mean length	PEPO G	270	85	64	77
$\rho$	PEPO G	0.0039	0.0065	0.0061	0.0048

Table 6: Training efficiency comparison between GRPO and PEPO G . We report throughput (iterations per second), mean response length, and the step-level computational overhead ratio

\rho

across datasets. The overhead ratio

\rho

measures the proportion of additional weight computation time within a full RL update step.

Table 10

Table 11

Figure 5: Qualitative comparison on Geometry3K, MathVerse, and LISA datasets. The GRPO-trained model exhibits perception failures and inconsistent reasoning, leading to incorrect answers. In contrast, the PEPO-trained model generates coherent, visually grounded reasoning chains that produce correct results, demonstrating the effectiveness of PEPO in enhancing multimodal reasoning.

Figure 6: Training curves on FGVC Aircraft (4-shot). We compare GRPO and PEPO G in terms of training reward, mean response length, mean visual similarity, and mean entropy.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Task(s)	Value
Shared RL configuration
Optimizer	1–5	AdamW
Train type	1–5	full
Precision	1–5	bfloat16
Gradient checkpointing	1–5	Enabled
Responses per question	1–5	8

Appendix

Table 1: Key hyperparameters across the five tasks.

Reward Design
Task(s)	Design
1, 3, 4	Format reward + Accuracy reward
2	Format reward + IoU reward
5	0.1 $\times$ Format reward + 0.9 $\times$ Accuracy reward
Prompt Design
1	First output the thinking process in <think> … </think> tags and then output the final answer in <answer> … </answer> tags.

Appendix

Table 2: Prompt and reward design across the five tasks.

$\alpha$	Geometry3K val	Geometry3K test	MathVista mini	MathVerse mini	LogicVista	Avg
0.00 (Perception-only)	21.07	26.73	52.81	45.31	34.93	36.17
0.02	20.60	25.87	53.88	44.99	34.73	36.01
0.05 (Default)	21.91	27.27	54.45	45.42	34.45	36.70
0.10	22.80	26.81	53.58	44.12	34.26	36.31
0.15	21.45	26.52	53.54	43.99	34.62	36.02

Appendix

Table 3: Ablation on gate strength

\alpha

of PEPO on geometry and logic reasoning benchmarks using the Qwen2.5-VL-3B-Instruct model.

Measure	Geometry3K val	Geometry3K test	FGVC Aircraft (4-shot)
L1 distance	19.00	24.50	60.85
L2 distance	20.90	26.08	67.33
Cosine similarity (Default)	22.80	26.81	75.79

Appendix

Table 4: Ablation on perception prior measures used to compute visual similarity.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models
Apr 2, 2026Zekai Ye, Qiming Li, Xiaocheng Feng +6RL for VLMs Large Vision-Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd
PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
Jun 7, 2026Qiming Li, Tianlun Li, Xiaolong Cheng +5Token-Level Credit Assignment VLM Reasoning
Amap CV Lab, Alibaba Group · Peking University
Structured Role-Aware Policy Optimization for Multimodal Reasoning
May 8, 2026Bingqing Jiang, Difan ZouToken-Level Credit Assignment Multimodal Reasoning
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong

Apr 2, 2026J/K move · Enter open · S save

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Zekai Ye, Qiming Li, Xiaocheng Feng, Ruihan Chen+5

Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd

RL for VLMs Large Vision-Language Models VLM Reasoning Token-Level Credit Assignment

Open paper AskPDF arXiv

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

Abstract

VLM Reasoning

Explore similar work

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Structured Role-Aware Policy Optimization for Multimodal Reasoning