NAS-PO: Native Attention-Strategy Policy Optimization for Vision-Language Models

Ke An; Shizhe Xiang; Yue Liu; Liang Chen; Pei Fu; Shaojie Zhang; Xiaoxing Hu; Yupei Wang

cs.CVNov 1, 2026

NAS-PO: Native Attention-Strategy Policy Optimization for Vision-Language Models

Authors: Ke An, Shizhe Xiang, Yue Liu, Liang Chen, Pei Fu, Shaojie Zhang, Xiaoxing Hu, Yupei Wang

Organizations: Beijing Institute of Technology · Tianjin University · University of Chinese Academy of Sciences · Xiaomi Inc. ·

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an effective post-training paradigm for vision-language models (VLMs). Standard RLVR, however, assigns identical outcome-derived credit to all generated tokens regardless of their perceptual relevance. Recent perception-aware methods estimate visual reliance through image perturbation or masking. These indirect proxies require perturbation branches, while corruption sensitivity does not necessarily identify the visual evidence used for reasoning. We instead shape model perception through native attention, an intrinsic model signal. Rather than treating attention as a post-hoc explanation, we formulate it as a reward-anchored, token-aligned routing signal that encourages attention patterns supporting visually grounded reasoning. To this end, we propose NAS-PO, a Native Attention-Strategy Policy Optimization framework. NAS-PO analyzes native attention with a visual mass policy that quantifies visual allocation and a conditional visual policy that captures spatial routing over visual tokens, making the optimization well-posed. Based on this, Native Attention-Strategy Advantage Shaping (NAS-AS) draws on the visual-mass policy to redistribute signed policy-gradient credit for visually engaged rollouts. The Adaptive Attention-Coupled (AAC) loss then aligns the visual-mass policy on all response tokens and the conditional visual policy on visually active ones, consolidating the visual-routing patterns that drive successful trajectories. Experiments demonstrate that NAS-PO substantially surpasses leading RL-tuned VLMs across 7 out-of-domain reasoning and perception benchmarks, with its effectiveness consistently validated across 8B and 32B. Visual-Routing Sensitivity (VRS) analysis further indicates that the optimized attention prioritizes visually load-bearing regions.

Figures & tables

Figure 1: NAS-PO decomposes the native attention into two policies: visual-mass policy $q$ for whether to rely on the image, and conditional visual policy $\rho$ for where to attend. Native Attention-Strategy Advantage Shaping (NAS-AS) optimizes this routing at trajectory level and the Adaptive Attention-Coupled (AAC) loss at token level.

Figure 2: Overview of NAS-PO. Given an image–query pair $(I,\mathcal{Q})$ , the standard GRPO pipeline produces $G$ clean rollouts and verifier-derived group-relative advantages $A_{i}$ . Using the same clean rollouts, NAS-PO reads native attention $a_{i,t}(j)$ and losslessly decomposes its visual component into a visual-mass policy $q_{i,t}$ , which captures whether and how much a response token relies on the image, and a conditional visual policy $\rho_{i,t}$ , which captures where attention is routed over visual tokens. The visual-mass policy also provides visually active indicators $c_{i,t}$ shared by NAS-AS and the conditional branch of AAC. At the trajectory level, NAS-AS summarizes visual reliance into a bounded factor $\alpha_{i}$ and forms $\tilde{A}_{i}=\alpha_{i}A_{i}$ . At the token level, AAC aligns $q$ on all valid response tokens and $\rho$ on visually active ones under the unshaped positive-advantage gate $[A_{i}]_{+}$ . The shaped policy-gradient objective and the adaptively weighted AAC loss jointly update the policy.

Model	General Multimodal Reasoning						Vision-Dependent Multimodal Reasoning				Overall
Model	MMK12	Geo3k	MathVerse	MathVista	We-Math	AVG	MMMU-Pro	MathVerse V	LogicVista	AVG	Overall
OneThinker-8B	47.19	41.90	64.3	69.47	49.36	54.44	40.75	43.54	48.60	44.30	50.64
NoisyRollout-8B	68.56	56.23	72.17	68.45	77.02	68.49	40.76	69.73	52.39	54.29	63.16
PAPO-8B	68.85	55.39	73.96	68.95	77.21	68.87	42.42	70.25	54.11	55.59	63.89
VPPO-8B	75.30	58.98	76.98	73.08	75.51	71.97	41.78	74.68	52.43	56.30	66.09
CPPO-8B	68.90	58.38	75.55	69.26	77.67	69.95	42.39	71.95	53.1	55.81	64.65

Table 1: Main results (avg@8 acc

\%

) on general and more vision-dependent multimodal reasoning tasks. MathVerse V refers to the vision-centric subset of MathVerse. AVG and Overall are unweighted means over the displayed evaluation configurations.

Figure 3: Training dynamics on Qwen3-VL-8B. (a) Average Accuracy@8 across eight configurations from seven out-of-domain benchmarks. The dotted guides mark GRPO’s final accuracy and step 50, where NAS-PO has already exceeded it. (b) Reward on ViRL39K. Faint lines show per-step values and solid lines show 10-step running averages. NAS-PO leads at all evaluation checkpoints and nearly all reward steps, showing faster learning from the early stages and a sustained advantage over GRPO.

Model	General Multimodal Reasoning						Vision-Dependent Multimodal Reasoning				Overall
Model	MMK12	Geo3k	MathVerse	MathVista	We-Math	$\Delta_{\mathrm{rel}}^{\%}$	MMMU-Pro	MathVerse V	LogicVista	$\Delta_{\mathrm{rel}}^{\%}$	Score	$\Delta_{\mathrm{rel}}^{\%}$
Qwen3-VL-8B-Instruct	26.28	43.57	30.75	63.30	46.73	–	19.03	29.38	39.01	–	37.26	–
+ GRPO	68.50	55.57	72.89	71.76	76.78	–	40.78	69.18	52.83	–	63.54	–
+ GRPO & NAS-AS	76.09	58.67	77.10	74.61	77.00	$\uparrow$ 5.20	41.04	74.14	54.03	$\uparrow$ 3.94	66.59	$\uparrow$ 4.80
+ GRPO & AAC	77.37	59.94	77.31	75.18	77.50	$\uparrow$ 6.31	43.10	74.23	53.77	$\uparrow$ 5.10	67.30	$\uparrow$ 5.92
+ GRPO & NAS-PO	77.49	60.13	78.36	75.24	78.09	$\uparrow$ 6.89	43.78	75.61	55.60	$\uparrow$ 7.49	68.04	$\uparrow$ 7.08

Table 2: Detailed ablation of Native Attention-Strategy Advantage Shaping (NAS-AS) and Adaptive Attention-Coupled (AAC) loss. Each component improves GRPO on every evaluation configuration, and their combination yields the best result throughout.

\Delta_{\mathrm{rel}}

denotes the relative gain over GRPO for the corresponding aggregate.

Figure 4: Visual-Routing Sensitivity. (a) Dropping the Top-15% patches flips NAS-PO far more often than the baselines on all seven benchmarks ( $\uparrow$ better). (b) Dropping the Bottom-15% barely affects it ( $\downarrow$ better). (c) Their difference, VRS, remains highest for NAS-PO across every drop fraction $K$ , reaching $0.31$ at $K{=}15\%$ versus $0.13$ for GRPO. Together, the interventions show that NAS-PO’s attention ranking identifies visually load-bearing regions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$\delta$	General Multimodal Reasoning						Vision-Dependent Multimodal Reasoning				Overall
$\delta$	MMK12	Geo3k	MathVerse	MathVista	We-Math	AVG	MMMU-Pro	MathVerse V	LogicVista	AVG	Overall
$0$	77.11	58.52	77.23	72.26	77.71	72.57	42.64	73.70	54.25	56.86	66.68
$0.10$	77.06	59.23	78.51	74.65	75.94	73.08	42.96	74.47	54.05	57.16	67.11
$0.25$	77.49	60.13	78.36	75.24	78.09	73.86	43.78	75.61	55.60	58.33	68.04
$0.40$	77.25	59.02	77.73	72.20	77.39	72.72	42.45	74.42	54.58	57.15	66.88

Appendix

Table 3: Sensitivity of GRPO+NAS-PO to the drop quantile

\delta

on Qwen3-VL-8B (avg@8 acc

\%

). MathVerse V denotes the vision-centric subset of MathVerse. AVG and Overall are unweighted means over the displayed evaluation configurations. Best results are bolded.

Category	Setting
Base model	Qwen3-VL-8B/32B-Instruct
Training data	ViRL39K
Reward	Binary answer accuracy
Prompt	You first think through the reasoning process. Then, provide your final answer enclosed within \boxed{}
Policy optimizer	GRPO
Rollouts per prompt	$G=8$

Appendix

Table 4: Training and evaluation configuration for the reported NAS-PO runs.

Hyperparameter	Value
Use NAS-AS	Enabled
NAS-AS score source	Native old visual mass $m^{\mathrm{old}}$
$\beta_{\min}$	0.9
Coverage exponent $\gamma$	1.0
Excess-mass transform	Square root
Use AAC	Enabled

Appendix

Table 5: NAS-PO-specific hyperparameters.

Figure 5: Aggregate training dynamics. (a) Unweighted average Accuracy@8 over the eight evaluation configurations. (b) Mean verifier reward at all 202 training steps. Thin lines show raw values, and thick lines show a 10-step moving average. Both single-run trajectories show an early and persistent within-run advantage for NAS-PO over GRPO.

Metric	GRPO	NAS-PO
Final average Accuracy@8	63.54	68.04
Mean Accuracy@8 over checkpoints	61.76	65.32
First step reaching 63.54	80	50
Mean reward, steps 1–30	0.160	0.345
Mean reward, final 20 steps	0.693	0.835

Appendix

Table 6: Summary of training dynamics. Accuracy values are percentages.

Figure 6: Complete checkpoint-wise Accuracy@8 curves. The eight configurations correspond to the seven benchmarks in the main paper, with MathVerse and its vision-dependent subset shown separately. NAS-PO improves the final checkpoint on every configuration and leads on 161 of the 168 configuration–checkpoint pairs.

Method	General Multimodal Reasoning						Vision-Dependent Multimodal Reasoning				Overall
Method	MMK12	Geo3k	MathVerse	MathVista	We-Math	AVG	MMMU-Pro	MathVerse V	LogicVista	AVG	Overall
GRPO	0.94	18.48	10.07	15.41	15.71	12.12	26.94	14.08	6.23	15.75	13.48
PAPO	1.00	21.59	13.14	9.48	14.38	11.92	32.37	17.61	7.41	19.13	14.62
VPPO	5.58	41.76	18.47	18.17	16.35	20.07	39.04	27.10	10.08	25.41	22.07
NAS-PO	10.64	50.06	25.11	21.48	29.61	27.38	53.16	38.73	22.62	38.17	31.43

Appendix

Table 7: Visual-Routing Sensitivity at

K{=}15\%

. All entries are percentages, and AVG and Overall are unweighted means over the displayed evaluation configurations. VRS

=\mathrm{flip}_{\mathrm{top}}-\mathrm{flip}_{\mathrm{bot}}

on originally-correct rollouts (

\uparrow

better). NAS-PO is highest on every benchmark.

Figure 7: Geometry reasoning. NAS-PO correctly treats $\angle ACB=25^{\circ}$ as an inscribed angle and obtains $\angle BAO=65^{\circ}$ (D). GRPO misidentifies the marked angle and returns $55^{\circ}$ (B).

Figure 8: Spatial reasoning. NAS-PO correctly tracks how the shaded semicircles meet across adjacent cube faces and selects C. GRPO reconstructs the folded net incorrectly and selects B.

Figure 9: Science reasoning. NAS-PO identifies the lacunae characteristic of bone tissue and selects D. GRPO interprets the image as loose connective tissue and selects F.

Figure 10: Chart reasoning. NAS-PO follows the solid Group A curve to the week-3 value of approximately $400$ grams. GRPO reads the value as approximately $300$ grams.

Figure 11: Residual perception error. Both NAS-PO and GRPO misinterpret the phase of the plotted sine function and select A instead of the ground-truth option D.

Figure 12: Residual reasoning error. NAS-PO first derives the correct radius of $5$ cm but then overrides it and answers $2.5$ cm. GRPO also answers $2.5$ cm. The case separates successful visual extraction from a later reasoning failure.

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 488 papers

Oct 15, 2026

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty is that visual task failures can arise from different sources: the same benchmark may be limited by perception for one model and by reasoning for another. We introduce Perception and Reasoning Attribution (PARA), which measures the remaining perceptual and reasoning headroom of each model--benchmark pair by comparing how many current errors are recovered with more accurate visual information versus a stronger reasoner. Across 4 vision-language models and 17 multimodal benchmarks, PARA broadly agrees with the expected characteristics of existing benchmarks while revealing substantial model-specific variation. More importantly, it shows that RL gains become smaller as perceptual headroom increases, but larger when reasoning explains more of the recoverable errors, suggesting that current RL exploits reasoning headroom more effectively than perceptual headroom. Motivated by this gap, we analyze how existing RL objectives affect visual perception and find that reward- and token-level signals influence visual attention only indirectly. We therefore propose GRAD, a group-relative attention distillation objective that uses the best rollout in each group to directly guide the visual attention of weaker rollouts. GRAD consistently improves performance across two model scales and seven benchmarks, and further complements existing RL methods.

Zeyuan Yang, Zhuohao Chen, Jiaben Chen +3

UMass Amherst · UIUC · MIT

Attention-Guided Distillation RL for VLMs Vision-Language Models VLM Reasoning Reasoning for VLMs VLM Evaluation+1

Ask PDF

Apr 2, 2026

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

While Reinforcement Learning from Verifiable Rewards (RLVR) has advanced reasoning in Large Vision-Language Models (LVLMs), prevailing frameworks suffer from a foundational methodological flaw: by distributing identical advantages across all generated tokens, these methods inherently dilute the learning signals essential for optimizing the critical, visually-grounded steps of multimodal reasoning. To bridge this gap, we formulate Token Visual Dependency, quantifying the causal information gain of visual inputs via the Kullback-Leibler (KL) divergence between visual-conditioned and text-only predictive distributions. Revealing that this dependency is highly sparse and semantically pivotal, we introduce Perception-Grounded Policy Optimization (PGPO), which is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level. Through a threshold-gated, mass-conserving mechanism, PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors. Extensive experiments based on the Qwen2.5-VL series across seven challenging multimodal reasoning benchmarks demonstrate that PGPO boosts models by 18.7% on average. Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning. Code will be released on https://github.com/Yzk1114/PGPO.

Zekai Ye, Qiming Li, Xiaocheng Feng +6

Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd

RL for VLMs Large Vision-Language Models VLM Reasoning Token-Level Credit Assignment

Ask PDF

Apr 27, 2026

Improving Vision-language Models with Perception-centric Process Reward Models

Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-language models (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL, highlighting the promise of perception-centric supervision as a general-purpose strategy. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released at https://github.com/RUCAIBox/Perceval.

Yingqian Min, Kun Zhou, Yifan Li +6

Gaoling School of Artificial Intelligence, Renmin University of China. · Bytedance. · University of California, San Diego. +1

Process Reward Models Object Hallucination in VLMs VLM Reasoning Visually Grounded Reasoning RL for VLMs

Ask PDF

Abstract

RL for VLMs

Explore similar work

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

Improving Vision-language Models with Perception-centric Process Reward Models