Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual evidence, and better preservation of specialized expertise in the unified student. All code and data are available at https://github.com/NEUIR/SCOPED.
Figures & tables
Figure 1: Overview Architecture of SCOPED.
Figure 2: Cross-Capability Interference in Mix-RL. (a) Average training reward of independent capability RL and Mix-RL. (b) Alignment of shared Mix-RL updates with capability-specific expert directions. (c) Pairwise gradient cosine similarity across Mix-RL training stages.
Figure 3: Analysis of Skill-Augmented Expert Supervision. (a) Capability retention and alignment, macro-averaged across four capabilities. (b) Within-capability trajectory embedding similarity with and without skill conditioning. (c) Layer-wise attention hit rate on task-relevant visual regions with and without skill conditioning.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Capability
Data Sources
# Samples
Perception & Grounding
PixMo-Count, Orsta-Data-47K
9,499
OCR & Chart
SlideVQA, Orsta-Data-47K
9,500
Symbolic Reasoning
We-Math 2.0 Pro, Orsta-Data-47K
9,500
Multi-Image Reasoning
Mantis-Instruct, M4-Instruct
9,500
Total
–
37,999
Appendix
Table 4: Statistics of the capability-specific reinforcement learning data. The reported counts correspond to the samples retained after preprocessing and filtering.
Hyperparameter
Value
Training epochs
3
Rollout batch size
128
Actor global batch size
64
Responses per prompt
8
Maximum prompt length
8,192
Maximum response length
8,192
Appendix
Table 5: Hyperparameters for capability-specialized expert training and skill-decoupled on-policy expert distillation.
Figure 4: Reward-based update alignment in Mix-RL and independent capability RL. (a) Alignment of shared Mix-RL updates with capability-specific optimization directions. (b) Alignment of independently trained experts with their respective capability directions.
Figure 5: Effectiveness of skill-decoupled expert supervision. SCOPED achieves stronger domain expert capability retention and alignment than alternative integration methods, while matched domain skills provide more expert-aligned guidance than random skills.
Figure 6: Qualitative visualization of skill-induced attention reallocation. Green boxes indicate target objects. Red and blue in the attention-change map indicate increased and decreased attention after skill conditioning.
Figure 7: Qualitative effect of skill-augmented expert supervision. Compared with SCOPED w/o Skill Augmentation, SCOPED strengthens responses supporting the correct answer while suppressing those associated with the erroneous trajectory. Blue indicates enhanced responses, while red indicates suppressed responses.
Figure 8: Case study of PixMo-Count.
Figure 9: Case study of MUIRBench.
Figure 10: Case study of MMSI-Bench.
Figure 11: Prompt used by the optimized capability expert to extract a reusable self-check principle from a successful trajectory.
Figure 13: Prompt used by the optimized capability expert to induce a domain-level skill from curated trajectory-derived evidence. The resulting skill contains shared guidance together with conditionally activated task-specific checks.
Figure 15: Extracted Capability skill for fine-grained perception.
Figure 16: Extracted Capability skill for OCR, document, and chart understanding.
Figure 17: Extracted Capability skill for multimodal symbolic reasoning.
Figure 18: Extracted Capability skill for multi-image reasoning.
On-policy knowledge distillation has proven effective for language models, yet its application to vision-language models (VLMs) remains underexplored. We observe that standard on-policy distillation can improve a student's output quality while failing to strengthen its reliance on visual input: on vision-critical tokens, the student's predictions remain largely unchanged whether or not fine-grained visual detail is present, even though the teacher's predictions depend heavily on it.To make this difference observable, we introduce visual advantage (VA), the token-level log-probability difference when the teacher scores a student-generated rollout with versus without access to fine-grained visual detail. VA is concentrated in a small minority of tokens, and these high-VA tokens are the ones that actually carry the visual supervision signal. This motivates a distillation objective that treats them differently from language scaffolding, so their contribution is not diluted by the abundant surrounding language tokens.We propose Visual-Advantage On-Policy Distillation (VA-OPD), which uses VA at two granularities: rollout-level reweighting by trajectory-averaged VA, and token-level KL averaged within high-VA and low-VA groups separately. We train on two math datasets (Geometry3K and ViRL39K) and evaluate on eight benchmarks covering both mathematical reasoning and visual understanding, across three teacher sizes (4B, 8B, and 32B) on the Qwen3-VL family. VA-OPD improves over standard on-policy distillation on every benchmark, with the gain growing monotonically along both the teacher-size and data-scale axes, suggesting that these factors compound consistently.
Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1
Vision-Language Models (VLMs) frequently suffer from visual perception errors and hallucinations that compromise answer accuracy in complex reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.
Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5
School of Mathematics, Tianjin University, Tianjin, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Shanghai Advanced Institute of Finance (SAIFS), East China Normal University, Shanghai, China +4
RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.
Naibin Gu, Chenxu Yang, Qingyi Si +7
Institute of Information Engineering, CAS, 2School of Cyber Security, UCAS, 3JD.COM