ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL

Xingyu Lu; Jinpeng Wang; YiFan Zhang; Shijie Ma; Xiao Hu; Tianke Zhang; Haonan fan; Kaiyu Jiang

cs.LGFeb 26, 2026

ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL

Authors: Xingyu Lu, Jinpeng Wang, YiFan Zhang, Shijie Ma, Xiao Hu, Tianke Zhang, Haonan fan, Kaiyu Jiang,

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China ·

Abstract

We propose ContextRL, a novel framework that leverages context augmentation to overcome these bottlenecks. Specifically, to enhance Identifiability, we provide the reward model with full reference solutions as context, enabling fine-grained process verification to filter out false positives (samples with the right answer but low-quality reasoning process). To improve Reachability, we introduce a multi-turn sampling strategy where the reward model generates mistake reports for failed attempts, guiding the policy to "recover" correct responses from previously all-negative groups. Experimental results on 11 perception and reasoning benchmarks show that ContextRL significantly improves knowledge discovery efficiency. Notably, ContextRL enables the Qwen3-VL-8B model to achieve performance comparable to the 32B model, outperforming standard RLVR baselines by a large margin while effectively mitigating reward hacking. Our in-depth analysis reveals the significant potential of contextual information for improving reward model accuracy and document the widespread occurrence of reward hacking, offering valuable insights for future RLVR research.

Figures & tables

Figure 1. Overview of ContextRL. (a) Context-augmented reward model . (b) Context-augmented policy. (c) Training workflow. method figure

Perception Benchmarks
Benchmark	SimpleVQA	MMStar	HallusionBench			HRBench8K			MME-Real-lite		Avg.
split	overall	overall	aAcc	fAcc	qAcc	single	cross	overall	percept	reason	Avg.
Qwen3-VL 8B	44.93	64.06	71.08	53.46	53.19	79.00	61.00	70.00	54.66	41.73	59.31
SFT	46.14	70.87	74.97	56.36	56.48	67.75	70.00	71.88	55.95	50.27	62.07
GRPO	46.86	71.87	75.50	58.09	55.60	80.50	70.25	75.38	57.83	50.67	64.25
DAPO	47.62	72.00	75.29	60.98	54.73	82.25	68.75	75.50	58.51	51.73	64.74

Table 1. Performance comparison of different models on perception and reasoning Benchmarks. We report the evaluation metrics on each split of the benchmark to provide a fine-grained presentation of the models’ performance. Optimal and sub-optimal performance for each metric is denoted in bold and underlined fonts, respectively.

Reward Model	w/o ref	w. answer	w. solution
Qwen3-VL 32B	46.25	50.03	81.98
Qwen3-VL 235B	51.25	54.60	82.12

Table 2. Reward Accuracy Analysis: The proportion of identified high-confidence false-positive samples under different reward models across context settings.

Benchmark	MME-Real	MMStar	HRBench8K	MathVista	We-Math
split	lite	overall	overall	mini	strict
Qwen3-VL 8B	49.61	64.06	70.00	76.80	59.14
all positive	53.93	72.87	73.25	78.60	63.05
w 10% false	53.88	73.93	73.00	78.30	62.76
w 20% false	54.72	73.00	73.50	78.30	61.33
w 30% false	53.41	72.47	73.49	77.00	61.29

Table 3. Model performance after SFT with different proportions of false-positive samples in the training data.

Method	SimpleVQA	MMStar	HallusionBench	HRBench8K	DynaMath	MathVista	LogicVista	We-Math	Avg.
ContextRL	48.35	73.00	64.20	75.63	68.42	78.90	60.40	64.48	66.67
w/o CAR	47.02	72.13	63.13	73.00	69.64	78.60	55.93	58.19	64.71
w/o CAS	46.68	71.27	62.90	74.38	69.58	77.90	55.48	62.86	65.13
w/o report	46.70	73.07	63.29	74.63	70.16	78.50	60.36	64.10	66.35
w/o scale	47.15	72.80	63.64	78.25	68.22	78.80	59.13	61.90	66.24

Table 4. Ablation Study:In the table, CAR denotes context-augmented reward. CAS denotes context-augmented sampling. w/o report indicates that the mistake report is not provided, and w/o scale indicates that advantage scaling is not applied.

Figure 2. False Positive Example (Hallucination).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure A-1. Regular reward instruction template. regular reward instruction

Panel of Figure 8 — Figure A-2. Context-augmented reward instruction template. context-augmented reward instruction

Figure A-3. False Positive Example (Reasoning Error).

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 472 papers

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
Mar 2, 2026Guanzheng Chen, Michael Qizhe Shieh, Lidong BingRL for Language Models Reinforcement Learning with Verifiable Rewards
National University of Singapore · MiroMind AI · absolute AI
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Jan 16, 2026Lecheng Yan, Ruizhe Li, Guanhua Chen +5Reward Hacking Memorization in Language Models
Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen +5
LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards
May 29, 2026Nianyi Lin, Jiajie Zhang, Lei Hou +1LLM Grounding RL for Language Model Reasoning
Tsinghua University

Mar 2, 2026J/K move · Enter open · S save

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

Guanzheng Chen, Michael Qizhe Shieh, Lidong Bing

National University of Singapore · MiroMind AI · absolute AI

RL for Language Models Reinforcement Learning with Verifiable Rewards LLM Grounding Reward Shaping Long-Context Language Model Evaluation

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.

Abstract

RL for VLMs

Explore similar work

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards