Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning and has become a standard post-training paradigm for contemporary language and vision-language models. However, the RLVR recipe introduces a significant risk of capability regression, in which models forget foundational skills after prolonged training without employing regularization strategies. We empirically confirm this concern, observing that open-source reasoning models suffer performance degradation on core capabilities such as perception and faithfulness. While imposing regularization terms like KL divergence can help prevent deviation from the base model, these terms are computed on the current task and therefore do not guarantee preservation of broader knowledge. Meanwhile, commonly used experience replay across heterogeneous domains makes it nontrivial to decide how much training emphasis each objective should receive. To address this, we propose RECAP-a replay strategy with dynamic objective reweighting for general knowledge preservation. Our reweighting mechanism adapts online using short-horizon signals of convergence and instability, shifting the post-training focus away from saturated objectives and toward underperforming or volatile ones. Our method is end-to-end and readily applicable to existing RLVR pipelines without training additional models or heavy tuning. Extensive experiments on benchmarks using Qwen2.5-VL-3B and Qwen2.5-VL-7B demonstrate the effectiveness of our method, which not only preserves general capabilities but also improves reasoning by enabling more flexible trade-offs among in-task rewards.
Figures & tables
Figure 1: General-capability comparison of base VLMs ( blue ) and their reasoning-tuned variants ( green / purple / red ) on six representative non-reasoning benchmarks (higher is better). A-OKVQA (knowledge-based VQA), AesBench ( Huang et al., 2024 ) (image aesthetics), VStar ( Wu and Xie, 2023 ) (spatio-temporal reasoning), VisOnly ( Kamoi et al., 2025 ) (vision-only recognition aggregate), OCRBench ( Liu et al., 2024 ) (text recognition), and R-Bench-Dis ( Li et al., 2025b ) (distribution-shift robustness). Across both Qwen2.5-VL families, reasoning-finetuned models generally underperform their base models on perception and robustness tasks, whereas MiMo-VL-7B-RL remains close to its SFT baseline.
Figure 2: Performance comparison between finetuning solely on math and with RECAP . Our model not only preserves base-model performance but also improves it ( ↑2% ), whereas the reasoning-only model quickly falls behind the base model after 100 iterations.
Figure 3: Overview of RECAP. Along with the target reasoning task, we sample data from general domains to maintain that knowledge during finetuning. Initially, the objectives of interest are weighted uniformly to optimize the main model. After a few iterations, we record the convergence behavior of individual objectives. Based on this behavior, we adjust the focus to prevent any objective from dominating and assign less weight to saturated ones.
Figure 4: Different rewards exhibit different convergence behavior. While the format reward is easy to optimize and initially has the highest convergence rate, it quickly saturates and thus yields a near-unity convergence rate ( c∼1 ) and low instability ( i∼0 ) after 50 steps. By contrast, the reasoning accuracy fluctuates the most, thereby steering the optimization toward the corresponding objective. IoU and ntp denote the IoU reward and next-token-prediction accuracy during training. The result is obtained in the first setting in our experiments below.
Model
SAT
ScienceQA
MathVista (mini)
ChartQA
InfoVQA
MMMU
Open-source reasoning baselines
VLAA-Thinker-3B
49.38
14.63
30.4
45.84
30.81
32.22
MM-R1-MGT-PerceReason
50.83
34.21
33.4
44.88
61.42
40.22
Ocean_R1_3B_Instruct
59.49
68.72
38.7
54.00
38.02
40.89
Qwen2.5VL-3b-RLCS
24.12
21.32
17.2
3.32
10.86
27.11
vision-grpo-qwen-2.5-vl-3b
50.57
4.17
32.4
67.80
58.29
37.22
Table 1: Benchmark results in the RLVR-only setting. We report accuracy scores on six benchmarks, where the MoDoMoDo baseline is trained to maximize performance. For this table only, we use the rule-based evaluator on the MathVista dataset instead of ”gpt-3.5-turbo” to align with Liang et al. (2025) .
Model
LISA
MMMU-PRO
AI2D
MathVista
MathVision
MathVerse
MMBench
VizWiz
OCRBenchv2
Open-source reasoning baselines
VLAA-Thinker-7B
63.14
26.30
75.45
63.90
11.18
29.87
75.95
47.57
40.23
Vision-R1-7B
47.30
26.76
0.00
61.80
18.75
23.32
69.46
53.12
24.63
OpenVLThinker-7B
42.73
21.79
59.94
59.10
5.59
19.26
71.53
52.89
28.30
Qwen2.5-VL-7B and our variants
Base model
65.13
25.55
67.62
61.70
9.54
26.29
71.82
50.82
39.49
Table 2: Benchmark results in large hybrid setting. We report accuracy scores (higher is better) on nine perception and reasoning benchmarks. Rows above the break are open-source reasoning models with different backbones; the lower block compares variants finetuned from the same Qwen2.5-VL-7B base model. Bold = best; underline = second best within the Qwen2.5-VL-7B family.
Figure 5: Evolution of format and accuracy rewards on the reasoning domain during training: Training curves for the format reward over full training (left), an early-training zoom (middle), and the accuracy reward (right). While the uniform baseline is better at maximizing the format reward, it falls behind our proposed method later in terms of accuracy, as we prioritize correct solutions over formatting once the model can follow the predefined template. Curves are smoothed with an exponential moving average for readability.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Domain
Answer Type
Rewards/Objectives
# samples
RefCOCO ( Kazemzadeh et al., 2014 )
Referring Expression Comprehension
2D Bounding Box
IoU, Answer Format
321327
LLaVA-OneVision-OCR ( Li et al., 2024 )
Scene Text-Centric Visual Question Answering
Natural Language
Next Token Prediction
66468
ThinkLite-VL-70k ( Wang et al., 2025c )
Math Reasoning & Natural Image/Chart Understanding
Natural Language
Acc, Thinking Format
69997
LISA-train ( Lai et al., 2024a )
Referring Expression
2D Bounding Box
IoU, Thinking Format
1326
GeoQAV ( Li et al., 2025a )
Math Visual Question Answering
Multiple Choice
Acc, Thinking Format
1969
SAT-train ( Ray et al., 2024b )
Spatial Visual Question Answering
Natural Language
Acc, Thinking Format
15000
Appendix
Table 3: Data statistics for each data source. We present the original volume of data (# samples).
Figure 8: Example conversation for finetuning. We ask the model to produce its reasoning chain within <think> … </think> and the final answer in <answer> … </answer> tags.
Figure 9: Generated answers from the reasoning-only finetuned model and our model. While the reasoning-only finetuned model generates long reasoning chains for simple visual questions, our model produces more concise answers, especially on non-mathematical tasks.
Figure 10: Reward dynamics and variability during RLVR training. Per-step rewards (light traces) and sliding-window means (dark curves) for six metrics: Next-Token Prediction, Answer-Format, Thinking-Format, IoU, Accuracy, and the Total-Reward Standard Deviation (lower-right). Asynchronous convergence and high variance motivate short-horizon statistics for dynamic objective reweighting rather than per-iteration magnitudes.
Figure 11: Evolution of per-objective coefficients. In the last 100 iterations, the coefficient for each objective is relatively consistent, with format rewards receiving the lowest focus while the supervised finetuning objective and accuracy rewards are emphasized due to their instability.
Figure 12: Final performance across metrics. We compare a uniform baseline with our dynamic reweighting. The gains on correctness-oriented metrics indicate that reallocating weight away from saturated format rewards toward harder objectives yields better solutions without sacrificing adherence to templates.
Figure 13: A modified illustrative two-task example from ( Navon et al., 2022 ) showing the convergence of comparison methods from different initialization points (black dots ∙ ). Each optimization trajectory is colored from yellow to red . The bold gray line represents the Pareto front. Overall, introducing noise in the first objective causes instability in many MTL methods. Methods that leverage per-step loss-magnitude statistics, such as FAMO, DWA, and UW, exhibit considerable instability during convergence from different initializations.
Segmentation
Depth
Surface Normal
Complexity
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Angle Distance ↓
Within t∘↑
Δm%↓
Mean
Median
11.25
22.5
30
STL
38.30
63.76
0.6754
0.2780
25.01
19.21
30.14
57.20
69.15
LS
39.29
65.33
0.5493
0.2263
28.15
23.96
22.09
47.50
61.08
5.59
SI
38.45
64.27
0.5354
0.2201
27.60
23.37
22.53
48.57
62.32
4.39
RLW
37.17
63.77
0.5759
0.2410
28.27
24.18
22.26
47.05
60.62
7.78
Appendix
Table 4: Evaluation results on NYUv2 scene understanding. Test performance for three tasks: semantic segmentation, depth estimation, and surface normals. We highlight the best loss-magnitude-based MTL method in bold and the best gradient-based MTL method with an underscore .
Figure 14: Loss curves during training on NYUv2. Compared with the training curves in our experiments (e.g., Figure 10 ), these curves are much smoother and more stable, making per-step statistics informative signals of learning progress.
Figure 15: Running time of different MTL methods. While robust to noise in some scenarios, gradient-based methods (denoted by blue ) often incur significant overhead ( ≈K times, because they compute per-objective gradients) compared with loss-magnitude-based methods (denoted by orange ).
Figure 16: Ablation on the trade-off α . Using only the convergence rate ( α=1 ) or only the inverse signal-to-noise ratio ( α=0 ) leads to unstable learning for the second and first objectives, respectively.
Figure 17: Ablation studies on the temperature T . T>1 acts as a regularization to avoid extreme reweighting (one domain dominates others) and stabilizes the training.
Model
SAT
ScienceQA
MathVista (mini)
ChartQA
InfoVQA
MMMU
MoDoMoDo
50.0
65.7
32.2
70.4
59.9
39.1
RECAP
55.2
71.6
33.2
70.4
60.8
42.4
α = 0.50, T = 1.0, W=10
52.0
71.6
33.4
68.1
58.5
40.4
α = 0.75, T = 5.0, W=10
54.4
71.2
32.9
70.0
60.7
41.0
α = 0.25, T = 5.0, W=10
55.3
72.2
33.7
66.1
56.8
39.3
α = 0.50, T = 5.0, W=50
51.9
70.5
32.9
69.9
59.7
40.8
Appendix
Table 5: Benchmark performance in RLVR-only setting. Ablation results when varying the temperature and convergence rate-instability trade-off.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Sequential RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.
Lirui Luo, Guoxi Zhang, Hongming Xu +3
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing Institute of Technology +1
Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.
Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1
Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capabilities of large language models, but often reduces the diversity of their reasoning behaviors. We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively suppress semantically distinct but equally rewarded reasoning modes. Once a mode becomes sufficiently under-sampled, standard on-policy RLVR rarely recovers it. Motivated by this observation, we propose TAR, a Targeted Adaptive Replay framework that preserves diverse correct reasoning behaviors already present in the base policy. TAR constructs a compact memory bank of diverse base-model solutions and selectively replays them through an adaptive mechanism that activates only when stored behaviors are being suppressed during RLVR. Across six mathematical reasoning benchmarks and two Qwen2.5 model scales, TAR consistently achieves a better balance between reasoning performance and diversity preservation than existing diversity-preserving post-training methods, while substantially reducing the forgetting of correct base-model solutions. Code is available at https://github.com/wenzhilics/tar.