Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

Hoang Phan; Xianjun Yang; Yuanshun Yao; Jingyu Zhang; Shengjie Bi; Xiaocheng Tang; Madian Khabsa; Lijuan Liu

cs.LGOct 24, 2025

Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

Authors: Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu,

Organizations: Meta Superintelligence Labs · New York University · Johns Hopkins University

Abstract

Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning and has become a standard post-training paradigm for contemporary language and vision-language models. However, the RLVR recipe introduces a significant risk of capability regression, in which models forget foundational skills after prolonged training without employing regularization strategies. We empirically confirm this concern, observing that open-source reasoning models suffer performance degradation on core capabilities such as perception and faithfulness. While imposing regularization terms like KL divergence can help prevent deviation from the base model, these terms are computed on the current task and therefore do not guarantee preservation of broader knowledge. Meanwhile, commonly used experience replay across heterogeneous domains makes it nontrivial to decide how much training emphasis each objective should receive. To address this, we propose RECAP-a replay strategy with dynamic objective reweighting for general knowledge preservation. Our reweighting mechanism adapts online using short-horizon signals of convergence and instability, shifting the post-training focus away from saturated objectives and toward underperforming or volatile ones. Our method is end-to-end and readily applicable to existing RLVR pipelines without training additional models or heavy tuning. Extensive experiments on benchmarks using Qwen2.5-VL-3B and Qwen2.5-VL-7B demonstrate the effectiveness of our method, which not only preserves general capabilities but also improves reasoning by enabling more flexible trade-offs among in-task rewards.

Figures & tables

Figure 1: General-capability comparison of base VLMs ( blue ) and their reasoning-tuned variants ( green / purple / red ) on six representative non-reasoning benchmarks (higher is better). A-OKVQA (knowledge-based VQA), AesBench ( Huang et al., 2024 ) (image aesthetics), VStar ( Wu and Xie, 2023 ) (spatio-temporal reasoning), VisOnly ( Kamoi et al., 2025 ) (vision-only recognition aggregate), OCRBench ( Liu et al., 2024 ) (text recognition), and R-Bench-Dis ( Li et al., 2025b ) (distribution-shift robustness). Across both Qwen2.5-VL families, reasoning-finetuned models generally underperform their base models on perception and robustness tasks, whereas MiMo-VL-7B-RL remains close to its SFT baseline.

Figure 2: Performance comparison between finetuning solely on math and with RECAP . Our model not only preserves base-model performance but also improves it (

\uparrow 2\%

), whereas the reasoning-only model quickly falls behind the base model after 100 iterations.

Figure 3: Overview of RECAP. Along with the target reasoning task, we sample data from general domains to maintain that knowledge during finetuning. Initially, the objectives of interest are weighted uniformly to optimize the main model. After a few iterations, we record the convergence behavior of individual objectives. Based on this behavior, we adjust the focus to prevent any objective from dominating and assign less weight to saturated ones.

Figure 4: Different rewards exhibit different convergence behavior. While the format reward is easy to optimize and initially has the highest convergence rate, it quickly saturates and thus yields a near-unity convergence rate (

c\sim 1

) and low instability (

i\sim 0

) after 50 steps. By contrast, the reasoning accuracy fluctuates the most, thereby steering the optimization toward the corresponding objective. IoU and ntp denote the IoU reward and next-token-prediction accuracy during training. The result is obtained in the first setting in our experiments below.

Open-source reasoning baselines
Model	SAT	ScienceQA	MathVista (mini)	ChartQA	InfoVQA	MMMU
VLAA-Thinker-3B	49.38	14.63	30.4	45.84	30.81	32.22
MM-R1-MGT-PerceReason	50.83	34.21	33.4	44.88	61.42	40.22
Ocean_R1_3B_Instruct	59.49	68.72	38.7	54.00	38.02	40.89
Qwen2.5VL-3b-RLCS	24.12	21.32	17.2	3.32	10.86	27.11
vision-grpo-qwen-2.5-vl-3b	50.57	4.17	32.4	67.80	58.29	37.22

Table 1: Benchmark results in the RLVR-only setting. We report accuracy scores on six benchmarks, where the MoDoMoDo baseline is trained to maximize performance. For this table only, we use the rule-based evaluator on the MathVista dataset instead of ”gpt-3.5-turbo” to align with Liang et al. (2025) .

Open-source reasoning baselines
Model	LISA	MMMU-PRO	AI2D	MathVista	MathVision	MathVerse	MMBench	VizWiz	OCRBenchv2
VLAA-Thinker-7B	63.14	26.30	75.45	63.90	11.18	29.87	75.95	47.57	40.23
Vision-R1-7B	47.30	26.76	0.00	61.80	18.75	23.32	69.46	53.12	24.63
OpenVLThinker-7B	42.73	21.79	59.94	59.10	5.59	19.26	71.53	52.89	28.30
Qwen2.5-VL-7B and our variants
Base model	65.13	25.55	67.62	61.70	9.54	26.29	71.82	50.82	39.49

Table 2: Benchmark results in large hybrid setting. We report accuracy scores (higher is better) on nine perception and reasoning benchmarks. Rows above the break are open-source reasoning models with different backbones; the lower block compares variants finetuned from the same Qwen2.5-VL-7B base model. Bold = best; underline = second best within the Qwen2.5-VL-7B family.

Figure 5: Evolution of format and accuracy rewards on the reasoning domain during training: Training curves for the format reward over full training (left), an early-training zoom (middle), and the accuracy reward (right). While the uniform baseline is better at maximizing the format reward, it falls behind our proposed method later in terms of accuracy, as we prioritize correct solutions over formatting once the model can follow the predefined template. Curves are smoothed with an exponential moving average for readability.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Answer Type	Rewards/Objectives	# samples
RefCOCO ( Kazemzadeh et al., 2014 )	Referring Expression Comprehension	2D Bounding Box	IoU, Answer Format	321327
LLaVA-OneVision-OCR ( Li et al., 2024 )	Scene Text-Centric Visual Question Answering	Natural Language	Next Token Prediction	66468
ThinkLite-VL-70k ( Wang et al., 2025c )	Math Reasoning & Natural Image/Chart Understanding	Natural Language	Acc, Thinking Format	69997
LISA-train ( Lai et al., 2024a )	Referring Expression	2D Bounding Box	IoU, Thinking Format	1326
GeoQAV ( Li et al., 2025a )	Math Visual Question Answering	Multiple Choice	Acc, Thinking Format	1969
SAT-train ( Ray et al., 2024b )	Spatial Visual Question Answering	Natural Language	Acc, Thinking Format	15000

Appendix

Table 3: Data statistics for each data source. We present the original volume of data (# samples).

L_{k}^{(t)}\leftarrow\frac{1}{N}\sum_{n=1}^{N}\ell_{n,k}^{(t)}(\theta^{(t)}),\quad\forall k

Appendix

Algorithm 1 Replay-Enhanced CApability Preservation (RECAP).

Figure 8: Example conversation for finetuning. We ask the model to produce its reasoning chain within <think> $\dots$ </think> and the final answer in <answer> $\dots$ </answer> tags.

Figure 9: Generated answers from the reasoning-only finetuned model and our model. While the reasoning-only finetuned model generates long reasoning chains for simple visual questions, our model produces more concise answers, especially on non-mathematical tasks.

Figure 10: Reward dynamics and variability during RLVR training. Per-step rewards (light traces) and sliding-window means (dark curves) for six metrics: Next-Token Prediction, Answer-Format, Thinking-Format, IoU, Accuracy, and the Total-Reward Standard Deviation (lower-right). Asynchronous convergence and high variance motivate short-horizon statistics for dynamic objective reweighting rather than per-iteration magnitudes.

Figure 11: Evolution of per-objective coefficients. In the last 100 iterations, the coefficient for each objective is relatively consistent, with format rewards receiving the lowest focus while the supervised finetuning objective and accuracy rewards are emphasized due to their instability.

Figure 12: Final performance across metrics. We compare a uniform baseline with our dynamic reweighting. The gains on correctness-oriented metrics indicate that reallocating weight away from saturated format rewards toward harder objectives yields better solutions without sacrificing adherence to templates.

Panel of Figure 13 — Figure 13: A modified illustrative two-task example from ( Navon et al., 2022 ) showing the convergence of comparison methods from different initialization points (black dots $\bullet$ ). Each optimization trajectory is colored from yellow to red . The bold gray line represents the Pareto front. Overall, introducing noise in the first objective causes instability in many MTL methods. Methods that leverage per-step loss-magnitude statistics, such as FAMO, DWA, and UW, exhibit considerable instability during convergence from different initializations.

		Segmentation		Depth		Surface Normal
Complexity		mIoU $\uparrow$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Angle Distance $\downarrow$		Within $t^{\circ}$ $\uparrow$			$\mathbf{\Delta m\%}\downarrow$
		mIoU $\uparrow$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Mean	Median	11.25	22.5	30
	STL	$38.30$	$63.76$	$0.6754$	$0.2780$	$25.01$	$19.21$	$30.14$	$57.20$	$69.15$
	LS	$39.29$	$65.33$	$0.5493$	$0.2263$	$28.15$	$23.96$	$22.09$	$47.50$	$61.08$	$5.59$
	SI	$38.45$	$64.27$	$0.5354$	$0.2201$	$27.60$	$23.37$	$22.53$	$48.57$	$62.32$	$4.39$
	RLW	$37.17$	$63.77$	$0.5759$	$0.2410$	$28.27$	$24.18$	$22.26$	$47.05$	$60.62$	$7.78$

Appendix

Table 4: Evaluation results on NYUv2 scene understanding. Test performance for three tasks: semantic segmentation, depth estimation, and surface normals. We highlight the best loss-magnitude-based MTL method in bold and the best gradient-based MTL method with an underscore .

Figure 14: Loss curves during training on NYUv2. Compared with the training curves in our experiments (e.g., Figure 10 ), these curves are much smoother and more stable, making per-step statistics informative signals of learning progress.

Figure 15: Running time of different MTL methods. While robust to noise in some scenarios, gradient-based methods (denoted by blue ) often incur significant overhead (

\approx K

times, because they compute per-objective gradients) compared with loss-magnitude-based methods (denoted by orange ).

Panel of Figure 16 — Figure 16: Ablation on the trade-off $\alpha$ . Using only the convergence rate ( $\alpha=1$ ) or only the inverse signal-to-noise ratio ( $\alpha=0$ ) leads to unstable learning for the second and first objectives, respectively.

Panel of Figure 17 — Figure 17: Ablation studies on the temperature $T$ . $T>1$ acts as a regularization to avoid extreme reweighting (one domain dominates others) and stabilizes the training.

Model	SAT	ScienceQA	MathVista (mini)	ChartQA	InfoVQA	MMMU
MoDoMoDo	50.0	65.7	32.2	70.4	59.9	39.1
RECAP	55.2	71.6	33.2	70.4	60.8	42.4
$\alpha$ = 0.50, T = 1.0, W=10	52.0	71.6	33.4	68.1	58.5	40.4
$\alpha$ = 0.75, T = 5.0, W=10	54.4	71.2	32.9	70.0	60.7	41.0
$\alpha$ = 0.25, T = 5.0, W=10	55.3	72.2	33.7	66.1	56.8	39.3
$\alpha$ = 0.50, T = 5.0, W=50	51.9	70.5	32.9	69.9	59.7	40.8

Appendix

Table 5: Benchmark performance in RLVR-only setting. Ablation results when varying the temperature and convergence rate-instability trade-off.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 426 papers

Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR
Aug 19, 2026Lirui Luo, Guoxi Zhang, Hongming Xu +3RL for Language Model Reasoning Replay-Based Continual Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing Institute of Technology +1
Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1RL for Language Model Reasoning Efficient Language Model Reasoning
University of Southern California · 2DEVCOM ARL
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
Oct 16, 2026Wen-Zhi Li, Sainyam Galhotra, Claire CardieReinforcement Learning RL for Language Model Reasoning
Cornell University

Aug 19, 2026J/K move · Enter open · S save

Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR

Lirui Luo, Guoxi Zhang, Hongming Xu, Rongqing Li+2

State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing Institute of Technology+1

RL for Language Model Reasoning Replay-Based Continual Learning Transfer Learning Reinforcement Learning with Verifiable Rewards Continual RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Sequential RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.

Abstract

RL Post-Training

Explore similar work

Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay