Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR

Lirui Luo; Guoxi Zhang; Hongming Xu; Rongqing Li; Cong Fang; Lifeng Fan

cs.LGAug 19, 2026

Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR

Authors: Lirui Luo, Guoxi Zhang, Hongming Xu, Rongqing Li, Cong Fang, Lifeng Fan

Organizations: State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI · Beijing Institute of Technology · Institute for Artificial Intelligence, Peking University

Abstract

Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Sequential RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.

Figures & tables

Figure 1: Shared reasoning explains modest forgetting while CPR harnesses it to improve performance. Later-task updates benefit earlier tasks on average, helping explain modest forgetting. CPR replaces some current-task prompts with previous-task prompts to harness shared reasoning and improve learning on the arriving and future tasks.

	Reasoning tasks	Task sequence	RLVR training	CL methods
TRACE ( Wang et al., 2023 )	✓	✓	$\times$	✓
CLiMB ( Srinivasan et al., 2022 )	✓	✓	$\times$	✓
Continual World ( Wolczyk et al., 2021 )	$\times$	✓	$\times$	✓
LIBERO ( Liu et al., 2023 )	$\times$	✓	$\times$	✓
Reasoning Gym ( Stojanovski et al., 2025 )	✓	$\times$	✓	$\times$
VisuLogic ( Xu et al., 2026 )	✓	$\times$	✓	$\times$

Table 1: Positioning CRG at the intersection of CL and RLVR. CRG combines reasoning tasks, a task sequence, RLVR training, and multiple CL methods. ✓ = present,

\times

= absent.

Figure 2: Policy training proceeds stage by stage through a sequence of verifiable reasoning tasks. Top : the LLM algorithmic sequence contains $T{=}10$ tasks, with jugs as the current task. Left : at each stage, the policy rolls out responses $y_{1},\dots,y_{G}$ to a prompt. A verifier scores the responses, and Group Relative Policy Optimization (GRPO) uses these scores to produce the stage policy $M_{t}$ . Wheel : the five task sequences cover two text-reasoning settings and three visual-reasoning settings. Bottom : a jugs example shows the policy’s reasoning and verifier-checked answer.

Figure 3: Seq. RLVR remains below MTRL despite modest forgetting. Each marker represents one task sequence. Across task sequences, performance on earlier tasks falls by only

2.47

percentage points on average, yet final performance reaches

88\%

of MTRL.

Figure 4: Task gradients are positively aligned on average under Seq. RLVR. The heatmap shows pairwise cosine similarities between task gradients on the ten-task LLM algorithmic sequence. The mean cosine is $0.345$ , and $82.2\%$ of task pairs have positive cosine.

Figure 5: A jugs case illustrates shared reasoning in both performance and reasoning behavior. Jugs performance improves before its own stage and again during later-task training. The aligned excerpts show a progression from unresolved reasoning to valid and more direct solutions.

Metric	Seq. RLVR	Reset (ReDo)	Reset (FIRE)	Regularization (EWC)	Optimizer (Muon)	Isolation (OSFT)	Regularization (KL)	Replay (CPR)
FWT	0.0	-1.5	-5.8	-1.7	+1.3	-3.4	+0.6	+2.4
TLG	0.0	-7.8	-4.0	+0.4	-0.8	-7.9	-5.0	+4.9
BWT	0.0	+3.0	-14.5	+2.7	+2.5	+2.7	+5.4	-0.9
Total	0.0	-6.3	-24.4	+1.4	+3.0	-8.5	+1.1	+6.4
Mean CTM	0.88	0.78	0.29	0.87	0.97	0.65	0.95	1.03

Table 2: CPR is the only method to reach MTRL-level performance on average. For each method, FWT, TLG, and BWT report their percentage-point contributions to the change in

\mathrm{FinalAvg}

relative to Seq. RLVR. Total is their sum.

Figure 6: Previous-task gradients better align updates with the full task set. For Qwen3-4B, each line compares a current-task gradient with an equal-weight combination of that gradient and the mean previous-task gradient. Dark diamonds show the averages across the nine stages.

Figure 7: Current-policy regeneration drives CPR’s replay gain. On LLM algorithmic reasoning, CPR approaches MTRL. Sample replay reuses trajectories generated by an earlier policy and underperforms no replay.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Stream	Ordered stages
LLM	Algorithmic Reasoning	Base Conversion $\rightarrow$ Binary Alternation $\rightarrow$ Binary Matrix $\rightarrow$ Caesar Cipher $\rightarrow$ Cryptarithm $\rightarrow$ Isomorphic Strings $\rightarrow$ Jugs $\rightarrow$ Matrix Rotation $\rightarrow$ String Manipulation $\rightarrow$ A::B Rewriting
LLM	Algebra Reasoning	Complex Arithmetic $\rightarrow$ Intermediate Integration $\rightarrow$ Polynomial Equations $\rightarrow$ Polynomial Multiplication $\rightarrow$ Simple Equations $\rightarrow$ Simple Integration
VLM	Quantitative Reasoning	Linear Quantity $\rightarrow$ Prime Quantity $\rightarrow$ Point Quantity $\rightarrow$ Counter Quantity
VLM	Spatial Reasoning	Assembly $\rightarrow$ Cross-sectional $\rightarrow$ Hexahedron $\rightarrow$ Three Views $\rightarrow$ Spatial Orders $\rightarrow$ Polyhedron Rotation
VLM	Positional Reasoning	Translation $\rightarrow$ Rotation $\rightarrow$ Comparative $\rightarrow$ Flip

Appendix

Table 3: Five task sequences define the Continual Reasoning Gym streams. Arrows show the training sequence. The LLM algorithmic, LLM algebra, VLM quantitative, VLM spatial, and VLM positional streams contain 10, 6, 4, 6, and 4 stages, respectively.

Modality	Setting	Stages	Total examples	Answer format
LLM	Algorithmic	10	20,000	Symbolic
LLM	Algebra	6	20,000	Symbolic
VLM	Quantitative	4	1,386	Four-way choice
VLM	Spatial	6	1,043	Four-way choice
VLM	Positional	4	743	Four-way choice
Total		30	—	—

Appendix

Table 4: Continual Reasoning Gym contains 30 stages across five task sequences. Each setting is split

8{:}2

into training and test sets. LLM examples are generated procedurally, with at most 20,000 total examples sampled for each text setting. Symbolic and four-way answers are scored by the corresponding task verifier.

Figure 8: Representative CRG tasks. One example is shown for each setting.

Group	Hyperparameter	Value
Model	LLM backbone	Qwen3-4B
	VLM backbone	Qwen2.5-VL-7B
RLVR	Advantage estimator	GRPO
	Entropy coefficient	$1\times 10^{-3}$
	Rollouts per prompt	5
	PPO clip range	$0.2$

Appendix

Table 5: All runs use the same core RLVR configuration. The text experiments use Qwen3-4B, and the visual experiments use Qwen2.5-VL-7B. Training updates all model parameters, including the VLM vision tower.

Method	Setting	Value
MTRL	Task access	Complete task pool available from the first update
	LLM task sampling	$P_{\mathrm{MTRL}}(k)=1/T$
CPR	Replay pool	Prompts from previous tasks
	Replay fraction	$\rho=0.5$
	Maximum replayed prompts per batch	$\min\{\lfloor\rho b\rfloor,\lfloor b/2\rfloor\}$ for batch size $b$
	Store capacity	One record per observed prompt with task identity and verifier metadata. Footprint reported in Appendix B.3

Appendix

Table 6: MTRL accesses the complete task pool from the first update, whereas CPR replays previous-task prompts. Both use the shared configuration in Table 5 .

Method	Setting	Value
EWC	Penalty coefficient	$1.0$
	Fisher decay	$1.0$
	Numerical stabilizer $\epsilon$	$10^{-8}$
	Fisher samples	All available samples
FIRE	Newton–Schulz steps	5
	Reset modules	Attention query and key projections

Appendix

Table 7: Each CL baseline changes a specific part of the shared training protocol. The table lists every baseline-specific override. All remaining settings follow Table 5 .

Figure 9: CPR turns previous-task prompts into current-policy RLVR updates. The top row contrasts stale trajectory replay with CPR’s response regeneration. The lower panel shows one CPR update during a later stage. CPR selects previous-task prompts, mixes them with current-task prompts, generates every response with the current policy, and scores each response with its task verifier before the GRPO update.

Setting	Per prompt (KiB)
LLM Algorithmic	1.02
LLM Algebra	1.01
VLM Quantitative	1.42
VLM Spatial	1.39
VLM Positional	1.34
Mean	1.24

Appendix

Table 8: Average serialized footprint per CPR replay record. Each row reports the mean over prompts in one CRG setting. The final row weights the five settings equally.

Method	Mean runtime (h)
Seq. RLVR	14.6
CPR	14.8

Appendix

Table 9: Average wall-clock time per 500-update run on eight NVIDIA B200 GPUs.

Figure 10: CPR is the only evaluated CL method that reaches MTRL-level performance on average. Bars show CTM for eight sequential methods in each setting. Seq. RLVR is dark gray. The six other interventions are light gray. CPR is blue. This level is reached or exceeded for LLM algebra, VLM quantitative reasoning, and VLM spatial reasoning. Its mean CTM is

1.03

, compared with

0.88

for Seq. RLVR.

Figure 11: CPR’s FinalAvg gains arise from FWT, TLG, or both. Each pair compares Seq. RLVR with CPR. Stacked bars show the four terms BaseAvg,

\frac{T-1}{T}\mathrm{FWT}

, TLG,

\frac{T-1}{T}\mathrm{BWT}

. Black markers show FinalAvg. The BWT effect varies across task sequences. On VLM quantitative reasoning, larger FWT and TLG outweigh a more negative BWT contribution.

Role	Method	Model	Steps	Base (%)	Final (%)	Final $-$ Base (%)	FWT (%)	TLG (%)	BWT (%)
baseline	Seq. RLVR	Qwen3-4B	500	17.5	49.9	+32.3	+17.2	+15.4	+1.6
MTRL reference	MTRL	Qwen3-4B	500	17.5	65.3	+47.7	–	–	–
CL method	FIRE	Qwen3-4B	500	17.5	5.6	-11.9	+10.0	+12.9	-37.5
CL method	ReDo	Qwen3-4B	500	17.5	26.8	+9.2	+9.5	+4.6	-4.4
CL method	EWC	Qwen3-4B	500	17.5	65.0	+47.5	+19.9	+30.1	-0.6
CL method	Muon	Qwen3-4B	500	17.5	57.4	+39.9	+20.0	+23.7	-2.0

Appendix

Table 10: Complete results for LLM algorithmic reasoning. Base reports pre-training accuracy. Final reports endpoint accuracy. Final

-