One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

Zihan Lin; Xiaohan Wang; Jie Cao; Jiajun Chai; Guojun Yin; Wei Lin; Ran He

cs.LGSep 6, 2026

One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

Authors: Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, Ran He

Organizations: MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences · Meituan, Beijing · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences

Abstract

Reinforcement learning (RL) across multiple domains can broaden the reasoning capabilities of large language models (LLMs), yet joint training often degrades individual-domain performance and can destabilize optimization. Existing work typically diagnoses such interference from a single-step view using first-order gradient alignment or curvature-based proxies. We show that this view can miss a critical form of sequential interference: same-point domain gradients may remain nearly orthogonal even when consecutive realized updates partially reverse one another in output space. We further show that consecutive token log-probability footprints recover this interaction directly from adjacent checkpoints as a local second-order interaction in output space, without explicitly reconstructing same-step curvature. Building on this insight, we propose OSOL, which designates a focus domain at each iteration, uses the preceding checkpoint footprint to rank token-level rebound risk, and applies a drift-ranked, adaptively scaled correction within the standard GRPO update. Our analysis shows that this correction suppresses the targeted cross-step output backtracking component. Controlled studies further show that cross-step backtracking is more strongly associated with subsequent task damage than same-point gradient diagnostics, while the preceding footprint ranks future rebound risk more accurately than Hessian-based proxies. On Qwen3-30B-A3B, OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.

Figures & tables

Figure 1: Mixed-domain GRPO leaves substantial domain-specific headroom relative to isolated training. Validation trajectories of Qwen3-4B trained with GRPO either separately on each domain or jointly on the four-domain mixture. Clear gaps emerge in Mathematics, Chat, and Instruction Following, with a smaller separation in Code, highlighting the domain dependence of joint-training degradation.

Figure 2: Same-point alignment misses sequential damage, while broader curvature estimation remains costly. (a) Positive same-point gradient cosine does not preclude substantial task damage after the subsequent domain update; circles highlight high-damage aligned cases. (b, c) Expanding Hessian-proxy parameter coverage modestly improves token-level rebound identification but increases peak GPU memory.

Method	Math	Code	IF	Chat	Domain Avg.
Base model	0.5312	0.3310	0.2100	0.6392	0.4279
GRPO	0.5632	0.2603	0.2066	0.6900	0.4300
GRPO $+$ KL 0.001	0.5538	0.2540	0.1900	0.7271	0.4312
MGS	0.5757	0.3858	0.2000	0.6625	0.4560
CGPO	0.5670	0.3835	0.2067	0.6541	0.4528
OSOL (ours)	0.5822	0.4529	0.1933	0.7003	0.4822

Table 1: Main results on the Qwen3-30B-A3B mixture-of-experts backbone. Math averages AIME24, AIME25, and AMC23 avg@8; Code uses LiveCodeBench avg@8; IF uses IFBench strict-prompt accuracy; Chat averages normalized WildBench-Hard and CreativeWriting v3 scores. Domain Avg. is the unweighted mean of the four domain scores. Higher is better. Best and second-best values are bolded and underlined.

Method	Math	Code	IF	Chat	Domain Avg.
Base model	0.1316	0.1751	0.1433	0.4466	0.2241
GRPO	0.2906	0.2119	0.2733	0.6284	0.3511
GRPO $+$ KL 0.001	0.2750	0.2249	0.3000	0.6802	0.3700
MGS	0.2948	0.0883	0.2833	0.6401	0.3266
CGPO	0.3094	0.1828	0.2567	0.6623	0.3528
OSOL (ours)	0.3111	0.2173	0.2933	0.6723	0.3735

Table 2: Main results on Qwen3-8B-Base. Domain definitions and averaging follow Table 1 . Higher is better. Best and second-best values are bolded and underlined.

Method	Math	Code	IF	Chat	Domain Avg.
OSOL without history residual	0.2996	0.2114	0.2966	0.6324	0.3600
Equal task weights	0.0615	0.0202	0.2467	0.3978	0.1815
Response-level residual	0.3202	0.2021	0.3000	0.6338	0.3640
Random residual	0.0698	0.0390	0.2400	0.3866	0.1838
OSOL (ours)	0.3111	0.2173	0.2933	0.6723	0.3735

Table 3: Ablation results on Qwen3-8B-Base. OSOL without history residual retains the one-step-one-lead focus structure while removing the checkpoint-history correction; the remaining variants ablate focus weighting, drift-based ranking, or token-level allocation.

Figure 3: Layer-35 signed gradient contributions expose cancellation hidden by the global cosine. Columns are seeds ordered by observed task interference; opposing module contributions can cancel under global aggregation.

Figure 4: Checkpoint history provides a stronger token-level rebound signal than Hessian-based proxies. Checkpoint-footprint ranking improves precision and enrichment across candidate fractions (left) and remains superior in ROC-AUC and average precision even when the Hessian proxy uses the realized optimizer update (right).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Rebound identification within negative-drift candidates. The checkpoint footprint outperforms the Hessian proxy in ROC-AUC, average precision, and top-5% precision; the non-token-resolved global scalar is included as a 0.5 ROC-AUC reference. Error bars show 95% confidence intervals.

Figure 6: Replicate-level association between interference diagnostics and realized task damage. Same-point global and module scores do not positively order the subsequent damage, whereas cross-step backtracking mass shows a positive association (

\rho_{s}=0.40

, 95% bootstrap CI

[0.05,0.69]

). The backtracking and damage measurements use disjoint probes.

Figure 7: Uncertainty and paired correlation gains for the three Study I diagnostics. Left: bootstrap estimates of Spearman correlation with task damage. Right: paired gains from replacing the same-point global or module diagnostic with cross-step backtracking mass. Both paired intervals remain positive.

Figure 8: Leave-one-replicate stability of the Study I correlations. Each point recomputes Spearman

\rho

after removing one replicate; horizontal ranges summarize all 32 deletions. The sign and ordering of the diagnostics are preserved throughout.

Figure 9: Sensitivity of the finite-difference Hessian proxy to the probe scale

\epsilon

. Left: the estimated HVP norm varies with

\epsilon

. Right: the induced policy KL of the virtual focus step, Hessian reaction, and predicted next step changes by orders of magnitude, showing that the probe scale also changes the output-space displacement being evaluated.

Setting	Configuration
Model backbones	Qwen3-30B-A3B and Qwen3-8B-Base
Training domains	Mathematics, code generation, instruction following, and open-ended Chat
Training examples	7.5K examples per domain
Focus-domain schedule	Uniform random sampling from $\mathcal{D}\setminus\{f_{k-1}\}$ ; consecutive steps cannot use the same focus domain
Prompt batch size	128 prompts per trainer step
Rollouts per prompt	8

Appendix

Table 4: Main-training and generation settings. The prompt batch size is counted before rollout expansion.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Local Conflict, Targeted Repair: A Perturbation Analysis of Multi-Domain RL
Jun 1, 2026Lei Yang, Siyu Ding, Deyi XiongRL for Language Models Reinforcement Learning
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Baidu Inc., Beijing, China
CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflicts
May 30, 2026Rui Zhang, Xinle Wu, Yao LuReward Modeling Reinforcement Learning
National University of Singapore Singapore
To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
Feb 13, 2026Haoqing Wang, Xiang Long, Ziheng Li +3RL for Language Models Gradient Interference
Samsung Research, Beijing, China · Peking University

Jun 1, 2026J/K move · Enter open · S save

Local Conflict, Targeted Repair: A Perturbation Analysis of Multi-Domain RL

Lei Yang, Siyu Ding, Deyi Xiong

TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Baidu Inc., Beijing, China

RL for Language Models Reinforcement Learning Multi-Task RL RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training improves large language models on individual domains but can degrade previously acquired capabilities. We study this interference in mathematical reasoning, code generation, question answering (QA), and creative writing (CW). Full-model domain gradients can be nearly orthogonal despite substantial selective degradation. Our structural analyses show sparse, weakly overlapping edits on computation routes that reasoning domains share, with directional alignment associated with transfer and conflict. A local perturbation model separates loss changes into a first-order term and a path-averaged curvature term, decomposes the latter along the actual rollback intervention, and provides a conditional explanation for recovery by short refresh. The experiments support this local sensitivity account. After Code

\rightarrow

Math

\rightarrow

\rightarrow

CW training, a short Re-Math refresh raises Math from 57.66 to 66.04 and achieves the highest average score among the evaluated baselines. A training-free rollback at a

9.8\%

budget recovers

57.4\%

of the QA-induced Math accuracy drop with a small QA change. Curvature probes at the Math checkpoint further show higher sensitivity along the selected direction than along matched random directions. Together, these results support sparse intervention and targeted refresh as effective ways to mitigate cross-domain degradation.

Abstract

RL for Language Model Reasoning

Explore similar work

Local Conflict, Targeted Repair: A Perturbation Analysis of Multi-Domain RL

CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflicts

To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models