One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
Authors: Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, Ran He
Organizations: MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences · Meituan, Beijing · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences
Reinforcement learning (RL) across multiple domains can broaden the reasoning capabilities of large language models (LLMs), yet joint training often degrades individual-domain performance and can destabilize optimization. Existing work typically diagnoses such interference from a single-step view using first-order gradient alignment or curvature-based proxies. We show that this view can miss a critical form of sequential interference: same-point domain gradients may remain nearly orthogonal even when consecutive realized updates partially reverse one another in output space. We further show that consecutive token log-probability footprints recover this interaction directly from adjacent checkpoints as a local second-order interaction in output space, without explicitly reconstructing same-step curvature. Building on this insight, we propose OSOL, which designates a focus domain at each iteration, uses the preceding checkpoint footprint to rank token-level rebound risk, and applies a drift-ranked, adaptively scaled correction within the standard GRPO update. Our analysis shows that this correction suppresses the targeted cross-step output backtracking component. Controlled studies further show that cross-step backtracking is more strongly associated with subsequent task damage than same-point gradient diagnostics, while the preceding footprint ranks future rebound risk more accurately than Hessian-based proxies. On Qwen3-30B-A3B, OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
Figures & tables
Figure 1: Mixed-domain GRPO leaves substantial domain-specific headroom relative to isolated training. Validation trajectories of Qwen3-4B trained with GRPO either separately on each domain or jointly on the four-domain mixture. Clear gaps emerge in Mathematics, Chat, and Instruction Following, with a smaller separation in Code, highlighting the domain dependence of joint-training degradation.
Figure 2: Same-point alignment misses sequential damage, while broader curvature estimation remains costly. (a) Positive same-point gradient cosine does not preclude substantial task damage after the subsequent domain update; circles highlight high-damage aligned cases. (b, c) Expanding Hessian-proxy parameter coverage modestly improves token-level rebound identification but increases peak GPU memory.
Method
Math
Code
IF
Chat
Domain Avg.
Base model
0.5312
0.3310
0.2100
0.6392
0.4279
GRPO
0.5632
0.2603
0.2066
0.6900
0.4300
GRPO + KL 0.001
0.5538
0.2540
0.1900
0.7271
0.4312
MGS
0.5757
0.3858
0.2000
0.6625
0.4560
CGPO
0.5670
0.3835
0.2067
0.6541
0.4528
OSOL (ours)
0.5822
0.4529
0.1933
0.7003
0.4822
Table 1: Main results on the Qwen3-30B-A3B mixture-of-experts backbone. Math averages AIME24, AIME25, and AMC23 avg@8; Code uses LiveCodeBench avg@8; IF uses IFBench strict-prompt accuracy; Chat averages normalized WildBench-Hard and CreativeWriting v3 scores. Domain Avg. is the unweighted mean of the four domain scores. Higher is better. Best and second-best values are bolded and underlined.
Method
Math
Code
IF
Chat
Domain Avg.
Base model
0.1316
0.1751
0.1433
0.4466
0.2241
GRPO
0.2906
0.2119
0.2733
0.6284
0.3511
GRPO + KL 0.001
0.2750
0.2249
0.3000
0.6802
0.3700
MGS
0.2948
0.0883
0.2833
0.6401
0.3266
CGPO
0.3094
0.1828
0.2567
0.6623
0.3528
OSOL (ours)
0.3111
0.2173
0.2933
0.6723
0.3735
Table 2: Main results on Qwen3-8B-Base. Domain definitions and averaging follow Table 1 . Higher is better. Best and second-best values are bolded and underlined.
Method
Math
Code
IF
Chat
Domain Avg.
OSOL without history residual
0.2996
0.2114
0.2966
0.6324
0.3600
Equal task weights
0.0615
0.0202
0.2467
0.3978
0.1815
Response-level residual
0.3202
0.2021
0.3000
0.6338
0.3640
Random residual
0.0698
0.0390
0.2400
0.3866
0.1838
OSOL (ours)
0.3111
0.2173
0.2933
0.6723
0.3735
Table 3: Ablation results on Qwen3-8B-Base. OSOL without history residual retains the one-step-one-lead focus structure while removing the checkpoint-history correction; the remaining variants ablate focus weighting, drift-based ranking, or token-level allocation.
Figure 3: Layer-35 signed gradient contributions expose cancellation hidden by the global cosine. Columns are seeds ordered by observed task interference; opposing module contributions can cancel under global aggregation.
Figure 4: Checkpoint history provides a stronger token-level rebound signal than Hessian-based proxies. Checkpoint-footprint ranking improves precision and enrichment across candidate fractions (left) and remains superior in ROC-AUC and average precision even when the Hessian proxy uses the realized optimizer update (right).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Rebound identification within negative-drift candidates. The checkpoint footprint outperforms the Hessian proxy in ROC-AUC, average precision, and top-5% precision; the non-token-resolved global scalar is included as a 0.5 ROC-AUC reference. Error bars show 95% confidence intervals.
Figure 6: Replicate-level association between interference diagnostics and realized task damage. Same-point global and module scores do not positively order the subsequent damage, whereas cross-step backtracking mass shows a positive association ( ρs=0.40 , 95% bootstrap CI [0.05,0.69] ). The backtracking and damage measurements use disjoint probes.
Figure 7: Uncertainty and paired correlation gains for the three Study I diagnostics. Left: bootstrap estimates of Spearman correlation with task damage. Right: paired gains from replacing the same-point global or module diagnostic with cross-step backtracking mass. Both paired intervals remain positive.
Figure 8: Leave-one-replicate stability of the Study I correlations. Each point recomputes Spearman ρ after removing one replicate; horizontal ranges summarize all 32 deletions. The sign and ordering of the diagnostics are preserved throughout.
Figure 9: Sensitivity of the finite-difference Hessian proxy to the probe scale ϵ . Left: the estimated HVP norm varies with ϵ . Right: the induced policy KL of the virtual focus step, Hessian reaction, and predicted next step changes by orders of magnitude, showing that the probe scale also changes the output-space displacement being evaluated.
Setting
Configuration
Model backbones
Qwen3-30B-A3B and Qwen3-8B-Base
Training domains
Mathematics, code generation, instruction following, and open-ended Chat
Training examples
7.5K examples per domain
Focus-domain schedule
Uniform random sampling from D∖{fk−1} ; consecutive steps cannot use the same focus domain
Prompt batch size
128 prompts per trainer step
Rollouts per prompt
8
Appendix
Table 4: Main-training and generation settings. The prompt batch size is counted before rollout expansion.
Reinforcement learning (RL) post-training improves large language models on individual domains but can degrade previously acquired capabilities. We study this interference in mathematical reasoning, code generation, question answering (QA), and creative writing (CW). Full-model domain gradients can be nearly orthogonal despite substantial selective degradation. Our structural analyses show sparse, weakly overlapping edits on computation routes that reasoning domains share, with directional alignment associated with transfer and conflict. A local perturbation model separates loss changes into a first-order term and a path-averaged curvature term, decomposes the latter along the actual rollback intervention, and provides a conditional explanation for recovery by short refresh. The experiments support this local sensitivity account. After Code → Math → QA → CW training, a short Re-Math refresh raises Math from 57.66 to 66.04 and achieves the highest average score among the evaluated baselines. A training-free rollback at a 9.8% budget recovers 57.4% of the QA-induced Math accuracy drop with a small QA change. Curvature probes at the Math checkpoint further show higher sensitivity along the selected direction than along matched random directions. Together, these results support sparse intervention and targeted refresh as effective ways to mitigate cross-domain degradation.
Lei Yang, Siyu Ding, Deyi Xiong
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Baidu Inc., Beijing, China
Reinforcement learning (RL) with verifiable rewards has achieved strong progress in reasoning-oriented LLMs, but extending it to multi-domain RL remains challenging due to reward unreliability in non-verifiable tasks and capability interference across domains. We propose CARE-RL to combine protocol-aware reward generation with capability-aware optimization for mitigating cross-domain conflicts. For non-verifiable tasks, the Protocol-Aware Generative Reward Model (PA-GRM) constructs prompt-level evaluation protocols and schemas before producing trace-conditioned rewards, enabling task-adaptive yet comparable evaluation of open-ended responses. For multi-domain optimization, Direction-Aware Capability Subspace Projection (DACSP) extracts historical capability directions from previous RL stages and modulates later updates by amplifying aligned components, suppressing conflicting components, and preserving orthogonal updates. Experiments across math, chat, and instruction-following benchmarks show that CARE-RL consistently outperforms standard multi-domain RL baselines, achieving Total Avg scores of 47.9 and 50.7 on Qwen2.5-7B and Qwen3-4B, respectively.
Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, instruction following, and agent) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits small mutual interferences, and reasoning-intensive domains have mutually synergistic effects. Furthermore, we analyze the internal mechanisms from the perspectives of information constraints, model prediction behavior and self-verification. Our homepage is at https://github.com/Mosi-AI/M2RL.
Haoqing Wang, Xiang Long, Ziheng Li +3
Samsung Research, Beijing, China · Peking University