Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artifacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.
Figures & tables
Figure 1: Three paradigms for consolidating domain-specific capabilities acquired through RLVR into one model. Merge combines expert task vectors, Mix RL trains jointly on mixed-domain data, and MOPD draws on both through multi-teacher on-policy distillation.
Tuning
AIME25
AIME26
GPQA
LCBv5
LCBv6
IFEval
IFBench
BFCLv3
Avg.
Qwen3-4B-Instruct-2507
Base
–
49.1
53.1
57.2
58.9
59.6
83.8
30.3
63.9
56.9875
+ Per-domain RL
Full
52.3
58.4
62.2
63.4
63.7
89.8
49.0
72.2
63.875
LoRA
51.5
57.1
61.4
63.2
63.2
90.8
50.2
72.3
63.7125
Fusion paradigms
Merge
Full
54.9
61.6
63.4
64.7
63.9
88.3
41.0
71.8
63.7
Table 1: Main results for the three multi-domain RLVR fusion paradigms across five target domains. The base model and per-domain RL serve as baselines, and all scores are reported as mean@32 . Column shading groups the eight benchmarks by domain, and Avg. reports their mean. Boldface marks the best fusion result in each column. Each Per-domain RL row aggregates five models rather than reporting a single model: each entry gives the performance of the expert trained for the corresponding benchmark domain. Appendix C reports complete cross-domain results for all experts. Tuning distinguishes full-parameter from LoRA experts and, for Merge, identifies the expert set being merged; Mix RL and MOPD are full-parameter throughout.
Figure 2: Cross-domain effects of per-domain RLVR. Each panel tracks all five experts on one benchmark throughout their respective training runs. The curve for the expert trained on the corresponding domain is bolded. Curves report mean@4 performance.
Figure 3: Geometry of the five task vectors τi . (a,b) Pairwise cosine similarity at layer 16 ; the unit diagonal is left blank. (c) The same quantity across all layers, averaged over three reasoning pairs (solid) and seven pairs involving instruction following or agent (dashed). The shaded band (right axis) gives each layer’s squared displacement relative to its parameter share, exceeding 1 where RL moves a layer more than its size predicts. (d) Layer- 16 cosine similarity versus cross-domain transfer measured in behavior, symmetrized across directions to give one point per pair and backbone.
Figure 4: Training performance of per-domain RL, Mix RL and MOPD. Merge is omitted because it has no training trajectory. Within each row, the first pair of panels reports AIME 2025/26 and the second reports IFBench. The two panels in each pair plot the same curves against optimization steps and the number of prompts sampled from the corresponding domain.
Requirements
Fusion GPU-h
Total GPU-h
Total ( × )
Experts
Supervision
Deploy
4B
8B
4B
8B
4B
8B
Per-domain RL
–
Verifiable reward
5
–
–
5,220
4,670
1.00
1.00
Fusion paradigms
Merge
Required
None
1
≈0
≈0
5,220
4,670
1.00
1.00
Mix RL
Not required
Verifiable reward
1
3,042
3,138
3,042
3,138
0.58
0.67
MOPD
Required
Teacher logprobs
1
0, 741
0, 899
5,960
5,569
1.14
1.19
Table 2: Requirements and costs of the three paradigms. Experts indicates whether the paradigm requires the five per-domain models; Supervision denotes the learning signal; and Deploy indicates the number of models served at inference. Fusion reports the cost of the fusion stage alone, while Total additionally includes the cost of training the five experts when required. The final two columns report Total relative to per-domain RL. Appendix B reports individual expert costs in detail.
Figure 5: Solution coverage in math and capability retention on the 4B backbone. (a,b) pass@k on AIME 2025/26, shown as absolute performance and paired differences from the base model. (c,d) Held-out SimpleQA-Verified and AA-LCR performance, with dashed lines marking the base model.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Per-domain RL
Mix RL
MOPD
Algorithm
GRPO
GRPO
Sampled-Token OPD
Rollout batch size
128
128
264
Mini batch size
128
128
264
Rollout n
16
16
4
Maximum prompt length
5,120
5,120
5,120
Maximum response length
16,384
16,384
16,384
Appendix
Table 3: Implementation details and training hyperparameters for per-domain RL, Mix RL and MOPD. Merge is omitted because it requires no training.
4B GPU-h
8B GPU-h
Per-domain RL
Math
1,824
1,847
Science
0, 849
0, 523
Code
1,435
0, 670
Instruction following
0, 412
0, 633
Agent
0, 700
0, 997
Appendix
Table 4: Per-run training costs underlying the aggregate costs in Table 2 . Costs are measured through step 300 for the five experts and MOPD, and through step 800 for Mix RL.
Expert
Tuning
AIME25
AIME26
GPQA
LCBv5
LCBv6
IFEval
IFBench
BFCLv3
Avg.
Qwen3-4B-Instruct-2507
Base
–
49.1
53.1
57.2
58.9
59.6
83.8
30.3
63.9
56.9875
Full
52.3
58.4
61.8
59.6
60.6
84.1
30.8
63.1
58.8375
Math
LoRA
51.5
57.1
59.9
59.6
60.0
83.9
30.9
62.6
58.1875
Full
50.4
57.4
62.2
59.6
60.9
83.9
30.8
64.2
58.675
Science
LoRA
47.0
54.7
61.4
59.4
59.7
83.7
30.5
63.4
57.475
Appendix
Table 5: Performance of each per-domain expert across all evaluation benchmarks.
Figure 6: Training score of each per-domain expert on its own domain. Curves are smoothed by a moving average, with the band indicating standard deviation over the same window.
Figure 7: Accuracy and response length of the Qwen3-8B TA merge versus the scaling coefficient λ , averaged over the evaluation sets across domains.
Method
AIME25
AIME26
GPQA
LCBv5
LCBv6
IFEval
IFBench
BFCLv3
Avg.
Qwen3-4B-Instruct-2507
Base
49.1
53.1
57.2
58.9
59.6
83.8
30.3
63.9
56.9875
+ Per-domain RL (Full)
52.3
58.4
62.2
63.4
63.7
89.8
49.0
72.2
63.875
+ Per-domain RL (LoRA)
51.5
57.1
61.4
63.2
63.2
90.8
50.2
72.3
63.7125
Full-parameter merge
Average
50.4
55.7
60.7
59.7
60.2
84.6
32.4
65.9
58.7
Appendix
Table 6: Comparison of merging methods. Within each group, methods combine the same five expert task vectors. Concat and SVD apply only to LoRA experts. TA and DARE-TA use λ=0.6 . For each backbone, the two + Per-domain RL rows report scores for the full-parameter and LoRA experts on their respective domains. Methods marked with † are reported as Merge in Table 1 .