Organizations: Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) · MoE Key Laboratory of System Control and Information Processing (Shanghai)
Large reasoning models (LRMs) excel at a long chain of reasoning but often fail to faithfully follow instructions regarding output format, constraints, or specific requirements. We investigate whether this gap can be closed by integrating an instruction-tuned model (ITM) into an LRM. Analyzing their differences in parameter space, namely task vectors, we find that their principal subspaces are nearly orthogonal across key modules, suggesting a lightweight merging with minimal interference. However, we also demonstrate that naive merges are fragile because they overlook the output format mismatch between LRMs (with explicit thinking and response segments) and ITMs (answers-only). We introduce RAIN-Merging (Reasoning-Aware Instruction-attention guided Null-space projection Merging), a gradient-free method that integrates instruction following while preserving thinking format and reasoning performance. First, with a small reasoning calibration set, we project the ITM task vector onto the null space of forward features at thinking special tokens, which preserves the LRM's structured reasoning mechanisms. Second, using a small instruction calibration set, we estimate instruction attention to derive module-specific scaling that amplifies instruction-relevant components and suppresses leakage. Across four instruction-following benchmarks and nine reasoning & general capability benchmarks, RAIN-Merging substantially improves instruction adherence while maintaining reasoning quality. The gains are consistent across model scales and architectures, translating to improved performance in agent settings.
Figures & tables
Figure 1: An overview of RAIN-Merging . In the case, the LRM arrives at the correct solution but ignores the required format and specific code. To preserve the reasoning structure, we perform training-free merging by combining a task vector projected onto the null space of the thinking format with instruction-attention guided coefficients. The merged model remains correct while satisfying the specified constraints. See Sec. 3 for details.
Figure 2: Principal subspace cosine similarity between LRM and ITM task vectors for each layer and submodule. The similarities are consistently low ( <0.1 ).
Figure 3: Two stages of our RAIN-Merging pipeline. (a) For each submodule, the ITM task vector is projected onto the null space preventing shifts in thinking format. (b) Given the instruction calibration set, we compute the instruction-attention score from attention outputs to obtain merging coefficients.
Method
Instruction Following
Reasoning & General
RT
IFEval
CELLO
Info
Complex
Avg.
Math
GPQA
Aider
Arena-
Avg.
Bench
Bench
Hard-v2
ITM
70.43
19.15
78.49
43.63
52.92
47.27
29.80
33.33
62.86
43.32
–
LRM
55.45
16.59
71.73
32.72
44.12
64.75
44.44
29.63
65.29
51.03
–
SFT
62.48
17.11
68.58
32.15
45.08
62.57
41.92
28.89
64.67
49.51
120.32
Data-free Merging
Table 1: Comprehensive comparison of instruction following and reasoning & general capabilities. We merge Qwen2.5-7B-Instruct (ITM) into DeepSeek-R1-Distill-Qwen-7B (LRM) and compare our RAIN-Merging against multiple merging methods as well as SFT trained on the same calibration data. “Avg.” denotes the average over all subsets. “RT” reports the run-time for merging or training in minutes. The best and second-best results are highlighted in bold and underlined , respectively.
Model
Instruction Following
Reasoning & General
IFEval
CELLO
Info
Complex
Avg.
Math
GPQA
Aider
Arena-
Avg.
Bench
Bench
Hard-v2
Qwen2.5-1.5B-Instruct
36.78
19.04
64.76
27.83
37.10
31.77
25.76
16.30
38.45
28.07
DeepSeek-R1-Distill-Qwen-1.5B
39.00
16.03
55.29
21.54
32.97
41.62
29.29
14.07
39.73
31.18
Qwen2.5-1.5B-RAIN-Merging
41.59
16.51
58.18
23.62
34.97
45.87
33.33
14.81
40.93
33.74
(relative gain)
+6.64%
+2.98%
+5.23%
+9.63%
+6.09%
+10.21%
+13.79%
+5.26%
+3.02%
+8.20%
Table 2: Merging performance and relative gains of RAIN-Merging across model three scales and two architectures. We merge the corresponding ITM into the LRM with base models: Qwen2.5-1.5B/14B/32B, and Llama-3.1-8B. “Avg.” denotes the average over all subsets. For each scale, the subsequent “ (relative gain) ” row reports the relative improvement of our method over the LRM, highlighted in green.
Figure 6Figure 7
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Principal subspace cosine similarity between DeepSeek-R1-Distill-Qwen-1.5B (LRM) and Qwen2.5-1.5B-Instruct (ITM) task vectors for each layer and submodule.
Figure A2: Principal subspace cosine similarity between DeepSeek-R1-Distill-Qwen-14B (LRM) and Qwen2.5-14B-Instruct (ITM) task vectors for each layer and submodule.
Domain
Raw size
Calibration
Validation
Math
93,700
50
50
Code
83,100
50
50
Science
173,000
50
50
Total
349,800
150
150
Appendix
Table A1: Reasoning calibration set construction from Mixture-of-Thoughts . We uniformly sample 50 examples per domain for calibration and 50 for validation. Raw sizes are taken from the official dataset composition page.
Figure A3: A sample illustration in our instruction calibration set.
Benchmark
Size
Constraint
Composition Type
Verification
Evaluation
Taxonomy
And
Chain
Selection
Nested
Code-Exec.
LLM-as-Judge
Aggregation
IFEval
541
25
✓
–
–
–
✓
–
strict_prompt_level_accuracy
CELLO
523
4
✓
✓
–
–
✓
–
average
InfoBench
500
5
✓
✓
–
–
–
✓
DRFR
ComplexBench
1,150
19
✓
✓
✓
✓
✓
✓
dependency-aware DRFR
Appendix
Table A2: Instruction-following benchmarks. We list dataset size, constraint taxonomy, composition types, verification, and aggregation strategy.
AIME2025
AIME2024
AMC23
GSM8K
Math500
MinervaMath
# Test samples
30
30
40
1,319
500
272
Appendix
Table A3: Test set sizes of the six math benchmarks used in our mathematical reasoning (Math) evaluation.
Method
Hyper-parameters
7B
Task Arithmetic
λ=1.0
SLERP
λ=1.0
Karcher
λ=1.0
TIES
k=0.8,λ=0.8
DARE-TIES
p=0.3,k=0.5,λ=1.2
Appendix
Table A4: The hyperparameters of various merging methods in Tab. 1 . λ means the global scaling coefficient in merging. k denotes the trim ratio in TIES-Merging. p means the drop rate in DARE merging. τ is sharpness the ACM. ρ is the pruning ratio in LEWIS. ω means the balance factor in AIM.
Method
Hyper-parameters
1.5B
7B
8B
14B
RAIN-Merging
λ=1.0
λ=1.0
λ=0.9
λ=1.0
Appendix
Table A5: The hyperparameters of RAIN-Merging in different model sizes. λ means the global scaling coefficient in RAIN-Merging.
Method
AIME2025
AIME2024
AMC23
GSM8K
Math500
Minerva
Avg.
ITM
10.00
10.00
67.50
86.66
73.80
35.66
47.27
LRM
30.00
50.00
80.00
91.36
89.00
48.16
64.75
SFT
33.33
43.33
75.00
90.75
87.40
45.59
62.57
Task Arithmetic
30.00
50.00
80.00
90.75
89.00
45.59
64.22
SLERP
30.00
46.67
77.50
91.05
89.20
48.53
63.82
Karcher
30.00
50.00
80.00
90.98
89.20
48.90
64.85
Appendix
Table A6: Math benchmark results under the same configuration as in Tab. 1 . “Avg.” denotes the average over all math benchmarks. The best and second-best results are highlighted in bold and underlined , respectively.
Method
AIME2025
AIME2024
AMC23
GSM8K
Math500
Minerva
Average
Qwen2.5-1.5B-Instruct
3.33
0.00
30.00
75.44
59.40
22.43
31.77
DeepSeek-R1-Distill-Qwen-1.5B
20.00
13.33
42.50
73.77
71.80
28.31
41.62
Qwen2.5-1.5B-RAIN-Merging
20.00
16.67
60.00
76.36
72.40
29.78
45.87
Llama-3.1-8B-Instruct
3.33
6.67
20.00
81.65
67.60
34.26
35.59
DeepSeek-R1-Distill-Llama-8B
30.00
40.00
75.00
90.52
80.40
45.37
60.21
Llama-3.1-8B-RAIN-Merging
30.00
43.33
77.50
90.30
82.80
47.79
61.95
Appendix
Table A7: Math benchmarks results under the same configuration as in Tab. 2 . “Avg.” denotes the average over all math benchmarks.
Figure A4: Instruction following and reasoning & generage performance of our RAIN-Merging using different global scalar λ . The configuration is the same as in Tab. 2 . The performance is measured by the average of the instruction following and reasoning & general capability benchmarks. The marked result is our choice in the experiments.
Figure A5: Instruction following and reasoning & generage performance of our RAIN-Merging using different reasoning calibration set sizes. The x axis represents the size of the reasoning calibration set with exponential scale. The configuration is the same as in Tab. 2 . The performance is measured by the average of the instruction following and reasoning & general capability benchmarks. The marked result is our choice in the experiments.
Figure A6: Heatmap of merging coefficients by our Stage 2 for each layer and attention head of DeepSeek-R1-Distill-Qwen-7B.
Method
Instruction Following
Reasoning & General
IFEval
CELLO
Info
Complex
Avg.
Math
GPQA
Aider
Arena-
Avg.
Bench
Bench
Hard-v2
LRM
55.45
16.59
71.73
32.72
44.12
64.75
44.44
29.63
65.29
51.03
RAIN-Merging (Rule)
63.22
19.03
74.53
35.66
48.11
68.75
54.55
33.33
65.73
55.59
RAIN-Merging (Open)
62.92
19.24
74.89
35.67
48.15
65.14
49.49
31.11
64.67
52.59
RAIN-Merging (Rule+Open)
64.03
19.63
75.64
36.70
49.00
67.43
53.03
35.56
65.29
55.32
Appendix
Table A8: Performance of RAIN-Merging with different instruction calibration sets in Stage 2. Rule is our original instruction calibration set from IFEval, Open is the new set from InforBench, Rule+Open simply concatenates both sets. We merge Qwen2.5-7B-Instruct (ITM) into DeepSeek-R1-Distill-Qwen-7B (LRM) under the same configuration as Tab. 1 .
Model
IFBench
XIFBench
EIFBench
Average
Qwen2.5-7B-Instruct
27.89
83.35
55.62
55.62
DeepSeek-R1-Distill-Qwen-7B
17.69
72.93
45.31
45.31
Qwen2.5-7B-RAIN-Merging
19.39
76.32
47.85
47.85
(relative gain)
+9.62%
+4.65%
+5.62%
+5.62%
Appendix
Table A9: Merging performance and relative gains of RAIN-Merging on three new instruction-following benchmarks. We merge the Qwen2.5-7B family under the same configuration as Tab. 1 . The subsequent “ (relative gain) ” row reports the relative improvement of our method over the LRM, highlighted in green.
Figure A7: Instruction-type accuracy comparison between the LRM (DeepSeek-R1-Distill-Qwen-7B) and RAIN-Merging (Qwen2.5-7B-RAIN-Merging) on IFEval. The results highlight the largest improvements on instructions like response_language , number_words , and no_comma .
Model
IFEval(200) HAcc
IFEval-paraphrase HAcc
Robustness
Qwen2.5-7B-Instruct
65.71
64.16
0.98
DeepSeek-R1-Distill-Qwen-7B
57.86
50.85
0.88
Qwen2.5-7B-RAIN-Merging
61.29
61.81
1.01
Appendix
Table A10: Robustness to paraphrased instructions on IFEval. We merge the Qwen2.5-7B family under the same configuration as Tab. 1 . HAcc is hard accuracy (%). Robustness is defined as HAcc(paraphrase)/HAcc(original)
Large Reasoning Models (LRMs) have demonstrated impressive capabilities in many tasks, yet they struggle with reliably following multiple instructions, either by failing to satisfy individual constraints or by struggling to balance competing constraints simultaneously. We formalize this challenge as the Constraint Adherence Problem (CAP). This paper introduces a novel framework that addresses CAP by representing instructions as a structured knowledge graph of constraints. Our approach, Constraint Relationship Graph Completion (CRGC), explicitly models relationships between constraints, identifies adherence challenges, and discovers ``bridge constraints'' that help the model better focus on and reconcile requirements. Bridge constraints act as auxiliary instructions that make primary constraints more salient and compatible. Unlike existing approaches that enhance instruction following through general training methods, CRGC specifically improves constraint satisfaction by leveraging the model's own knowledge to create better pathways for generation. Experiments across three popular instruction following datasets demonstrate that our approach reduces constraint violations by 39% compared to standard prompting while maintaining reasoning abilities of large reasoning models.
Zhengyi Zhao, Shubo Zhang, Huimin Wang +7
The Chinese University of Hong Kong · Tencent Jarvis Lab · Westlake University +2
Reasoning-oriented large language models often achieve strong problem-solving performance by generating long chains of thought, but this behavior substantially increases inference cost and latency. In contrast, instruction-tuned models tend to answer more concisely, yet often lack comparable reasoning ability. This accuracy-efficiency mismatch motivates a lightweight approach that combines the strengths of both models without full model retraining. In this paper, we propose GRIP (Granular Reward-guided Interpolation of Parameters), a reward-guided parameter interpolation framework for efficient reasoning. Given a reasoning model and an instruction model with identical architectures, GRIP assigns learnable interpolation ratios to individual modules and optimizes only these ratios while keeping both source models frozen. The interpolation ratios are trained with a reward signal that favors responses that are both correct and concise. Experiments show that GRIP achieves a better accuracy-efficiency trade-off than fixed or search-based merging baselines and further reveals module-wise fusion patterns associated with efficient reasoning.
Reasoning language models (RLMs) demonstrate impressive performance by leveraging test-time compute in the form of reasoning tokens. However, this behavior makes adapting RLMs to new domains challenging and expensive. The reason is that further training can disturb the learned behavior and degrade model performance. This makes it difficult to leverage supervised fine-tuning data with human-written solutions: although it contains high-quality annotations, it lacks reasoning tokens. In this work, we show how, despite this challenge, such data can be used efficiently for RLM adaptation. For this, we first use standard instruction tuning. Next, we leverage model merging to combine the instruction-tuned model with the original RLM, picking the merging ratio such that the resulting model's reasoning behavior on the target domain is recovered. We evaluate our method across four RLMs on coding and text summarization tasks, where it improves target-task performance by up to 11.0% while preserving reasoning behavior and limiting the out-of-distribution score degradation to on average 0.7%. Importantly, our adaptations are efficient and economical, costing less than USD \10$ per model.
Yu-Du Feng, Niels Mündler-Sasahara, Mark Vero +1
Department of Computer Science, ETH Zurich, Zurich, Switzerland.