Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning
Authors: Zeqiu Yu, Xuesheng Zhang, Wenxiao Zhao, Shixiao Wang
Organizations: University of Virginia, Department of Computer Engineering · Baidu, Inc. · University of California, Los Angeles · University of Birmingham
Long-to-Short (L2S) model merging seeks to combine the strong reasoning capability of long chain-of-thought models with the concise style of base models, but existing methods that rely on linear interpolation and first-order activation statistics fail to capture how the loss landscape responds to parameter displacement, leading to degraded accuracy on hard tasks. We propose Low-Rank Hierarchical Merging (LHM), a training-free framework that replaces heuristic statistics with lightweight second-order curvature information. LHM consists of three components: (1) a low-rank Hessian approximation that efficiently estimates per-layer Hessian traces via Hutchinson's estimator on random inputs, eliminating the need for task-specific data; (2) a Layer Second-Order Sensitivity (LSOS) index that combines Hessian trace with task-vector norm to quantify per-layer fusion difficulty; and (3) a Dual-Constraint Coefficient Mapping (DCCM) that converts LSOS scores into layer-adaptive merge coefficients while jointly satisfying a fusion-error bound and a global length-reduction target. Experiments across six mathematical benchmarks on 1.5B and 7B models show that LHM outperforms state-of-the-art methods such as ACM-TIES and ORCA, achieving over 4% accuracy gain and up to 93.1% reduction in output length relative to the long-CoT model.
Figures & tables
Figure 1: (Left) Accuracy–length Pareto plot across merging methods on the 1.5B benchmark suite. LHM (star) dominates all baselines by simultaneously achieving higher accuracy and shorter responses. (Right) Relative accuracy degradation when the merge coefficient is reduced by 0.2 for high-LSOS versus low-LSOS layers, confirming that second-order sensitivity identifies layers critical to reasoning performance.
Figure 2: (Left) LSOS index distribution across the 28 transformer layers of DeepSeek-R1-Distill-Qwen-7B. High-LSOS layers cluster near the beginning and end of the network, correlating with known reasoning-critical regions. (Right) Comparison of Hutchinson Hessian trace estimates (rank k=16 ) against a finite-difference ground truth, showing <2% relative error with negligible computational overhead.
Figure 3: Overall framework of Low-Rank Hierarchical Merging (LHM). The framework then comprises three modules: (1) Low-Rank Hessian Approximation, which estimates loss-landscape curvature using Rademacher random vectors to compute per-layer Hessian traces; (2) LSOS Index, which combines these traces with the ℓ2 norms of the task vector via element-wise multiplication to evaluate layer importance; and (3) Dual-Constraint Coefficient Mapping (DCCM), which applies min–max normalization and dual orthogonal constraints (balancing accuracy preservation and length reduction) to generate layer-wise merge coefficients λi . The merged model θmerged preserves strong reasoning accuracy while producing substantially shorter outputs.
Figure 4: (Left) Normalized Hessian trace estimation error as a function of the rank k , showing rapid convergence beyond k=16 . (Right) Wall-clock time for Hessian trace computation at different ranks compared with the full finite-difference baseline ( × ), confirming O(dk) scaling.
Model
Accuracy (%) ↑
Response Length ↓
GSM
M500
Min.
Oly.
Coll.
AIME
Avg.
GSM
M500
Min.
Oly.
Coll.
AIME
Avg.
Qwen2.5-Math-1.5B
75.2
62.8
8.1
9.4
16.3
6.7
29.8
823
1102
1028
1284
1073
1215
1088
R1-Distill-1.5B
82.4
77.9
22.6
35.2
35.8
30.0
47.3
2123
4142
4427
7864
4284
8814
5276
Model Soup
76.8
68.2
14.5
22.4
26.3
16.7
37.5
1523
2784
2934
4986
2912
5847
3498
Task Arithmetic
78.6
71.2
18.3
29.5
31.2
20.0
41.5
1342
2314
2432
4218
2341
5126
2962
TIES
79.8
72.4
19.7
30.8
32.1
20.0
42.5
1184
2187
2253
4073
2184
4891
2795
Table 1: Results on 1.5B models (Qwen2.5-Math-1.5B base + R1-Distill-Qwen-1.5B reasoning). Abbreviations: Min. (Minerva Math), Oly. (OlympiadBench), Coll. (CollegeMath). “Avg.” is the mean across all six benchmarks. Response Length denotes the mean number of output tokens.
Model
Accuracy (%) ↑
Avg. Length ↓
GSM
M500
Oly.
Avg.
GSM
M500
Oly.
Avg.
Qwen2.5-Math-7B
57.4
51.0
16.7
41.7
1012
1293
1930
1412
R1-Qwen-7B
89.5
87.8
48.3
75.2
1050
2901
5781
3244
TA
89.5
84.6
45.9
73.3
557
1485
2398
1480
AIM
91.5
84.6
48.4
74.8
598
1665
2428
1564
ACM-TA
90.9
84.7
46.4
74.0
579
1500
2342
1474
Table 2: Results on 7B models (Qwen2.5-Math-7B base + R1-Distill-Qwen-7B reasoning). Three representative benchmarks are shown for space; Avg. is over all six benchmarks.
Configuration
Avg. Accuracy (%)
Avg. Length
TA Baseline
41.5
2962
+ Low-Rank Hessian (L)
43.2
2847
+ LSOS Index (L+S)
45.9
2631
+ DCCM (Full LHM-TA)
46.5
2239
AIM Baseline
43.8
2696
+ Low-Rank Hessian (L)
44.9
2563
Table 3: Ablation on 1.5B models: each row cumulatively adds one LHM component. LSOS yields the largest accuracy gain, while DCCM yields the largest length reduction.
Figure 5: Ablation study visualisation. (Left) Average accuracy across configurations on TA and AIM baselines, each with incremental addition of LHM components. (Right) Corresponding average response lengths, showing that DCCM is the primary driver of length reduction.
Figure 6: (Left) Average accuracy and average response length as a function of the Hessian rank k on the 1.5B benchmark suite. Performance stabilises at k≥16 . (Right) Pareto curves of accuracy vs. response length for different global target coefficients λˉ∈{0.3,0.4,0.5,0.6} .
Model merging has emerged as a practical approach to combine capabilities of specialized large language models (LLMs) without additional training. In the Long-to-Short (L2S) scenario, merging a base model with a long-chain-of-thought reasoning model aims to preserve reasoning accuracy while reducing output length. Existing methods rely on Task Arithmetic and its variants, which implicitly assume that model outputs vary linearly with the merging coefficient -- an assumption we show is systematically violated in L2S settings. We provide the first theoretical justification for layer-adaptive merging: we prove that merging error is bounded by a term proportional to the per-layer Hessian norm (Proposition~1), and establish that the Fisher Information Matrix (FIM) is a principled, computable proxy for this bound via the Fisher-Hessian equivalence at local optima. Building on this theory, we propose FIM-Merging, which computes diagonal FIM using only random token inputs (no domain-specific calibration data required) and uses it to assign per-layer merging coefficients. On the 7B L2S benchmark, FIM-TIES achieves state-of-the-art performance on five out of six evaluation benchmarks, including a +6.2 point gain on MATH500 over ACM-TIES (90.2 vs.\ 84.0), while requiring no calibration data. On the 1.5B benchmark, FIM-TIES achieves an average accuracy of 47.3, surpassing the previous best ACM-TIES (43.3) by +3.9 points, while reducing average response length by 91.9% relative to the long-CoT model. Our framework also provides a unified theoretical explanation for why existing layer-adaptive methods such as ACM empirically outperform uniform merging.
Reasoning models achieve strong performance on complex problems by leveraging long chains of thought, but this deliberate reasoning incurs substantial inference-time cost. The Long-to-Short (L2S) reasoning problem seeks to preserve accuracy while reducing generated tokens. Yet, current training-free model merging approaches rely on brittle, fixed-hyperparameter arithmetic methods that force suboptimal compromises. We introduce Evo-L2S, a multi-objective evolutionary model merging framework that explicitly optimizes accuracy and output length to recover a Pareto front of merged models. To make this search computationally tractable, we propose an entropy-based subset sampling technique that substantially reduces fitness-estimation overhead. Across six mathematical reasoning benchmarks, Evo-L2S reduces reasoning length by over 50% at the 1.5B and 7B scales while preserving or improving problem-solving accuracy; at 14B, the steeper Pareto front reveals that attainable compression depends on the intrinsic compressibility of the reasoning model. Overall, Evo-L2S shows that reasoning models can be made substantially more concise while preserving strong problem-solving performance.
Mario Iacobelli, Adrian Robert Minut, Tommaso Mencattini +4
Independent Researcher · Sapienza University of Rome · EPFL +2
Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a Rank-1 Subspace phenomenon: while raw optimization steps oscillate violently, consecutive merged checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a river-valley landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose Extra-Merge, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.
Wenjie Zhou, Bohan Wang, Hongtao Zhang +3
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2