Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning
Organizations: University of Virginia, Department of Computer Engineering · Baidu, Inc. · University of California, Los Angeles · University of Birmingham
Abstract
Long-to-Short (L2S) model merging seeks to combine the strong reasoning capability of long chain-of-thought models with the concise style of base models, but existing methods that rely on linear interpolation and first-order activation statistics fail to capture how the loss landscape responds to parameter displacement, leading to degraded accuracy on hard tasks. We propose Low-Rank Hierarchical Merging (LHM), a training-free framework that replaces heuristic statistics with lightweight second-order curvature information. LHM consists of three components: (1) a low-rank Hessian approximation that efficiently estimates per-layer Hessian traces via Hutchinson's estimator on random inputs, eliminating the need for task-specific data; (2) a Layer Second-Order Sensitivity (LSOS) index that combines Hessian trace with task-vector norm to quantify per-layer fusion difficulty; and (3) a Dual-Constraint Coefficient Mapping (DCCM) that converts LSOS scores into layer-adaptive merge coefficients while jointly satisfying a fusion-error bound and a global length-reduction target. Experiments across six mathematical benchmarks on 1.5B and 7B models show that LHM outperforms state-of-the-art methods such as ACM-TIES and ORCA, achieving over 4% accuracy gain and up to 93.1% reduction in output length relative to the long-CoT model.
Figures & tables
| Model | Accuracy (%) | Response Length | ||||||||||||
| GSM | M500 | Min. | Oly. | Coll. | AIME | Avg. | GSM | M500 | Min. | Oly. | Coll. | AIME | Avg. | |
| Qwen2.5-Math-1.5B | 75.2 | 62.8 | 8.1 | 9.4 | 16.3 | 6.7 | 29.8 | 823 | 1102 | 1028 | 1284 | 1073 | 1215 | 1088 |
| R1-Distill-1.5B | 82.4 | 77.9 | 22.6 | 35.2 | 35.8 | 30.0 | 47.3 | 2123 | 4142 | 4427 | 7864 | 4284 | 8814 | 5276 |
| Model Soup | 76.8 | 68.2 | 14.5 | 22.4 | 26.3 | 16.7 | 37.5 | 1523 | 2784 | 2934 | 4986 | 2912 | 5847 | 3498 |
| Task Arithmetic | 78.6 | 71.2 | 18.3 | 29.5 | 31.2 | 20.0 | 41.5 | 1342 | 2314 | 2432 | 4218 | 2341 | 5126 | 2962 |
| TIES | 79.8 | 72.4 | 19.7 | 30.8 | 32.1 | 20.0 | 42.5 | 1184 | 2187 | 2253 | 4073 | 2184 | 4891 | 2795 |
| Model | Accuracy (%) | Avg. Length | ||||||
| GSM | M500 | Oly. | Avg. | GSM | M500 | Oly. | Avg. | |
| Qwen2.5-Math-7B | 57.4 | 51.0 | 16.7 | 41.7 | 1012 | 1293 | 1930 | 1412 |
| R1-Qwen-7B | 89.5 | 87.8 | 48.3 | 75.2 | 1050 | 2901 | 5781 | 3244 |
| TA | 89.5 | 84.6 | 45.9 | 73.3 | 557 | 1485 | 2398 | 1480 |
| AIM | 91.5 | 84.6 | 48.4 | 74.8 | 598 | 1665 | 2428 | 1564 |
| ACM-TA | 90.9 | 84.7 | 46.4 | 74.0 | 579 | 1500 | 2342 | 1474 |
| Configuration | Avg. Accuracy (%) | Avg. Length |
| TA Baseline | 41.5 | 2962 |
| + Low-Rank Hessian (L) | 43.2 | 2847 |
| + LSOS Index (L+S) | 45.9 | 2631 |
| + DCCM (Full LHM-TA) | 46.5 | 2239 |
| AIM Baseline | 43.8 | 2696 |
| + Low-Rank Hessian (L) | 44.9 | 2563 |
Language Model Merging
Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.
See all 114 papers