GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning

Lam So; Canhui Wu; Han Lin

cs.CLAug 26, 2026

GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning

Authors: Lam So, Canhui Wu, Han Lin

Organizations: Peking University · Xi’an Jiaotong University

Abstract

Reasoning-oriented large language models often achieve strong problem-solving performance by generating long chains of thought, but this behavior substantially increases inference cost and latency. In contrast, instruction-tuned models tend to answer more concisely, yet often lack comparable reasoning ability. This accuracy-efficiency mismatch motivates a lightweight approach that combines the strengths of both models without full model retraining. In this paper, we propose GRIP (Granular Reward-guided Interpolation of Parameters), a reward-guided parameter interpolation framework for efficient reasoning. Given a reasoning model and an instruction model with identical architectures, GRIP assigns learnable interpolation ratios to individual modules and optimizes only these ratios while keeping both source models frozen. The interpolation ratios are trained with a reward signal that favors responses that are both correct and concise. Experiments show that GRIP achieves a better accuracy-efficiency trade-off than fixed or search-based merging baselines and further reveals module-wise fusion patterns associated with efficient reasoning.

Figures & tables

Figure 1: Overview of GRIP. GRIP learns module-wise sigmoid-controlled fusion ratios between a reasoning model and a non-thinking model, and updates them with an RL-based objective using rewards based on correctness and response length.

Methods	AIME25		MATH500		GSM8K		GPQA-D †		LCB †		Avg
Methods	Acc.	Tok.	Acc.	Tok.	Acc.	Tok.	Acc.	Tok.	Acc.	Tok.	Acc.	Tok.
Qwen3
Thinking	73.3	19630	89.4	5802	94.5	1350	66.7	9057	56.0	18439	76.0 (+0.0)	10856 (100.0%)
Instruct	36.7	10555	85.6	1623	92.6	304	46.5	459	30.3	2824	58.3 (-17.7)	3153 (29.0%)
Linear	73.3	15977	88.0	4339	93.6	1223	70.3	8591	53.1	15814	75.7 (-0.3)	9189 (84.6%)
SLERP	76.7	16467	89.0	4442	94.2	1231	69.2	8449	53.7	15770	76.5 (+0.5)	9272 (85.4%)

Table 1: Main comparison of accuracy and inference efficiency on five reasoning benchmarks. Acc. denotes task accuracy, and Tok. denotes the average number of generated tokens per example. The Avg column reports the mean performance across AIME25, MATH500, GSM8K, GPQA-D, and LCB. Best results are highlighted in bold. † marks out-of-domain benchmarks: GRIP and the CMA-ES baseline are both trained on math data only (DeepScaleR-preview).

Figure 2: Module-targeted coefficient sweeps on Qwen3-4B. The attention sweep is nearly flat in both macro pass@1 and length, while the FFN sweep changes both metrics substantially. The global sweep follows the FFN trend but pays a larger length cost because all modules are moved together.

Figure 3: Inter-layer standard deviation of attention and FFN fusion ratios during training. Both start near zero, grow rapidly in the first few hundred steps, and stabilize around $\approx 0.30$ , showing that training breaks the initially uniform blend.

Figure 4: Learned per-layer fusion ratios after $750$ training steps for attention and FFN modules.

Figure 5: Mean fusion ratio $\alpha$ over training, with the $36$ layers grouped into early ( $0$ – $8$ ), middle ( $9$ – $26$ ), and late ( $27$ – $35$ ) bands; shaded regions show within-band standard deviation. Left: attention. Right: FFN. The early band stays close to the $0.8$ initialization, while middle and late bands move substantially toward Instruct, and the FFN middle band moves more aggressively than any attention band.

Figure 6: RL optimization is smoother than CMA-ES. Per-coordinate RMS shift between adjacent updates, measured every $10$ gradient steps for RL and every generation for CMA-ES.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Mean response length during GRIP training on the Qwen3-4B pair through step $750$ .

Figure 8: Raw reward during GRIP training on the Qwen3-4B pair through step $750$ .

Method	Avg Acc.	Avg Tok.
Layer-wise ( $36+2$ params)	73.5	7571
Module-wise ( $72+2$ params)	76.5	7930

Appendix

Table 2: Ablation comparing layer-wise and module-wise interpolation on the same five evaluation benchmarks. The layer-wise result is evaluated at step

230

, while the module-wise result is the GRIP configuration reported in Table 1 . Module-wise interpolation improves average accuracy by

3.0

points while using a comparable number of generated tokens.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 371 papers

RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format
Feb 26, 2026Zhehao Huang, Yuhang Liu, Baijiong Lin +5Instruction Following Language Model Merging
Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) · MoE Key Laboratory of System Control and Information Processing (Shanghai)
Giving Credit Where It's Due: Redundancy-Aware Learning for Efficient Reasoning
Sep 22, 2026Yuqing Zhou, Hong Wang, Manqing Mao +8RL for Language Model Reasoning Step-Level Credit Assignment in RL
George Mason University · Amazon, Inc
Hint Tuning: Less Data Makes Better Reasoners
May 9, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Overthinking in Language Models LLM Fine-Tuning
University of Electronic Science and Technology of China · Xiaohongshu Inc. · National University of Singapore

Feb 26, 2026J/K move · Enter open · S save

RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format

Zhehao Huang, Yuhang Liu, Baijiong Lin, Yixin Lou+4

Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) · MoE Key Laboratory of System Control and Information Processing (Shanghai)

Instruction Following Language Model Merging Large Reasoning Models Task Vector Merging LLM Reasoning

Open paper AskPDF arXiv

Large reasoning models (LRMs) excel at a long chain of reasoning but often fail to faithfully follow instructions regarding output format, constraints, or specific requirements. We investigate whether this gap can be closed by integrating an instruction-tuned model (ITM) into an LRM. Analyzing their differences in parameter space, namely task vectors, we find that their principal subspaces are nearly orthogonal across key modules, suggesting a lightweight merging with minimal interference. However, we also demonstrate that naive merges are fragile because they overlook the output format mismatch between LRMs (with explicit thinking and response segments) and ITMs (answers-only). We introduce RAIN-Merging (Reasoning-Aware Instruction-attention guided Null-space projection Merging), a gradient-free method that integrates instruction following while preserving thinking format and reasoning performance. First, with a small reasoning calibration set, we project the ITM task vector onto the null space of forward features at thinking special tokens, which preserves the LRM's structured reasoning mechanisms. Second, using a small instruction calibration set, we estimate instruction attention to derive module-specific scaling that amplifies instruction-relevant components and suppresses leakage. Across four instruction-following benchmarks and nine reasoning & general capability benchmarks, RAIN-Merging substantially improves instruction adherence while maintaining reasoning quality. The gains are consistent across model scales and architectures, translating to improved performance in agent settings.

Abstract

Efficient Language Model Reasoning

Explore similar work

RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format

Giving Credit Where It's Due: Redundancy-Aware Learning for Efficient Reasoning

Hint Tuning: Less Data Makes Better Reasoners