HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning

Donggyun Kim; Jack Lu; Chanwoo Kim; Mengye Ren; Seunghoon Hong

cs.CLOct 2, 2026

HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning

Authors: Donggyun Kim, Jack Lu, Chanwoo Kim, Mengye Ren, Seunghoon Hong

Organizations: KAIST School of Computing · NYU Courant Institute School of Mathematics, Computing, and Data Science

Abstract

Long-form thinking traces can substantially improve the multi-step reasoning performance of large language models (LLMs), but they introduce high inference-time overhead, with latency dominated by sequential decoding. We propose HyperThink, a text-to-parameter approach that amortizes this reasoning computation into a single query-conditioned parameter update: a lightweight hypernetwork reads the question and predicts updates to a small subset of the base LLM's parameters, while a vector-quantized decoder constrains them to a finite set of reusable patterns to improve robustness and transfer. Trained end-to-end on outputs from the base model itself, HyperThink eliminates long thinking traces at test time: after one hypernetwork forward pass, the adapted model generates a concise step-by-step solution and final answer without an intermediate trace, using far fewer tokens while retaining strong reasoning performance. Empirically, HyperThink improves the low-latency region of the accuracy-latency trade-off on mathematical and general reasoning tasks, with its strongest gains in the near-non-thinking regime.

Figures & tables

Figure 1: Conceptual comparison. (a) Non-Thinking mode produces the step-by-step response directly from the query. (b) Thinking mode produces a long thinking trace before producing the response, incurring high cost and latency. (c) HyperThink amortizes the reasoning computation into a query-conditioned parameter update

\Delta\theta

. The hypernetwork predicts this update in a single forward pass, enabling the adapted LLM to produce the response directly without a thinking trace.

Figure 2: HyperThink architecture. A frozen text encoder embeds the query

\mathbf{q}

into text embeddings, which are jointly encoded with learnable embeddings

U_{0}

by a bias encoder. The output bias tokens

U(\mathbf{q})

are decoded to final bias updates

\Delta\mathbf{b}(\mathbf{q})

by a VQ decoder that discretizes the contextualized parameter representations using a learned codebook

\mathcal{E}

Figure 3: Maximum likelihood objective for HyperThink . HyperThink predicts query-specific bias parameters and injects them into the base LLM, and is trained to steer the Non-Thinking response distribution into that of Thinking mode.

Reasoning Mode	GSM8K-Test			MATH-500
	Acc (%)	Pass@5 (%)	FLOPs (G)	Acc (%)	Pass@5 (%)	FLOPs (G)
Thinking Mode ( Qwen Team, 2025 )	73.81	87.41	2503.71	52.64	80.40	4535.70
Budget-Controlled Thinking Mode ( Qwen Team, 2025 )	36.15	56.94	502.36	43.76	61.48	1198.27
Native Non-Thinking Mode ( Qwen Team, 2025 )	58.82	80.14	407.66	46.84	69.00	962.35
System 2 Distillation ( Yu et al., 2024 )	61.91	75.06	458.21	31.40	50.80	838.52
TokenSkip ( Xia et al., 2025 )	61.64	81.50	437.18	37.12	57.40	1562.24

Table 1: Main results on GSM8K-Test and MATH-500 with Qwen3-0.6B backbone. We report average accuracy, Pass@5, and FLOPs for inference over 5 samples for all methods.

Reasoning Mode	GSM8K-Test			MATH-500
	Acc (%)	Pass@5 (%)	FLOPs (G)	Acc (%)	Pass@5 (%)	FLOPs (G)
Thinking Mode ( Bakouch et al., 2025 )	92.27	96.51	8340.31	88.56	93.60	28460.97
Budget-Controlled Thinking Mode ( Qwen Team, 2025 )	62.23	83.02	3748.67	46.80	63.20	5645.65
Native Non-Thinking Mode ( Bakouch et al., 2025 )	74.81	95.15	4900.09	70.20	84.20	7143.64
System 2 Distillation ( Yu et al., 2024 )	74.72	89.99	3400.70	62.00	79.60	5510.84
TokenSkip ( Xia et al., 2025 )	81.08	93.10	3015.82	54.56	74.00	10991.38

Table 2: Main results on GSM8K-Test and MATH-500 with SmolLM3-3B backbone. We report average accuracy, Pass@5, and FLOPs for inference over 5 samples for all methods.

Figure 4: Average Pass@5 versus latency of Qwen3-0.6B on math reasoning tasks.

Figure 5: Average Pass@5 versus latency of SmolLM3-3B on general reasoning tasks.

Figure 6: Average Pass@5 versus latency of Olmo-3-7B-Think on general reasoning tasks.

Bias Only	TTA	VQ Bottleneck	GSM8K-Test			MATH-500
Bias Only	TTA	VQ Bottleneck	Acc (%)	Pass@5 (%)	FLOPs (G)	Acc (%)	Pass@5 (%)	FLOPs (G)
✗	✗	✗	61.91	75.06	458.21	31.40	50.80	838.52
✓	✗	✗	60.88	82.11	440.72	45.56	66.20	1065.50
✓	✓	✗	59.74	80.14	425.39	43.32	66.90	1046.11
✓	✓	✓	61.06	82.11	477.28	48.76	73.40	1150.83

Table 3: Component-wise ablation study on HyperThink architecture.

Adaptable Parameters	GSM8K-Test		MATH-500
Adaptable Parameters	Acc (%)	FLOPs (G)	Acc (%)	FLOPs (G)
LoRA	60.71	503.44	46.92	1130.62
Prompt Tuning	60.18	506.68	43.52	879.76
Bias (Ours)	61.06	477.28	48.76	1150.83

Table 4: Ablation of adaptable parameters.

Figure 7: Subject distribution and top keywords per VQ code.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Reasoning Mode	AIME			LiveCodeBench
	Acc (%)	Pass@5 (%)	FLOPs (G)	Acc (%)	Pass@5 (%)	FLOPs (G)
Thinking Mode ( Bakouch et al., 2025 )	41.00	66.67	125796.78	42.80	63.50	136267.93
Budget-Controlled Thinking Mode ( Qwen Team, 2025 )	4.33	5.00	9541.25	9.05	14.25	7361.08
Native Non-Thinking Mode ( Bakouch et al., 2025 )	9.00	18.33	20458.29	19.75	34.00	10572.33
System 2 Distillation ( Yu et al., 2024 )	1.00	3.33	8443.08	7.80	12.75	6640.46
TokenSkip ( Xia et al., 2025 )	3.33	10.00	21778.68	9.90	15.50	23672.06

Appendix

Table 5: Main results on four general reasoning benchmarks with SmolLM3-3B backbone. We report average accuracy, Pass@5, and FLOPs for inference over 5 samples for all methods.

Reasoning Mode	AIME			LiveCodeBench
	Acc (%)	Pass@5 (%)	FLOPs (G)	Acc (%)	Pass@5 (%)	FLOPs (G)
Thinking Mode ( Team Olmo, 2025 )	69.00	83.33	301968.24	76.90	93.25	200170.05
Budget-Controlled Thinking Mode ( Qwen Team, 2025 )	3.33	6.67	19103.66	2.60	6.00	15693.76
Native Non-Thinking Mode ( Team Olmo, 2025 )	18.00	31.67	52997.12	40.35	60.25	28264.01
System 2 Distillation ( Yu et al., 2024 )	6.00	16.67	12918.68	14.10	22.50	16312.32
TokenSkip ( Xia et al., 2025 )	2.33	8.33	60068.92	11.65	21.00	56019.72

Appendix

Table 6: Main results on four general reasoning benchmarks with Olmo-3-7B-Think backbone. We report average accuracy, Pass@5, and FLOPs for inference over 5 samples for all methods.

Bias Layers	Qwen3-0.6B				SmolLM3-3B
Bias Layers	GSM8K-Test		MATH-500		GSM8K-Test		MATH-500
	Acc	Pass@5	Acc	Pass@5	Acc	Pass@5	Acc	Pass@5
All	63.44	83.62	40.84	61.20	82.24	92.27	59.36	75.40
First 8	62.52	81.73	43.84	65.00	82.27	92.72	63.52	77.40
First half	63.62	83.32	41.36	62.80	81.85	92.49	61.04	76.40
Last half	60.59	81.58	42.52	64.80	84.75	95.15	67.36	81.20

Appendix

Table 7: Sensitivity analysis on the target bias layers. We use the later-layer window as the default: last 8 blocks for Qwen3-0.6B and last half of blocks for SmolLM3-3B.

	GSM8K-Train		GSM8K-Test		MATH-500
Variant	Acc. (%)	Pass@5 (%)	Acc. (%)	Pass@5 (%)	Acc. (%)	Pass@5 (%)
HyperThink w/o VQ	67.44	86.83	59.74	80.14	43.32	66.90
HyperThink	67.53	86.67	61.06	82.11	48.76	73.40

Appendix

Table 8: Train–test diagnostic for the VQ bottleneck using Qwen3-0.6B trained on GSM8K. Both variants are evaluated using five sampled responses per query.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 356 papers

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs
Feb 28, 2026Jie Cao, Tianwei Lin, Juekai Lin +7Efficient Language Model Reasoning CoT Reasoning
Zhejiang University · Tencent
HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps
Jun 2, 2026Xin Liu, Runsong Zhao, Xinyu Liu +8Memory-Augmented Language Models Efficient Language Model Reasoning
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · China Unicom Cloud-Link, China · NiuTrans Research, Shenyang, China
Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate
Jun 15, 2026Hanyu Lin, Min Cai, Jiawei Wen +1Latent Reasoning LLM Reasoning
1Shenzhen University · University of Alberta

Feb 28, 2026J/K move · Enter open · S save

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

Jie Cao, Tianwei Lin, Juekai Lin, Zhenxuan Fan+6

Zhejiang University · Tencent

Efficient Language Model Reasoning CoT Reasoning Large Reasoning Models Overthinking in Language Models

Open paper AskPDF arXiv

Long chain-of-thought~(CoT) reasoning improves the problem-solving capability of large reasoning models~(LRMs), but often incurs substantial inference cost and overthinking. Existing efficiency methods primarily rely on prompting, post-hoc compression, or length-regularized optimization; these approaches can be sensitive to model scale, depend on external estimators, or under-allocate reasoning to difficult instances. We propose Draft-Thinking, a lightweight adaptation framework that learns a model-adapted draft CoT: a compact, self-contained working derivation that directly supports the final answer. We identify a Cross-Scale Draft Mismatch: concise traces generated by a strong teacher can be too sparse for a smaller target model. To address this mismatch, Progressive Draft Adaptation first transfers a concise reasoning-form prior through supervised fine-tuning (SFT) and then applies two-stage reinforcement learning (RL) with progressively harder data and relaxed response budgets to calibrate the reasoning granularity to the target model. The method requires neither post-hoc trajectory compression nor an explicit length-penalty reward. At inference time, draft, adaptive, and step-by-step prompts provide efficiency-oriented, balanced, and detailed reasoning modes, respectively. The step-by-step mode largely preserves the original model's accuracy while using fewer tokens. With only 847 training problems and 120 RL optimization steps, Draft-Thinking substantially reduces reasoning cost across six mathematical and scientific reasoning benchmarks. On MATH500, its draft mode reduces the average response length from 5,668 to 986 tokens, an 82.6% reduction, while achieving 90.6% accuracy compared with 93.0% for the original long CoT model.

Abstract

Efficient Language Model Reasoning

Explore similar work

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate