Training Large Reasoning Models Efficiently via Progressive Thought Encoding

Zeliang Zhang; Xiaodong Liu; Hao Cheng; Hao Sun; Chenliang Xu; Jianfeng Gao

cs.LGFeb 18, 2026

Training Large Reasoning Models Efficiently via Progressive Thought Encoding

Authors: Zeliang Zhang, Xiaodong Liu, Hao Cheng, Hao Sun, Chenliang Xu, Jianfeng Gao

Organizations: University of Rochester · Microsoft Research

Abstract

Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate reasoning into fixed-size vector representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, on six widely used challenging mathematical benchmarks show consistent gains: our method achieves +19.3% improvement over LoRA-based fine-tuning and +29.9% over LRMs without fine-tuning on average, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints.

Figures & tables

Figure 1: Overview of our method. During the rollout process, the model continuously learns the dropped tokens to achieve a balance between generation efficiency and long-term memory.

Figure 2: The computation of context state $S$ .

Studied Models	Methods	Maximum TFLOPs of Attention	Peak GPU Mem. (%)	Mean GPU Mem. (%)	Math500 pass@1	Olympiad pass@1	Minerva Math pass@1	AMC pass@1	AIME2024 pass@16	AIME2025 pass@16	Avg.
Qwen2.5-3B-Instruct	Baseline	–	–	–	50.8	27.2	16.1	34.3	20.0	13.3	26.9
	LoRA	4.2	82.8	63.5	53.2 +2.4	27.8 +0.6	15.9 -0.2	35.9 +1.6	20.0 0.0	16.7 +3.4	28.2 +1.3
	$\text{LoRA}_{c}$	2.6 -1.6	38.0 -44.8	31.0 -32.5	50.0 -0.8	27.7 +0.5	16.1 0.0	33.1 -1.2	16.7 -3.3	10.0 -3.3	25.6 -1.3
	Ours	2.7 -1.5	45.3 -37.5	32.6 -30.9	54.0 +3.2	29.0 +1.8	16.2 +0.1	45.0 +10.7	20.0 0.0	16.7 +3.4	30.1 +3.2
Qwen2.5-7B-Instruct	Baseline	–	–	–	56.8	34.7	18.5	48.4	23.3	16.6	33.1
	LoRA	5.7	85.8	59.3	59.4 +2.6	38.7 +4.0	23.4 +4.9	50.6 +2.2	30.0 +6.7	26.7 +10.1	38.1 +5.0

Table 1: Comparison of methods across different models on benchmark datasets. The best average performance per model is highlighted in bold. Note: Benchmark improvements are reported relative to Baseline, while FLOPs/Memory reductions are reported relative to LoRA.

Panel of Figure 3 — Figure 3: Evaluation of Qwen-7B-Instruct and DeepSeek-R1-Distill-Llama-8B models trained by different methods on four benchmarks. We set the same maximum number of tokens for generation as 3072, and vary the KV cache window length from 768 to 3072. Each value corresponds to the mean pass@1 score over five independent runs.

	Models	Qwen2.5-4B-Instruct				Qwen2.5-7B-Instruct				DeepSeek-R1- Distill-Llama-8B
Dataset	Method	768	1024	1536	Avg.	768	1024	1536	Avg.	768	1024	1536	Avg.
AIME2024	Baseline	10.0	16.6	20.0	15.53	23.3	13.3	23.3	19.97	3.3	3.3	20.0	8.87
	LoRA	10.0 0.0	13.3 -3.3	20.0 0.0	14.43 -1.1	10.0 -13.3	23.3 +10.0	30.0 +6.7	21.10 +1.1	3.3 0.0	3.3 0.0	23.3 +3.3	9.97 +1.1
	$\text{LoRA}_{c}$	13.3 +3.3	13.3 -3.3	16.7 -3.3	14.43 -1.1	16.6 -6.7	20.0 +6.7	20.0 -3.3	18.87 -1.1	6.7 +3.4	16.7 +13.4	36.7 +16.7	10.03 +1.2
	Ours	16.7 +6.7	20.0 +3.4	20.0 0.0	18.90 +3.4	26.6 +3.3	26.6 +13.3	30.0 +6.7	27.73 +7.8	26.7 +23.4	30.0 +26.7	56.7 +36.7	37.80 +28.9
AIME2025	Baseline	6.7	13.3	13.3	11.10	10.0	16.7	16.6	14.43	6.7	10.0	20.0	12.23

Table 2: AIME2024 and AIME2025 pass@16 results (%). Maximum generation length is 6,144 tokens. KV cache window sizes range from 768 to 1,536. Note: Improvements are reported relative to Baseline.

Figure 5: Performance on MATH-500 under a fixed 1K context window as the maximum generation length increases from 3K to 64K.

Figure 6: The statistics on the generation length.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Evaluation of Qwen2.5-4B-Instruct models.

Ratio	25%	20%	15%	10%	5%
Score	50.8	51.4	51.9	50.7	49.6

Appendix

Table A1 : Effect of eviction ratios on MATH-500.

Explore similar work

Efficient Reasoning via Constrained Optimization in Latent Space
Sep 28, 2026Zhinan Hou, Xingchen Li, Keyou YouOverthinking in Language Models Efficient Language Model Reasoning
Tsinghua University
Efficient Reasoning with Balanced Thinking
Mar 12, 2026Yulin Li, Tengyao Tu, Li Ding +5Efficient Language Model Reasoning Large Reasoning Models
Harbin Institute of Technology (Shenzhen) · Zhongguancun Academy · Huawei Noah’s Ark Lab +3
LightThinker++: From Reasoning Compression to Memory Management
Apr 4, 2026Yuqi Zhu, Jintian Zhang, Zhenjie Wan +7Efficient Language Model Reasoning CoT Compression
Zhejiang University · Ant Group

Sep 28, 2026J/K move · Enter open · S save

Efficient Reasoning via Constrained Optimization in Latent Space

Zhinan Hou, Xingchen Li, Keyou You

Tsinghua University

Overthinking in Language Models Efficient Language Model Reasoning Quadratic Programming Efficient Language Model Inference Latent Space Optimization Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) have shown remarkable reasoning capabilities, yet they still suffer from overthinking, generating redundant reasoning steps which incur substantial token consumption. Existing methods, such as suppressing reflective keywords or forcing shorter reasoning lengths, attempt to mitigate this issue but inevitably truncate necessary steps and induce underthinking, thereby compromising performance. To address this dilemma, we investigate the latent representations and observe that efficient reasoning steps naturally cluster into a concentrated region in latent space, while those deviating from this region tend to produce verbose sequences. To leverage this, we keep reasoning focused within this region via a quadratic program which projects deviating hidden states back into the region. Then we propose a novel training-free framework to achieve efficient reasoning that reduces token generation costs without sacrificing performance. Extensive experiments conducted on four models ranging from 1.5B to 14B, and across six benchmarks in math reasoning, coding, and scientific QA, validate the effectiveness of our method, up to a 12.1% improvement in accuracy while reducing generated tokens by 11.8% to 52.8%. Codes are available at https://github.com/hzn18/Opt4Reasoning.

Abstract

Explore similar work

Efficient Reasoning via Constrained Optimization in Latent Space

Efficient Reasoning with Balanced Thinking

LightThinker++: From Reasoning Compression to Memory Management