Extending CoT through RL has been widely used to enhance the reasoning capabilities of LLMs. However, due to the sparsity of reward signals, it can also induce undesirable thinking patterns such as overthinking, i.e., generating redundant intermediate reasoning content. In this work, we argue that a major source of such redundancy is inefficient reflection, which often manifests in two problematic patterns: Indiscriminate Reflection, where the model performs broad, low-impact checks throughout reasoning, and Repetitive Reflection, where it repeatedly re-verifies an already established conclusion. To address this, we introduce a graph-based CoT optimization framework. Specifically, we convert each linear CoT into a directed acyclic graph (DAG) with explicit dependency edges, and design a dual pruning strategy: branch-level pruning removes weakly contributing reflection branches, while depth-level pruning eliminates late-stage re-verification. We distill this behavior via a three-stage pipeline: (1) SFT to initialize the policy on pruned concise traces, (2) DPO to prefer correct but less redundant trajectories, and (3) GRPO with length penalty to jointly optimize answer correctness and efficiency. Experiments show that our approach reduces the average reasoning tokens by 42% while maintaining or improving accuracy.
Figures & tables
Figure 1: Relationship between average reasoning tokens and accuracy averaged across all benchmarks for different methods.
Figure 2: Training pipeline of our graph-based redundancy-aware post-training framework. We first perform cold-start SFT using compressed chain-of-thoughts obtained via iterative graph construction and redundancy pruning, which instills an inductive bias toward concise reasoning. On top of the SFT model, we conduct DPO by ranking correct trajectories according to redundancy, preferring low-redundancy outputs over highly redundant ones. Finally, we apply GRPO with length penalty, jointly optimizing answer correctness and reasoning efficiency to obtain the final model.
Method
AIME24
AIME25
AMC23
OlympiadBench
MATH500
Average
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Open-Source R1-Style Models
Skywork-OR1-7B He et al. (2025)
37.08
11890
24.67
12831
73.50
5241
51.41
6075
85.96
3753
54.52
7958
OREAL-7B Lyu et al. (2025)
38.75
10907
28.75
11309
81.25
4510
56.44
5164
88.83
3023
58.80
6983
AReaL-boba-RL-7B Mei et al. (2025)
36.67
11407
25.42
12365
71.56
5337
52.91
6037
87.12
3904
54.74
7810
Light-R1-DS-7B Wen et al. (2025a)
39.67
12036
31.00
12498
78.25
6073
54.74
5306
87.58
3020
58.25
7787
Table 1: Performance comparison across multiple math reasoning benchmarks (Accuracy ↑ and Average Length ↓ ). ∗ We adapt the original implementation of O1-Pruner and TokenSkip to the DeepSeek-R1-Distill-Qwen models.
Figure 3: Stage-wise ablation across five benchmarks. (a) Accuracy (%). (b) Relative reasoning length measured by the average number of generated tokens, normalized to the Base setting (token ratio; Base =1.0, lower is better). The compared configurations follow a cumulative training recipe: starting from Base , we sequentially add SFT , then DPO , and finally GRPO .
Statistic
Full CoT
Pruned CoT
Total Samples
3335
3335
Avg. Nodes
27.8
15.6
Avg. Review Nodes
16.8
4.5
Avg. Tokens
6468
4439
Total Cost
–
$20
Table 2: Dataset and graph statistics before and after pruning.
Figure 4: Changes in reasoning length and reasoning token usage before and after training on AIME24.
CoT Variant
Accuracy ( ↑ )
Consistency ( ↑ )
Full-CoT
98.95
99.60
Graph-Pruned
93.70
90.69
Len-Trunc
73.60
69.10
Table 3: Comparison of accuracy and consistency across different Chain-of-Thought (CoT) pruning methods.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Node Type
Precision
Recall
F1
Review
0.9048
0.9661
0.9344
Progress
0.8947
0.8500
0.8718
Atomicity Valid (%)
85.29
Appendix
Table 4: Type classification performance and step atomicity validity of constructed graph nodes.
Name
Value
Epochs
8
Global batch size
64
Max sequence length
8192
Optimizer
AdamW
Learning rate
1×10−5
LoRA rank
32
Appendix
Table 5: Hyper-parameters for SFT training.
Name
Value
Epochs
5
Global batch size
64
Max sequence length
8192
Optimizer
AdamW
Learning rate
1×10−7
Appendix
Table 6: Hyper-parameters for DPO training.
Name
Value
KL coefficient β
1×10−3
Rollouts per input
8
Sampling temperature
1.0
Max response length
12000
Global batch size
64
Optimizer
AdamW
Appendix
Table 7: Hyper-parameters for GRPO training.
Figure 5: RL training dynamics for different model scales. (a) 7B; (b) 1.5B. Left: mean reward. Right: response length (tokens).
Figure 6: Instruction for updating a graph-structured chain-of-thought
Figure 7: Qualitative comparison of reflection behavior between the base model and our trained model. The left column shows the original CoT, and the right column shows its graph-structured representation. Red-highlighted text indicate reflection-related content. Compared to the base model, our trained model exhibits reduced and more focused reflection.
Large Language Models (LLMs) have achieved remarkable success in complex reasoning tasks through Chain-of-Thought (CoT) prompting. However, these models often exhibit "computational overthinking," generating redundant reasoning steps that increase latency and cost without improving accuracy. Recent studies suggest that CoT trajectories can be significantly pruned, yet existing methods often rely on forcing a static thinking budget, heuristic filtering, sub-optimal early exit via classification, or expensive re-training. In this paper, we introduce OS-Pruner, a lightweight plug-in framework that formulates chain-of-thought pruning as an optimal stopping problem. Given a reasoning prefix, OS-Pruner learns whether further reasoning is worth its token cost by optimizing an explicit utility that trades off final-answer accuracy against generated length. Our novel formulation enables the model to dynamically assess the sufficient point of termination for a reasoning chain. OS-Pruner is designed to be lightweight during both training and inference, and to provide users with fine-grained control over the reasoning-effort vs. accuracy trade-off. On diverse reasoning benchmarks and base models, OS-Pruner achieves 20-60% reduction in generation length with minimal accuracy sacrifice.
Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias +2
Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).
Hui Xie, Jie Liu, Ziyue Qiao +1
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Long chain-of-thought~(CoT) reasoning improves the problem-solving capability of large reasoning models~(LRMs), but often incurs substantial inference cost and overthinking. Existing efficiency methods primarily rely on prompting, post-hoc compression, or length-regularized optimization; these approaches can be sensitive to model scale, depend on external estimators, or under-allocate reasoning to difficult instances. We propose Draft-Thinking, a lightweight adaptation framework that learns a model-adapted draft CoT: a compact, self-contained working derivation that directly supports the final answer. We identify a Cross-Scale Draft Mismatch: concise traces generated by a strong teacher can be too sparse for a smaller target model. To address this mismatch, Progressive Draft Adaptation first transfers a concise reasoning-form prior through supervised fine-tuning (SFT) and then applies two-stage reinforcement learning (RL) with progressively harder data and relaxed response budgets to calibrate the reasoning granularity to the target model. The method requires neither post-hoc trajectory compression nor an explicit length-penalty reward. At inference time, draft, adaptive, and step-by-step prompts provide efficiency-oriented, balanced, and detailed reasoning modes, respectively. The step-by-step mode largely preserves the original model's accuracy while using fewer tokens. With only 847 training problems and 120 RL optimization steps, Draft-Thinking substantially reduces reasoning cost across six mathematical and scientific reasoning benchmarks. On MATH500, its draft mode reduces the average response length from 5,668 to 986 tokens, an 82.6% reduction, while achieving 90.6% accuracy compared with 93.0% for the original long CoT model.