Large Reasoning Models (LRMs) achieve impressive performance on complex reasoning tasks via Chain-of-Thought (CoT) reasoning, which enables them to generate intermediate thinking tokens before arriving at the final answer. However, LRMs often suffer from significant overthinking, spending excessive compute time even after the answer is generated early on. Prior work has identified the existence of an optimal reasoning length such that truncating reasoning at this point significantly shortens CoT outputs with virtually no change in performance. However, determining optimal CoT lengths for practical datasets is highly non-trivial as they are fully task and model-dependent. In this paper, we precisely address this and design Terminator, an early-exit strategy for LRMs at inference to mitigate overthinking. The central idea underpinning Terminator is that the first arrival of an LRM's final answer is often predictable, and we leverage these first answer positions to create a novel dataset of optimal reasoning lengths to train Terminator. Powered by this approach, Terminator achieves significant reductions in CoT lengths of 14%-55% on average across four challenging practical datasets: MATH-500, AIME 2025, HumanEval, and GPQA, while outperforming current state-of-the-art methods and reducing inference latency by more than 2x compared to the original LRM.
Figures & tables
Figure 1: Pareto Frontier. Terminator defines the Pareto frontier on 14 of the 16 (LRM, benchmark) pairings, outperforming prior work. Each point represents a method’s accuracy and compression rate, with lower compression rates indicating greater token savings and hence compute. The dashed line traces the Pareto frontier connecting non-dominated solutions. We refer to Sec. 5.2 and Table 7 in App. C for more details.
Figure 2: Event-Locked Averaging of Token-Confidence. Event-locked averaging shows a consistent agreement on spiking behavior at the answer position in each CoT, but disagrees elsewhere. On the other hand, this phenomenon is not readily observable in the single-sample case. Figures on the left show the Token-Confidence [ 8 ] and log-probability trajectories throughout reasoning for a single, randomly selected sample; figures on the right show the effect of event-locked averaging on the position of the first arrival of the final answer across all CoTs. The 3200 CoTs used are a random subset of our training set, which combines AIME (1983–2024), MATH, OpenCoder-SFT, and OpenScience. Figs. 15 , 16 , 17 and 18 in App. C show similar trends for each dataset separately. The standard error (SE) is shown as a shaded region and becomes clearer when zoomed in.
Figure 3: Token Usage Frequency Shift. “Thinking token” usage changes depending on whether the final answer has been generated in the CoT. Rates are computed by counting the raw number of occurrences of the token before and after the answer, and then normalizing each count by the respective number of tokens in the before and after bins. The arrival of the final answer is hinted at by changes in the rates for these tokens. The relative length of a CoT is captured by its dot size, where a longer CoT has a larger dot. App. C demonstrates similar results for other “thinking tokens” in Fig. 19 and for each data source in Figs. 20 , 21 , 22 and 23 .
Figure 4: Early stopping via Terminator . Terminator is a binary probe classifier that predicts whether to exit or not at every CoT token. Once the majority of prediction bits within a window ( 10 here) are 1 , </think> is injected into the LRM’s token stream to stop thinking.
Figure 5: Training-Dataset Curation Process. We use an LRM to (1) extract final answer a^ from final solution s , (2) identify the earliest position of a^ in the CoT r , and (3) verify that the position was correct. If it was, then we can extract the exact position of a^ from the CoT at the final token-index extraction step; otherwise, we retry the identification step with feedback.
Method
Latency (s)
Throughput
(tok/s)
Qwen3-8B
Vanilla
32.68±9.59
151.5±4.4
Terminator
14.10±6.27
135.2±2.0
Qwen3-14B
Vanilla
43.38±13.98
98.0±2.0
Table 1: Latency Analysis. Latency and throughput benchmarks on MATH-500 problems for Qwen3-based vanilla and Terminator models. Values are mean ± 95% CI.
Figure 6: Effects of Early CoT Termination. Test set CoTs are evaluated after truncating them at various points via </think> and asking the LRM for a final solution and answer. Diamond-shaped markers show the hindsight-optimal answer position (not achievable). Terminator is close to optimality on all datasets.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Training Convergence. During training, our proposed Terminator architecture achieves the best balance between the highest Macro-F1 validation scores and the lowest compute overhead.
Figure 8: Effect of Varying Window Size. We fix the threshold to 0.7 and vary the size of the window used by Terminator at inference-time. Terminator is robust to changes in the window size; Figures 17-21 in our manuscript show that the predicted probability of Terminator during CoT generation tends to only increase. This means that once the predicted probability crosses the threshold, it tends not to fall back below it, and the choice of window size has little or no effect.
Figure 9: Effect of Varying Predictive Threshold. We fix the window size to 10 and vary the threshold τ used by Terminator to predict whether to exit early. Terminator is robust to changes in the predictive threshold: lowering τ makes the method more aggressive, leading to substantially higher compression rates, while accuracy changes only modestly across datasets. In particular, datasets such as MATH-500 and HumanEval exhibit little performance degradation across thresholds from τ=0.1 to τ=0.9 , suggesting that Terminator learns a stable confidence signal for identifying viable early-exit points.
Supervised
Consistent
Model
tolerance
threshold
window size
tolerance
threshold
window size
Qwen3-8B
0.25
0.6526
10
0.25
0.8790
10
Qwen3-14B
0.25
0.6377
10
0.25
0.8679
10
Ministral-3-8B
0.10
0.8173
10
0.025
0.9973
10
Ministral-3-14B
0.10
0.8306
10
0.025
0.9973
10
Appendix
Table 2: Hyperparameters used for Supervised and Consistent linear probes for Thought Calibration. Ministral models’ full names are Ministral-3-xxB-Reasoning-2512. Tolerance and threshold refer to δ and λ , respectively.
MATH-500
AIME 2025
GPQA
HumanEval
Model
Acc ( ↑ )
CR ( ↓ )
Acc ( ↑ )
CR ( ↓ )
Acc ( ↑ )
CR ( ↓ )
Acc ( ↑ )
CR ( ↓ )
Supervised
Qwen3-8B
90.1%
93.89%
65.8%
81.54%
52.6%
78.87%
71.8%
92.92%
Qwen3-14B
89.8%
91.92%
63.3%
71.29%
54.6%
81.90%
74.8%
87.12%
Ministral-3-8B
87.7%
87.80%
83.7%
91.16%
47.3%
71.78%
47.2%
87.16%
Ministral-3-14B
87.3%
95.86%
59.4%
79.77%
49.5%
73.81%
27.6%
96.25%
Appendix
Table 3: Performance comparison of Supervised and Consistent probes for Thought Calibration across models and tasks. Ministral models’ full names are Ministral-3-xxB-Reasoning-2512.
Figure 10: Terminator Recovers Event-Locked Average Spiking. The exit positions predicted by Terminator (center) recovers similar spiking behavior in the event-locked averaged Token-Confidence as the ground-truth answer positions (left) . The histogram of differences between the exit positions (right) shows that Terminator ’s predicted exit positions are close to the ground-truth. Note that the y-axis on the histogram is log-scaled.
Figure 11: Terminator Token Usage Biases. The exit positions predicted by Terminator recover the same biases in the “thinking token” occurrence rates as the ground-truth answer positions. The inset axes on each panel show the percentage of dots that lie above the diagonal when the ground-truth and Terminator answer positions are used.
Math
Coding
Science
Method
MATH-500
AIME25
HumanEval
GPQA
Overall
Acc ↑
CR ↓
Acc ↑
CR ↓
Acc ↑
CR ↓
Acc ↑
CR ↓
Acc ↑
CR ↓
Single-domain training ( Fig. 13 )
91%
47%
66%
70%
96%
67%
55%
82%
77%
66.5%
Cross-domain training ( Table 7 )
91%
45%
69%
71%
96%
70%
56%
86%
78%
68%
Appendix
Table 4: Cross-Domain vs. Single-Domain Training. Comparison of accuracy and compression ratio when Terminator is trained on a single-domain (based on Fig. 13 ) and all domains (based on Table 7 ). Cross-domain training (the proposed, unified model) and single-domain training are both valid approaches; cross-domain achieves a 1% increase in accuracy over single-domain, but at a 1.5% worse compression rate.
Figure 12: Adaptive Thresholding for Terminator . We plot Terminator at various thresholds (blue stars, fixed for all domains) and compare with a domain-adaptive threshold selection (red stars). The tuples correspond to the thresholds chosen for (MATH-500, AIME 25, HumanEval, GPQA) accordingly. For example, “(0.9, 0.3, 0.1, 0.1)” corresponds to choosing thresholds 0.9, 0.3, 0.1, 0.1 for MATH-500, AIME 25, HumanEval, and GPQA, respectively. We enumerate all possible threshold combinations and retain those that define the Pareto frontier. Domain adaptation provides incremental gains but requires domain knowledge at inference time.
Figure 13: OOD Performance of Terminator . The best trade-off between accuracy and compression rate is achieved when the evaluation set is in-distribution with the training dataset. Here the out-of-distribution performance of Terminator with respect to the compression rate (left) and the accuracy (right) for Qwen3-8B is shown. Training datasets are listed along the row axis, and the evaluation sets are listed across the column axis. For example, training Terminator on MATH and evaluating on HumanEval yields a compression rate of 67% and an accuracy of 95% . Every training dataset has an in-domain evaluation dataset, i.e. MATH → MATH-500, AIME 1983–2024 → AIME25, OpenCoder-SFT → HumanEval, and OpenScience → GPQA.
Figure 14: Pipeline Cost-Benefit Analysis. We report an estimate on the number of prompts to Terminator needed to break even with the compute cost associated with running our data curation pipeline (left) and on the number of GPU-seconds saved per prompt (right) . We use the Qwen3-8B throughput data from Table 1 and calculate the compute cost across fixed token savings (i.e. a reduction in the number of tokens by 30%,40%,50%, and 60% ) and fixed prompt lengths of 4,000,8,000 and 16,000 tokens. At the time of writing, the number of monthly Qwen3-8B downloads from Hugging Face is ∼9.9 million [ 14 ] . If just a single prompt were run per download on Qwen3-8B with Terminator , the compute offset would be compensated for many times over.
Token-level prediction
Chunk-level prediction
Hindsight-optimal labels
Terminator (proposed)
Underperforms Terminator
Consistency-based labels
Computationally infeasible
Prior work (underperforms)
Appendix
Table 5: Combinations of exit supervision and prediction granularity. Terminator combines hindsight-optimal labels with token-level prediction. The remaining combinations either correspond to prior work, underperform our method, or are computationally infeasible.
Math
Coding
Science
Method
MATH-500
AIME25
HumanEval
GPQA
Overall
Acc ↑
CR ↓
Acc ↑
CR ↓
Acc ↑
CR ↓
Acc ↑
CR ↓
Acc ↑
CR ↓
Terminator (Proposed)
90.7%
45.1%
69.4%
70.7%
95.7%
69.9%
55.7%
85.7%
77.9%
67.8%
Chunk Variant
80.8%
18.1%
34.9%
43.0%
86.6%
18.4%
45.4%
16.9%
61.9%
24.1%
Appendix
Table 6: Effect of chunk-level exit prediction. We train a chunk-level Terminator variant using the \n\n delimiter so that, for each CoT, the positive exit label is assigned to the end of the chunk containing the hindsight-optimal exit position. At inference time, exit predictions are made only at chunk boundaries. All other hyperparameters, including the exit threshold, are kept the same as in Terminator .
Math
Coding
Science
Method
MATH-500
AIME25
HumanEval
GPQA
Overall
Acc ↑
Tok ↓
CR ↓
Acc ↑
Tok ↓
CR ↓
Acc ↑
Tok ↓
CR ↓
Acc ↑
Tok ↓
CR ↓
Acc ↑
CR ↓
Qwen3-8B
Vanilla
91.1%
5,037
100%
74.4%
14,499
100%
94.9%
3,792
100%
58.0%
8,594
100%
79.6%
100%
NoThinking
80.7%
809
16.1%
22.0%
2,355
18.6%
84.6%
353
11.8%
46.0%
1,204
15.8%
58.3%
15.6%
DEER
79.9%
2,602
52.0%
21.4%
10,349
67.8%
93.7%
3,275
83.6%
50.3%
8,553
99.6%
61.3%
75.8%
Appendix
Table 7: Performance of Terminator and Baselines. ↑ Indicates that higher values are better, while ↓ indicates that lower values are better. CR is the compression rate, reported here as the mean per-sample compression rate. Tok is the mean number of tokens per sample. Bold and Underlined values highlight the best and second-best performing early exit methods, respectively. Terminator demonstrates superior accuracy-efficiency trade-offs (best or second-best performance across 28 out of 32 metrics). Fig. 1 in Sec. 5 shows the results of this table on the Pareto frontier.
Figure 15: Event-Locked Averaging of Token-Confidence for AIME (1983–2024). A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected AIME (1983–2024) problems.
Figure 16: Event-Locked Averaging of Token-Confidence for MATH. A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected MATH problems.
Figure 17: Event-Locked Averaging of Token-Confidence for OpenCoder-SFT. A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected OpenCoder-SFT problems.
Figure 18: Event-Locked Averaging of Token-Confidence for OpenScience. A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected OpenScience problems.
Figure 19: Token Usage Frequency Shift. An extension of the results shown in Fig. 3 , highlighting additional “thinking tokens.” While most “thinking tokens” shown here have some bias, often occurring before and after the first occurrence of the final answer as indicated by the Above Diagonal statistic, some tokens, like “ so ,” are close to an equal rate on average.
Figure 20: Token Usage Frequency Shift for AIME (1983–2024. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected AIME (1983–2024) problems.
Figure 21: Token Usage Frequency Shift for MATH. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected MATH problems.
Figure 22: Token Usage Frequency Shift for OpenCoder-SFT. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected OpenCoder-SFT problems.
Figure 23: Token Usage Frequency Shift for OpenScience. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected OpenScience problems.
Figure 24: Token Occurrence Rate vs CoT Length. For some tokens, such as these three “thinking tokens,” occurrence rates decrease rapidly as CoT length increases. Interestingly, the side of the answer (either “before” or “after”) with the highest rate is the one that decays the most (the “before” rate for hmm and okay and the “after” rate for another ), while the side with the lower rate sees only a slight decrease or increase. For each plot, the lengths are placed into ten bins with percentile-based bin edges. In other words, each bin contains approximately 10% of the samples. Shaded regions indicate the 95% confidence interval.
Figure 25: Predicted Probabilities Event-Locked Averaging. The dashed vertical line shows where Terminator terminates the CoT with a sliding window of 10 and an exit threshold of 0.7, as indicated by the horizontal dotted line. We show the average predicted probability stream across all test problems from MATH-500, AIME25, HumanEval, and GPQA. Figs. 26 , 27 , 28 and 29 show predictions streams from individual, randomly drawn samples from each data source.
Figure 26: Predicted Probabilities for AIME25. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from AIME25.
Figure 27: Predicted Probabilities for MATH-500. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from MATH-500.
Figure 28: Predicted Probabilities for HumanEval. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from HumanEval.
Figure 29: Predicted Probabilities for GPQA. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from GPQA.
Figure 30: First Answer Occurrence Histogram. A histogram of the first occurrence of the final answer for each data source used in our training dataset is shown.
Figure 31: Compression Ratio Histograms for Terminator . Each histogram shows the frequency of an achieved compression rate when early-exiting with Terminator . ↓ indicates that a lower compression rate is better, as it results in more tokens saved with our method. Three CoTs are sampled per data source.
Figure 32: Predicted Probabilities for MATH-500. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from MATH-500. The beginning and the end are truncated for better visibility.
Figure 33: Predicted Probabilities for MATH-500. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from MATH-500. The beginning and the end are truncated for better visibility.
Figure 34: Predicted Probabilities for AIME25. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from AIME-2025. The beginning and the end are truncated for better visibility.
Figure 35: Predicted Probabilities for AIME25. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from AIME-2025. The beginning and the end are truncated for better visibility.
Chain-of-thought (CoT) reasoning improves large reasoning models (LRMs) on complex tasks but often produces long, redundant traces. Recent training-free early-exit methods shorten these traces by choosing an intermediate point to stop reasoning. We study one such strategy that injects an end-of-think token (EoT, </think>) at this point to trigger the reasoning-to-answering transition, and find that the injected EoT does not always induce a clean answering phase. Answering-phase generation can continue before the model regenerates another EoT, with the span preceding this regenerated EoT scaling with the reasoning tokens saved by early exit and exhibiting continued reasoning behavior. We call this spurious CoT termination, where reasoning-like generation continues into the answering phase. We hypothesize that insufficient attention to the injected EoT contributes to spurious CoT termination and probe this hypothesis with Exit-token Attention Biasing (EAB). Across four LRMs, five benchmarks, and two early-exit methods, increasing attention to the injected EoT reduces spurious CoT termination and answering-phase length. These results reveal a limitation of controlling LRMs by externally matching their explicit think-block format. Inserting the EoT token conforms to this format but does not by itself guarantee the intended reasoning-to-answering transition. Our code is available at https://github.com/Seunghee-Koh/Spurious-CoT-Termination.
Seunghee Koh, Sungjae Choi, Minchan Kwon +2
Korea Advanced Institute of Science and Technology, South Korea
Large Language Models (LLMs) have achieved remarkable success in complex reasoning tasks through Chain-of-Thought (CoT) prompting. However, these models often exhibit "computational overthinking," generating redundant reasoning steps that increase latency and cost without improving accuracy. Recent studies suggest that CoT trajectories can be significantly pruned, yet existing methods often rely on forcing a static thinking budget, heuristic filtering, sub-optimal early exit via classification, or expensive re-training. In this paper, we introduce OS-Pruner, a lightweight plug-in framework that formulates chain-of-thought pruning as an optimal stopping problem. Given a reasoning prefix, OS-Pruner learns whether further reasoning is worth its token cost by optimizing an explicit utility that trades off final-answer accuracy against generated length. Our novel formulation enables the model to dynamically assess the sufficient point of termination for a reasoning chain. OS-Pruner is designed to be lightweight during both training and inference, and to provide users with fine-grained control over the reasoning-effort vs. accuracy trade-off. On diverse reasoning benchmarks and base models, OS-Pruner achieves 20-60% reduction in generation length with minimal accuracy sacrifice.
Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias +2
Large reasoning models (LRMs) like OpenAI o1 and DeepSeek-R1 achieve high accuracy on complex tasks by adopting long chain-of-thought (CoT) reasoning paths. However, the inherent verbosity of these processes frequently results in redundancy and overthinking. To address this issue, existing works leverage Group Relative Policy Optimization (GRPO) to reduce LRM output length, but their static length reward design cannot dynamically adapt according to the relative problem difficulty and response length distribution, causing over-compression and compromised accuracy. Therefore, we propose SmartThinker, a novel GRPO-based efficient reasoning method with progressive CoT length calibration. SmartThinker makes a two-fold contribution: First, it dynamically estimates the optimal length with peak accuracy during training and guides overlong responses toward it to reduce response length while sustaining accuracy. Second, it dynamically modulates the length reward coefficient to avoid the unwarranted penalization of correct reasoning paths. Extensive experiment results show that SmartThinker achieves up to 52.5% average length compression with improved accuracy, and achieves up to 16.6% accuracy improvement on challenging benchmarks like AIME25. The source code can be found at https://github.com/SJTU-RTEAS/SmartThinker.
Chenzhi Hu, Qinzhe Hu, Yuhang Xu +6
Shanghai Jiao Tong University · Beihang University