cs.LGMar 13, 2026
SaveTERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning
Organizations: UT Austin · EPFL · ENS Paris-Saclay · Télécom Paris (IP Paris)
Abstract
Large Reasoning Models (LRMs) achieve impressive performance on complex reasoning tasks via Chain-of-Thought (CoT) reasoning, which enables them to generate intermediate thinking tokens before arriving at the final answer. However, LRMs often suffer from significant overthinking, spending excessive compute time even after the answer is generated early on. Prior work has identified the existence of an optimal reasoning length such that truncating reasoning at this point significantly shortens CoT outputs with virtually no change in performance. However, determining optimal CoT lengths for practical datasets is highly non-trivial as they are fully task and model-dependent. In this paper, we precisely address this and design Terminator, an early-exit strategy for LRMs at inference to mitigate overthinking. The central idea underpinning Terminator is that the first arrival of an LRM's final answer is often predictable, and we leverage these first answer positions to create a novel dataset of optimal reasoning lengths to train Terminator. Powered by this approach, Terminator achieves significant reductions in CoT lengths of 14%-55% on average across four challenging practical datasets: MATH-500, AIME 2025, HumanEval, and GPQA, while outperforming current state-of-the-art methods and reducing inference latency by more than 2x compared to the original LRM.
Figures & tables
Figure 1: Pareto Frontier. Terminator defines the Pareto frontier on 14 of the 16 (LRM, benchmark) pairings, outperforming prior work. Each point represents a method’s accuracy and compression rate, with lower compression rates indicating greater token savings and hence compute. The dashed line traces the Pareto frontier connecting non-dominated solutions. We refer to Sec. 5.2 and Table 7 in App. C for more details.
Figure 2: Event-Locked Averaging of Token-Confidence. Event-locked averaging shows a consistent agreement on spiking behavior at the answer position in each CoT, but disagrees elsewhere. On the other hand, this phenomenon is not readily observable in the single-sample case. Figures on the left show the Token-Confidence [ 8 ] and log-probability trajectories throughout reasoning for a single, randomly selected sample; figures on the right show the effect of event-locked averaging on the position of the first arrival of the final answer across all CoTs. The 3200 CoTs used are a random subset of our training set, which combines AIME (1983–2024), MATH, OpenCoder-SFT, and OpenScience. Figs. 15 , 16 , 17 and 18 in App. C show similar trends for each dataset separately. The standard error (SE) is shown as a shaded region and becomes clearer when zoomed in.
Figure 3: Token Usage Frequency Shift. “Thinking token” usage changes depending on whether the final answer has been generated in the CoT. Rates are computed by counting the raw number of occurrences of the token before and after the answer, and then normalizing each count by the respective number of tokens in the before and after bins. The arrival of the final answer is hinted at by changes in the rates for these tokens. The relative length of a CoT is captured by its dot size, where a longer CoT has a larger dot. App. C demonstrates similar results for other “thinking tokens” in Fig. 19 and for each data source in Figs. 20 , 21 , 22 and 23 .
Figure 4: Early stopping via Terminator . Terminator is a binary probe classifier that predicts whether to exit or not at every CoT token. Once the majority of prediction bits within a window ( here) are , </think> is injected into the LRM’s token stream to stop thinking.
Figure 5: Training-Dataset Curation Process. We use an LRM to (1) extract final answer from final solution , (2) identify the earliest position of in the CoT , and (3) verify that the position was correct. If it was, then we can extract the exact position of from the CoT at the final token-index extraction step; otherwise, we retry the identification step with feedback.
| Method | Latency (s) | Throughput |
|---|---|---|
| (tok/s) | ||
| Qwen3-8B | ||
| Vanilla | ||
| Terminator | ||
| Qwen3-14B | ||
| Vanilla | ||
Table 1: Latency Analysis. Latency and throughput benchmarks on MATH-500 problems for Qwen3-based vanilla and Terminator models. Values are mean 95% CI.
Figure 6: Effects of Early CoT Termination. Test set CoTs are evaluated after truncating them at various points via </think> and asking the LRM for a final solution and answer. Diamond-shaped markers show the hindsight-optimal answer position (not achievable). Terminator is close to optimality on all datasets.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Training Convergence. During training, our proposed Terminator architecture achieves the best balance between the highest Macro-F1 validation scores and the lowest compute overhead.
Figure 8: Effect of Varying Window Size. We fix the threshold to 0.7 and vary the size of the window used by Terminator at inference-time. Terminator is robust to changes in the window size; Figures 17-21 in our manuscript show that the predicted probability of Terminator during CoT generation tends to only increase. This means that once the predicted probability crosses the threshold, it tends not to fall back below it, and the choice of window size has little or no effect.
Figure 9: Effect of Varying Predictive Threshold. We fix the window size to and vary the threshold used by Terminator to predict whether to exit early. Terminator is robust to changes in the predictive threshold: lowering makes the method more aggressive, leading to substantially higher compression rates, while accuracy changes only modestly across datasets. In particular, datasets such as MATH-500 and HumanEval exhibit little performance degradation across thresholds from to , suggesting that Terminator learns a stable confidence signal for identifying viable early-exit points.
| Supervised | Consistent | |||||
|---|---|---|---|---|---|---|
| Model | tolerance | threshold | window size | tolerance | threshold | window size |
| Qwen3-8B | 0.25 | 0.6526 | 10 | 0.25 | 0.8790 | 10 |
| Qwen3-14B | 0.25 | 0.6377 | 10 | 0.25 | 0.8679 | 10 |
| Ministral-3-8B | 0.10 | 0.8173 | 10 | 0.025 | 0.9973 | 10 |
| Ministral-3-14B | 0.10 | 0.8306 | 10 | 0.025 | 0.9973 | 10 |
Table 2: Hyperparameters used for Supervised and Consistent linear probes for Thought Calibration. Ministral models’ full names are Ministral-3-xxB-Reasoning-2512. Tolerance and threshold refer to and , respectively.
| MATH-500 | AIME 2025 | GPQA | HumanEval | |||||
| Model | Acc ( ) | CR ( ) | Acc ( ) | CR ( ) | Acc ( ) | CR ( ) | Acc ( ) | CR ( ) |
| Supervised | ||||||||
| Qwen3-8B | 90.1% | 93.89% | 65.8% | 81.54% | 52.6% | 78.87% | 71.8% | 92.92% |
| Qwen3-14B | 89.8% | 91.92% | 63.3% | 71.29% | 54.6% | 81.90% | 74.8% | 87.12% |
| Ministral-3-8B | 87.7% | 87.80% | 83.7% | 91.16% | 47.3% | 71.78% | 47.2% | 87.16% |
| Ministral-3-14B | 87.3% | 95.86% | 59.4% | 79.77% | 49.5% | 73.81% | 27.6% | 96.25% |
Table 3: Performance comparison of Supervised and Consistent probes for Thought Calibration across models and tasks. Ministral models’ full names are Ministral-3-xxB-Reasoning-2512.
Figure 10: Terminator Recovers Event-Locked Average Spiking. The exit positions predicted by Terminator (center) recovers similar spiking behavior in the event-locked averaged Token-Confidence as the ground-truth answer positions (left) . The histogram of differences between the exit positions (right) shows that Terminator ’s predicted exit positions are close to the ground-truth. Note that the y-axis on the histogram is log-scaled.
Figure 11: Terminator Token Usage Biases. The exit positions predicted by Terminator recover the same biases in the “thinking token” occurrence rates as the ground-truth answer positions. The inset axes on each panel show the percentage of dots that lie above the diagonal when the ground-truth and Terminator answer positions are used.
| Math | Coding | Science | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MATH-500 | AIME25 | HumanEval | GPQA | Overall | |||||
| Acc | CR | Acc | CR | Acc | CR | Acc | CR | Acc | CR | |
| Single-domain training ( Fig. 13 ) | 91% | 47% | 66% | 70% | 96% | 67% | 55% | 82% | 77% | 66.5% |
| Cross-domain training ( Table 7 ) | 91% | 45% | 69% | 71% | 96% | 70% | 56% | 86% | 78% | 68% |
Table 4: Cross-Domain vs. Single-Domain Training. Comparison of accuracy and compression ratio when Terminator is trained on a single-domain (based on Fig. 13 ) and all domains (based on Table 7 ). Cross-domain training (the proposed, unified model) and single-domain training are both valid approaches; cross-domain achieves a 1% increase in accuracy over single-domain, but at a 1.5% worse compression rate.
Figure 12: Adaptive Thresholding for Terminator . We plot Terminator at various thresholds (blue stars, fixed for all domains) and compare with a domain-adaptive threshold selection (red stars). The tuples correspond to the thresholds chosen for (MATH-500, AIME 25, HumanEval, GPQA) accordingly. For example, “(0.9, 0.3, 0.1, 0.1)” corresponds to choosing thresholds 0.9, 0.3, 0.1, 0.1 for MATH-500, AIME 25, HumanEval, and GPQA, respectively. We enumerate all possible threshold combinations and retain those that define the Pareto frontier. Domain adaptation provides incremental gains but requires domain knowledge at inference time.
Figure 13: OOD Performance of Terminator . The best trade-off between accuracy and compression rate is achieved when the evaluation set is in-distribution with the training dataset. Here the out-of-distribution performance of Terminator with respect to the compression rate (left) and the accuracy (right) for Qwen3-8B is shown. Training datasets are listed along the row axis, and the evaluation sets are listed across the column axis. For example, training Terminator on MATH and evaluating on HumanEval yields a compression rate of and an accuracy of . Every training dataset has an in-domain evaluation dataset, i.e. MATH MATH-500, AIME 1983–2024 AIME25, OpenCoder-SFT HumanEval, and OpenScience GPQA.
Figure 14: Pipeline Cost-Benefit Analysis. We report an estimate on the number of prompts to Terminator needed to break even with the compute cost associated with running our data curation pipeline (left) and on the number of GPU-seconds saved per prompt (right) . We use the Qwen3-8B throughput data from Table 1 and calculate the compute cost across fixed token savings (i.e. a reduction in the number of tokens by and ) and fixed prompt lengths of and tokens. At the time of writing, the number of monthly Qwen3-8B downloads from Hugging Face is million [ 14 ] . If just a single prompt were run per download on Qwen3-8B with Terminator , the compute offset would be compensated for many times over.
| Token-level prediction | Chunk-level prediction | |
|---|---|---|
| Hindsight-optimal labels | Terminator (proposed) | Underperforms Terminator |
| Consistency-based labels | Computationally infeasible | Prior work (underperforms) |
Table 5: Combinations of exit supervision and prediction granularity. Terminator combines hindsight-optimal labels with token-level prediction. The remaining combinations either correspond to prior work, underperform our method, or are computationally infeasible.
| Math | Coding | Science | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | MATH-500 | AIME25 | HumanEval | GPQA | Overall | |||||
| Acc | CR | Acc | CR | Acc | CR | Acc | CR | Acc | CR | |
| Terminator (Proposed) | 90.7% | 45.1% | 69.4% | 70.7% | 95.7% | 69.9% | 55.7% | 85.7% | 77.9% | 67.8% |
| Chunk Variant | 80.8% | 18.1% | 34.9% | 43.0% | 86.6% | 18.4% | 45.4% | 16.9% | 61.9% | 24.1% |
Table 6: Effect of chunk-level exit prediction. We train a chunk-level Terminator variant using the \n\n delimiter so that, for each CoT, the positive exit label is assigned to the end of the chunk containing the hindsight-optimal exit position. At inference time, exit predictions are made only at chunk boundaries. All other hyperparameters, including the exit threshold, are kept the same as in Terminator .
| Math | Coding | Science | ||||||||||||
| Method | MATH-500 | AIME25 | HumanEval | GPQA | Overall | |||||||||
| Acc | Tok | CR | Acc | Tok | CR | Acc | Tok | CR | Acc | Tok | CR | Acc | CR | |
| Qwen3-8B | ||||||||||||||
| Vanilla | 91.1% | 100% | 74.4% | 100% | 94.9% | 100% | 58.0% | 100% | 79.6% | 100% | ||||
| NoThinking | 80.7% | 16.1% | 22.0% | 18.6% | 84.6% | 11.8% | 46.0% | 15.8% | 58.3% | 15.6% | ||||
| DEER | 79.9% | 52.0% | 21.4% | 67.8% | 93.7% | 83.6% | 50.3% | 99.6% | 61.3% | 75.8% | ||||
Table 7: Performance of Terminator and Baselines. Indicates that higher values are better, while indicates that lower values are better. CR is the compression rate, reported here as the mean per-sample compression rate. Tok is the mean number of tokens per sample. Bold and Underlined values highlight the best and second-best performing early exit methods, respectively. Terminator demonstrates superior accuracy-efficiency trade-offs (best or second-best performance across 28 out of 32 metrics). Fig. 1 in Sec. 5 shows the results of this table on the Pareto frontier.
Figure 15: Event-Locked Averaging of Token-Confidence for AIME (1983–2024). A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected AIME (1983–2024) problems.
Figure 16: Event-Locked Averaging of Token-Confidence for MATH. A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected MATH problems.
Figure 17: Event-Locked Averaging of Token-Confidence for OpenCoder-SFT. A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected OpenCoder-SFT problems.
Figure 18: Event-Locked Averaging of Token-Confidence for OpenScience. A reproduction of Fig. 2 , but only using CoTs from 800 randomly selected OpenScience problems.
Figure 19: Token Usage Frequency Shift. An extension of the results shown in Fig. 3 , highlighting additional “thinking tokens.” While most “thinking tokens” shown here have some bias, often occurring before and after the first occurrence of the final answer as indicated by the Above Diagonal statistic, some tokens, like “ so ,” are close to an equal rate on average.
Figure 20: Token Usage Frequency Shift for AIME (1983–2024. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected AIME (1983–2024) problems.
Figure 21: Token Usage Frequency Shift for MATH. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected MATH problems.
Figure 22: Token Usage Frequency Shift for OpenCoder-SFT. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected OpenCoder-SFT problems.
Figure 23: Token Usage Frequency Shift for OpenScience. A reproduction of Fig. 19 , but only using CoTs from 800 randomly selected OpenScience problems.
Figure 24: Token Occurrence Rate vs CoT Length. For some tokens, such as these three “thinking tokens,” occurrence rates decrease rapidly as CoT length increases. Interestingly, the side of the answer (either “before” or “after”) with the highest rate is the one that decays the most (the “before” rate for hmm and okay and the “after” rate for another ), while the side with the lower rate sees only a slight decrease or increase. For each plot, the lengths are placed into ten bins with percentile-based bin edges. In other words, each bin contains approximately 10% of the samples. Shaded regions indicate the 95% confidence interval.
Figure 25: Predicted Probabilities Event-Locked Averaging. The dashed vertical line shows where Terminator terminates the CoT with a sliding window of 10 and an exit threshold of 0.7, as indicated by the horizontal dotted line. We show the average predicted probability stream across all test problems from MATH-500, AIME25, HumanEval, and GPQA. Figs. 26 , 27 , 28 and 29 show predictions streams from individual, randomly drawn samples from each data source.
Figure 26: Predicted Probabilities for AIME25. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from AIME25.
Figure 27: Predicted Probabilities for MATH-500. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from MATH-500.
Figure 28: Predicted Probabilities for HumanEval. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from HumanEval.
Figure 29: Predicted Probabilities for GPQA. Terminator ’s predicted probability stream for early-exiting on four randomly chosen samples from GPQA.
Figure 30: First Answer Occurrence Histogram. A histogram of the first occurrence of the final answer for each data source used in our training dataset is shown.
Figure 31: Compression Ratio Histograms for Terminator . Each histogram shows the frequency of an achieved compression rate when early-exiting with Terminator . indicates that a lower compression rate is better, as it results in more tokens saved with our method. Three CoTs are sampled per data source.
Figure 32: Predicted Probabilities for MATH-500. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from MATH-500. The beginning and the end are truncated for better visibility.
Figure 33: Predicted Probabilities for MATH-500. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from MATH-500. The beginning and the end are truncated for better visibility.
Figure 34: Predicted Probabilities for AIME25. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from AIME-2025. The beginning and the end are truncated for better visibility.
Figure 35: Predicted Probabilities for AIME25. Terminator ’s predicted probabilities for early-exit on a randomly chosen sample from AIME-2025. The beginning and the end are truncated for better visibility.