The Detection-Extraction Gap: Models Know the Answer Before They Can Say It

Hanyang Wang; Mingxuan Zhu

cs.CLApr 8, 2026

The Detection-Extraction Gap: Models Know the Answer Before They Can Say It

Authors: Hanyang Wang, Mingxuan Zhu

Organizations: The University of Chicago · Imperial College London

Abstract

Modern reasoning models continue generating long after the answer is already determined. Across five model configurations, two families, and three benchmarks, we find that 52--88% of chain-of-thought tokens are produced after the answer is recoverable from a partial prefix. This post-commitment generation reveals a structural phenomenon: the detection-extraction gap. Free continuations from early prefixes recover the correct answer even at 10% of the trace, while forced extraction fails on 42% of these cases. The answer is recoverable from the model state, yet prompt-conditioned decoding fails to extract it. We formalize this mismatch via a total-variation bound between free and forced continuation distributions, yielding quantitative estimates of suffix-induced shift. Exploiting this asymmetry, we propose Black-box Adaptive Early Exit (BAEE), which uses free continuations for both detection and extraction, truncating 70--78% of serial generation while improving accuracy by 1--5pp across all models. For thinking-mode models, early exit prevents post-commitment overwriting, yielding gains of up to 5.8pp; a cost-optimized variant achieves 68--73% reduction at a median of 9 API calls. Code is available at https://github.com/EdWangLoDaSc/know2say.

Figures & tables

Figure 1: Two core findings. (a) Commitment map for 32B-Think: 69% of CoT tokens follow the commitment boundary (75% under PSC threshold; Table 2 ). (b) The detection–extraction gap: psc detects recoverability from 10% prefix, yet efa fails to extract on 42% of these problems. baee exploits this structure for 70–78% serial reduction with 1–5 pp accuracy gains (§ 4.5 ).

Figure 2: Method pipeline. At prefix fraction $f{=}0.1$ , psc (free continuation) recovers the correct answer 82% of the time, while efa (forced extraction) succeeds on only 34%. baee uses free continuations for both detection and extraction, sidestepping the gap entirely.

Category	Details
Models	Qwen3-32B-(Think/NoThink), Qwen3-8B-(Think/NoThink) Yang et al. (2025a) , and GPT-OSS-120B (medium reasoning) Agarwal et al. (2025) .
Benchmarks	MATH-500 : 500 problems (primary); GPQA-Diamond : 198 problems (validation); HumanEval : 164 problems (code).
Prefix Grid	$\{0.10,0.20,\dots,0.90\}$ (9 checkpoints).
Sampling	Full rollouts: 4 (temp. 1.0); psc samples: 8; entropy top- $k$ : 20; efa max tokens: 64.
Grading	SymPy symbolic equivalence (MATH-500); Exact match (GPQA).
Re-grading	Fixed efa suffix-stripping bug (90 evaluations flipped). Verified against independent SymPy re-grade with zero discrepancies.

Table 1: Compact summary of the experimental setup.

Model	Acc.	Commitment profile			Latency reduction
Model	Acc.	Commit	95% CI	Post-commit	CoT len	Main-rollout red.	Avg calls
Qwen3-32B-Think	82%	25%	[23%, 26%]	75%	2878	72%	7.7
Qwen3-32B-NoThink	91%	39%	[35%, 42%]	61%	698	73%	8.0
Qwen3-8B-Think	76%	27%	[25%, 29%]	73%	2881	68%	7.3
Qwen3-8B-NoThink	90%	48%	[44%, 52%]	52%	696	73%	8.0
GPT-OSS-120B	96%	36%	[34%, 38%]	64%	823	85%	8.6

Table 2: Post-commitment generation metrics on MATH-500 (re-graded). “Acc.” = fraction of problems where the first of 4 sampled rollouts is correct (single-rollout accuracy, not best-of-4). Commitment fraction defined conditionally on solvable instances (

\geq

1/4 correct); CIs from 10K bootstrap resamples. Latency reduction shown for psc -8 all (accuracy-optimal; see Table 5 for all strategies).

Figure 3: Main results. (a) efa accuracy by prefix length. (b) psc agreement (

>

70% from the first checkpoint). (c) Commitment distributions (Think median

\sim

25%, NoThink

\sim

40%). (d) Commitment increases monotonically with problem difficulty.

Problem	GT	PSC@10%	EFA@10%	Failure mode
$1-2+3-4+\cdots+99-100$	$-50$	8/8	$50$	Sign dropped
$f(x)=2^{x}$ ; find $\sqrt{f(f(f(f(1))))}$	$256$	8/8	$4$	Intermediate value

Table 3: Detection–extraction gap: two case studies (Qwen3-32B-Think, 10% prefix, psc 8/8). efa (shaded) extracts the wrong answer despite full recoverability via free continuation.

Figure 4: Commitment maps. (a) 32B-Think: 69% post-commitment. (b) 8B-NoThink: 35% post-commitment, later and more variable.

Model	Benchmark	Acc.	Commit	Post-commit	CoT len	PSC@10%	EFA@10%	Main-rollout red.
8B-Think	MATH-500	76%	27%	73%	2881	70%	25%	68%
8B-Think	GPQA-Diamond	77%	37%	63%	6884	64%	26%	49%
8B-NoThink	MATH-500	90%	48%	52%	696	74%	21%	73%
8B-NoThink	GPQA-Diamond	74%	38%	62%	1601	60%	34%	48%
32B-Think	MATH-500	82%	25%	75%	2878	75%	34%	72%
32B-Think	GPQA-Diamond	81%	33%	67%	6192	73%	34%	67%

Table 4: GPQA-Diamond vs MATH-500. The gap persists on a harder benchmark, but commitment occurs later and baee reduction is correspondingly lower.

Figure 5: (a) Failure taxonomy for 208 gap instances. (b) Gap magnitude across models.

Figure 6: MATH-500 vs GPQA-Diamond. (a,b) PSC vs EFA across prefixes. (c,d) Commitment distributions.

Figure 7: Entropy analysis. (a) Per-token entropy along the CoT. (b) Wrong problems (dashed) show higher entropy. (c) Post/pre-commit ratio separates Think (

>

1) from NoThink (

<

1).

Strategy	GT?	Calls	32B-Think	32B-NoThink	8B-Think	8B-NoThink
Full CoT	—	1	82% / —	91% / —	76% / —	90% / —
Naïve efa	No	$\sim$ 6	34% / 90%	38% / 90%	35% / 90%	28% / 90%
EFA-oracle	Yes	$\sim$ 6	82% / 67%	91% / 48%	76% / 67%	90% / 38%
psc -8 all	No	72	86% / 74%	92% / 78%	81% / 70%	91% / 77%
psc -8 adaptive	No	9 (med.)	82% / 72%	91% / 73%	76% / 68%	90% / 73%

Table 5: Accuracy / main-rollout reduction for each baee strategy. “Reduction” = serial main-rollout tokens avoided ( not total billed tokens; see § 4.5.3 ). psc -8 all probes every checkpoint; psc -8 adaptive stops at first trigger. Format: Acc. / Reduction .

	MATH-500			GPQA-Diamond
Model	$\boldsymbol{\theta^{*}}$	$\boldsymbol{\Delta}$ Acc	Red.	$\boldsymbol{\theta^{*}}$	$\boldsymbol{\Delta}$ Acc	Red.
32B-Think	1.000	$+$ 1.8 pp	67%	0.875	$+$ 1.0 pp	54%
32B-NoThink	0.875	$+$ 0.8 pp	71%	1.000	$\pm$ 0	36%
8B-Think	1.000	$+$ 1.2 pp	63%	0.750	$+$ 3.0 pp	47%
8B-NoThink	1.000	$+$ 1.4 pp	67%	0.750	$+$ 3.0 pp	47%
GPT-OSS-120B	1.000	$+$ 0.4 pp	81%	0.625	$+$ 4.0 pp	72%

Table 6: Calibrated threshold

\theta^{*}

and held-out test performance on MATH-500 (250-problem split) and GPQA-Diamond (99-problem split).

\theta^{*}=1.0

for 4/5 MATH models; GPQA selects lower thresholds, consistent with its harder problems relaxing the FP constraint.

MATH-500
Model	CoT	Ser. red.	Est. tot.	Ratio
32B-Think	2 879	63%	11 228	3.9 $\times$
32B-NoThink	699	55%	3 006	4.3 $\times$
8B-Think	2 881	57%	10 372	3.6 $\times$
8B-NoThink	697	55%	2 927	4.2 $\times$
GPT-OSS-120B	823	76%	4 115	5.0 $\times$

Table 7: Compute redistribution under baee (MATH-500). Estimated totals use measured continuation lengths (Appendix L ). GPQA-Diamond ratios are lower (3.1–5.0

\times

); see Appendix P .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Base acc.	BAEE acc.	$\boldsymbol{\Delta}$ Acc	Main-rollout red.	Exit Rate
8B-Think	75.6%	81.4%	$+$ 5.8 pp	69.8%	80.6%
8B-NoThink	89.8%	90.6%	$+$ 0.8 pp	77.0%	89.4%
32B-Think	82.0%	86.4%	$+$ 4.4 pp	74.3%	85.6%
32B-NoThink	91.4%	92.0%	$+$ 0.6 pp	78.3%	90.6%
GPT-OSS-120B	95.6%	96.2%	$+$ 0.6 pp	85.6%	96.0%

Appendix

Table 8: psc -8 BAEE accuracy and main-rollout reduction under offline majority-vote simulation at

\theta=0.625

(5/8 trigger). Base is full-CoT accuracy;

\Delta

Acc is the change from early exit. “Token reduction” refers to serial main-rollout tokens only. All models: 500 problems.

Figure 8: Offline majority-vote BAEE under aggressive thresholds. (a) Accuracy change vs full CoT across

\theta

thresholds: Think models (solid lines) gain up to +5.8 pp; NoThink models (dashed) can degrade for low

\theta

. (b) Pareto frontier of accuracy vs main-rollout reduction; large dots mark

\theta=0.625

. Dotted lines show full-CoT baseline accuracy for each model.

Figure 9: Outcome breakdown at

\theta=0.625

(offline majority-vote simulation). Green = overthinking corrected (wrong under full CoT, correct under BAEE); red = BAEE harmed (correct under full CoT, wrong under BAEE); blue = always correct; gray = always wrong.

Model	N wrong	Max psc	psc $\geq 50\%$	psc $\geq 75\%$
32B-Think	17	0.50	1 (at 25%)	0
32B-NoThink	43	0.75	3	3
8B-Think	20	0.31	0	0
8B-NoThink	51	0.88	6	2
GPT-OSS-120B	3	0.19	0	0

Appendix

Table 9: Maximum psc accuracy across all checkpoints for problems with 0/4 correct rollouts. NoThink models show nonzero proxy FP at

\theta=0.75

. “N wrong” uses 0/4 correct rollouts as a proxy for unsolvable instances.

Setting ( $\theta$ )	Accuracy	Mean savings	Proxy FP rate
32B-NoThink, $\theta=0.750$	91.4%	76.4%	7.0% (3/43)
32B-NoThink, $\theta=0.875$	91.4%	73.3%	0.0% (0/43)
32B-NoThink, $\theta=1.000$	91.4%	69.2%	0.0% (0/43)
8B-NoThink, $\theta=0.750$	89.8%	75.4%	3.9% (2/51)
8B-NoThink, $\theta=0.875$	89.8%	72.9%	3.9% (2/51)
8B-NoThink, $\theta=1.000$	89.8%	67.8%	0.0% (0/51)

Appendix

Table 10: NoThink threshold sweep (500-problem runs): BAEE simulation accuracy / mean main-rollout reduction / proxy FP rate on 0/4-wrong subset.

Figure 10: Threshold-sweep frontier under PSC-8. Left: NoThink FP–savings trade-off on 500-problem runs. Right: mean savings as a function of $\theta$ across all models, with selected operating points highlighted.

Feature	TP mean	FP mean	Gap
PSC@10% (first checkpoint)	0.85	0.33	$+$ 0.52
Mean PSC across all checkpoints	0.86	0.44	$+$ 0.42
Max PSC	0.99	0.85	$+$ 0.14
PSC spread (max $-$ min)	0.46	0.77	$-$ 0.31
Number of PSC drops	1.3	3.2	$-$ 1.9
Checkpoint of max PSC	22%	44%	$-$ 22%

Appendix

Table 11: Trajectory features distinguishing true positives (correct, high psc ) from false positives (wrong, high psc ). All differences are large and statistically significant (

p<10^{-5}

, permutation test).

Figure 11: Discriminative signals for filtering high- psc false positives. (a) PSC@10% distribution: TPs concentrate at high values; FPs spread across low values. (b) Number of psc drops: TPs are near-monotone; FPs oscillate. (c) FP rate decreases with stricter mean-PSC thresholds on all benchmarks.

Figure 12: PSC monotonicity: MATH-500 (non-monotone) vs GPQA-Diamond (monotone).

Perturbation	$f=0.10$		$f=0.50$
	Mean PSC	Drops > 10 pp	Mean PSC	Drops > 10 pp
Intact (control)	97.5%	—	97.8%	—
Truncate last 20%	96.0%	5/50	97.5%	3/50
Shuffle last 30%	96.9%	6/50	97.0%	3/50
Random replace 30%	94.8%	14/50	88.5%	17/50

Appendix

Table 12: Prefix perturbation results. Even replacing 30% of tokens with random vocabulary items reduces psc by only 2.8 pp at

f=0.10

	32B-Think			8B-NoThink			GPT-OSS-120B
Suffix	EFA	PSC	Gap	EFA	PSC	Gap	EFA	PSC	Gap
original	35%	80%	45	27%	78%	51	26%	94%	68
natural	41%	80%	39	28%	78%	50	23%	94%	71
soft	35%	80%	45	29%	78%	49	18%	94%	76
plain	5%	80%	75	22%	78%	56	0%	94%	94
direct	17%	80%	63	16%	78%	62	11%	94%	83

Appendix

Table 13: efa accuracy at 10% prefix for five forcing templates across three models spanning two families, two scales, and two reasoning modes. The gap persists universally.

	Prefix fraction $f$
Configuration	0.10	0.20	0.30	0.50	0.70
32B-Think	$\geq 0.54$	$\geq 0.44$	$\geq 0.29$	$\geq 0.09$	$\geq 0.05$
8B-Think	$\geq 0.51$	$\geq 0.40$	$\geq 0.25$	$\geq 0.07$	$\geq 0.04$
GPT-OSS	$\geq 0.70$	$\geq 0.55$	$\geq 0.38$	$\geq 0.12$	$\geq 0.06$
32B-NoThink	$\geq 0.41$	$\geq 0.30$	$\geq 0.18$	$\geq 0.06$	$\geq 0.03$

Appendix

Table 14: Lower bounds on

d_{\mathrm{TV}}(P_{\mathrm{free}},\,P_{\mathrm{forced}})

derived from measured gaps via Proposition 2 . Each entry is

\mathrm{gap}_{k}=\textsc{psc}{}(k)-\textsc{efa}{}(k)

Suffix	Mean gap (pp)	$d_{\mathrm{TV}}$ lower bound	Expected shift
natural ( \boxed -like)	53.3	$\geq 0.53$	lowest
original ( \boxed{ )	54.7	$\geq 0.55$	low
soft (“answer is”)	56.7	$\geq 0.57$	moderate
direct (“Final answer:”)	69.3	$\geq 0.69$	high
plain (bare number)	75.0	$\geq 0.75$	highest

Appendix

Table 15: Suffix-ranking calibration. Mean gap (

\geq d_{\mathrm{TV}}

lower bound) across 32B-Think, 8B-NoThink, and GPT-OSS vs. qualitative distributional-shift expectation. The ranking is perfectly monotone (Spearman

\rho=1.0

Fraction	PSC mean	EFA acc	Gap
2%	90.1%	32.4%	57.7 pp
4%	88.6%	32.4%	56.2 pp
5%	90.4%	29.4%	61.0 pp
8%	91.9%	38.2%	53.7 pp
10%	91.2%	35.3%	55.9 pp
15%	91.5%	47.1%	44.5 pp

Appendix

Table 16: Fine-grained checkpoint results (32B-Think, 50 problems). psc agreement is already high at 5%, confirming that the 10% grid does not inflate post-commitment estimates.

Model	Mean commitment fraction by difficulty
Model	L1 (easy)	L2	L3	L4	L5 (hard)
32B-Think	13%	20%	23%	29%	32%
32B-NoThink	32%	28%	35%	45%	51%
8B-Think	18%	23%	26%	30%	33%
8B-NoThink	31%	39%	42%	58%	63%
GPT-OSS-120B	22%	26%	33%	40%	48%

Appendix

Table 17: Mean commitment fraction by problem difficulty level (MATH levels 1–5).

Figure 13: Post-commitment fraction across MATH-500 and GPQA-Diamond. Solid bars = MATH-500, hatched = GPQA-Diamond. The Think–NoThink gap is consistent across benchmarks.

Figure 14: Commitment map for GPQA-Diamond (same visualization as Figure 4 ). The commitment boundary appears later than on MATH-500, consistent with harder problems requiring more genuine reasoning, but substantial post-commitment generation remains (61–67%).

		BAEE		SC-8-full
Bench	Model	Acc	Ratio	Acc	Ratio	$\boldsymbol{\Delta}$ Acc
MATH-500	32B-Think	85.0%	4.1 $\times$	71.8%	8.0 $\times$	+13.2
	32B-NoThink	91.4%	3.6 $\times$	81.4%	8.0 $\times$	+10.0
	8B-Think	80.0%	3.9 $\times$	65.2%	8.0 $\times$	+14.8
	8B-NoThink	90.2%	3.6 $\times$	77.2%	8.0 $\times$	+13.0
	GPT-OSS-120B	96.2%	4.8 $\times$	92.4%	8.0 $\times$	+3.8

Appendix

Table 18: BAEE vs SC-8-full: accuracy and token cost on MATH-500 and GPQA-Diamond. “Ratio” = total tokens / single full-CoT tokens. BAEE achieves higher accuracy at lower total cost on both benchmarks.

Model	Acc.	Solvable	Commit	Post-commit
32B-Think	64.6%	138/164	14.7%	85.3%
32B-NoThink	82.9%	154/164	13.3%	86.7%
8B-Think	64.0%	130/164	14.1%	85.9%
8B-NoThink	74.4%	149/164	12.4%	87.6%

Appendix

Table 19: HumanEval results (4 models, 164 problems). Post-commitment fractions are the highest across all benchmarks (85–88%), and the Think–NoThink gap nearly vanishes (

<

2 pp).

Model	Full CoT	BAEE	$\boldsymbol{\Delta}$ Acc	Reduction
32B-Think	64.6%	78.2%	+13.6	71.1%
32B-NoThink	82.9%	90.2%	+7.3	78.2%
8B-Think	64.0%	77.6%	+13.6	71.3%
8B-NoThink	74.4%	82.3%	+7.9	72.2%

Appendix

Table 20: BAEE on HumanEval (

\theta=0.75

). Think models gain +13.6 pp accuracy, the largest overthinking correction across all benchmarks.

Figure 15: HumanEval results. (a) psc agreement across prefix fractions. (b) efa accuracy (code extraction is harder than math extraction). (c) Commitment distributions: all models commit by 10–20%. (d) Detection–extraction gap persists on code generation.

Figure 16: Post-commitment fraction across three benchmarks (MATH-500, GPQA-Diamond, HumanEval) and five models. HumanEval shows the highest post-commitment fraction (85–88%) across all models, with a near-zero Think–NoThink gap. GPT-OSS-120B is not available for HumanEval (API limitation).

		MATH-500		GPQA-D
Method	Access	$\Delta$ Acc	CR	$\Delta$ Acc	CR
Qwen3-14B (DEER) / Qwen3-8B-Think (ours)
Vanilla	—	—	100%	—	100%
NoThinking	black-box	$-$ 5.6	27%	$-$ 9.5	32%
TCC	black-box	$-$ 0.6	61%	$-$ 0.5	61%
Dynasor-CoT	black-box	$-$ 0.2	72%	$-$ 0.4	40%

Appendix

Table 21: Pareto comparison with concurrent methods on shared benchmarks. DEER numbers from Yang et al. [2025b] Table 1 (Qwen3-14B as closest size match to our 8B/32B); our numbers from Tables 2 – 4 . “CR” = compression rate (lower = more reduction).

\Delta

Acc = accuracy change vs vanilla full CoT.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

Large Language Models Decide Early and Explain Later
Apr 24, 2026Ayan Datta, Zhixue Zhao, Bhuvanesh Verma +3Efficient Language Model Reasoning LLM Inference Acceleration
IIIT Hyderabad · University of Sheffield, UK · Goethe University, Frankfurt am Main, Germany
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Mar 5, 2026Siddharth Boppana, Annabel Ma, Max Loeffler +5Faithfulness of Language Model Reasoning Language Model Reasoning Evaluation
Goodfire AI · Harvard University, Cambridge, MA
</think> Doesn't Stop Reasoning: Analysis of Spurious CoT Termination
Sep 3, 2026Seunghee Koh, Sungjae Choi, Minchan Kwon +2CoT Reasoning Efficient Language Model Reasoning
Korea Advanced Institute of Science and Technology, South Korea

Apr 24, 2026J/K move · Enter open · S save

Large Language Models Decide Early and Explain Later

Ayan Datta, Zhixue Zhao, Bhuvanesh Verma, Radhika Mamidi+2

IIIT Hyderabad · University of Sheffield, UK · Goethe University, Frankfurt am Main, Germany

Efficient Language Model Reasoning LLM Inference Acceleration Early Stopping for Language Model Reasoning LLM Reasoning Reasoning Trace Analysis

Open paper AskPDF arXiv

Large Language Models often achieve strong performance by generating long intermediate chain-of-thought reasoning. However, it remains unclear when a model's final answer is actually determined during generation. If the answer is already fixed at an intermediate stage, subsequent reasoning tokens may constitute post-decision explanation, increasing inference cost and latency without improving correctness. We study the evolution of predicted answers over reasoning steps using forced answer completion, which elicits the model's intermediate predictions at partial reasoning prefixes. Focusing on Qwen3-4B and averaging results across all datasets considered, we find that predicted answers change in only 32% of queries. Moreover, once the final answer switch occurs, the model generates an average of 760 additional reasoning tokens per query, accounting for a substantial fraction of the total reasoning budget. Motivated by these findings, we investigate early stopping strategies that halt generation once the answer has stabilized. We show that simple heuristics, including probe-based stopping, can reduce reasoning token usage by 500 tokens per query while incurring only a 2% drop in accuracy. Together, our results indicate that a large portion of chain-of-thought generation is redundant and can be reduced with minimal impact on performance.

Abstract

Efficient Language Model Reasoning

Explore similar work

Large Language Models Decide Early and Explain Later

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

</think> Doesn't Stop Reasoning: Analysis of Spurious CoT Termination