On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

Jaewon Sok; Jewon Yeom; Seonghyeon Park; Jeongjae Park; Hwiyeong Lee; Taesup Kim

cs.CLOct 16, 2026

On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

Authors: Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim

Organizations: Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University

Abstract

Large language models often answer factual questions incorrectly in closed-book settings, and such errors are usually attributed to missing knowledge. Yet models often encode answers they fail to produce. We show that many closed-book errors are first-token recall failures: the model commits to a wrong first token, but once given the right one it completes the answer and often recognizes it as correct. We study this with First-Token Cueing, a behavioral probe that prefills the response with the first token of the gold answer. Model uncertainty peaks at the first answer token, where it already predicts correctness, and among gold prefixes of any length the first token alone accounts for the largest share of recovered errors. Across 16 open models, this one token corrects between a fifth and a half of the errors on multi-token answers in English QA and improves accuracy in all seven languages of a multilingual benchmark. Most of the errors it corrects are also solved by the model's own sampled reasoning, a criterion for encoded knowledge, and models recognize correct starts they do not produce: they continue a correct start more confidently than an incorrect start of the same answer type and prior probability, yet rarely find it without the gold answer. These results locate a recoverable class of factual errors at the first decoding step, where inference-time methods may gain the most.

Figures & tables

Figure 1: Phonemic cueing in human tip-of-the-tongue states (left; Brown and McNeill, 1966 ; James and Burke, 2000 ) and First-Token Cueing in LLMs (right). In both, production fails at the start of the answer, and supplying its first unit, a phoneme or a token, lets the speaker or the model complete it.

Figure 2: Uncertainty localizes to the first token. AUROC and PRR (left) and mean entropy (right) by answer-token position for Llama-3.1 8B on TriviaQA. The first token has the highest entropy, and its entropy predicts correctness nearly as well as the average over the whole answer.

Figure 3: First-Token Cueing improves accuracy for every model. Average accuracy on TriviaQA, NQ-Open, and WebQuestions for 16 models, Baseline (light) vs. First-Token Cueing (dark), on questions whose gold answer has at least two tokens. Annotations give absolute (pp) and relative (%) gains. Results on all questions are in Figure 12 .

Figure 4: First-Token Cueing improves accuracy in every language for every model on MKQA. Each cell shows the gain (Cueing $-$ Baseline, pp) and the Baseline accuracy in parentheses, on questions with at least two gold tokens. Languages are sorted by mean Baseline accuracy. Results on all questions are in Figure 13 .

Figure 5: The first token covers less of the answer in non-Latin scripts. Mean length of the gold first token in characters (top), share of questions whose first token is a single character (middle), and mean gold answer length in tokens (bottom), averaged over the Gemma-3, Llama-3, and Qwen-2.5 tokenizers. English QA pools TriviaQA, NQ-Open, and WebQuestions; per tokenizer in Table 20 .

Figure 6: The first token alone recovers more errors than all longer prefixes combined. Shortest gold prefix

k

that makes a Baseline error correct, for Llama-3.1 8B. Curve

L\geq\ell

restricts to gold answers of at least

\ell

tokens; “none” marks errors not recovered by the full gold answer.

			Errors recovered (%)
Model	Stratum	Errors	Cue	Oracle
Llama-3.1 8B	IN	2,004 (43%)	54.0	63.0
	NOT-IN	2,624 (57%)	40.4	24.8
Llama-3.1 70B	IN	1,265 (41%)	49.6	57.9
	NOT-IN	1,793 (59%)	44.6	22.6

Table 1: The cue recovers errors beyond the model’s own first-token candidates. Baseline errors recovered by First-Token Cueing and by the oracle over these candidates (judge; three English benchmarks pooled;

L\geq 2

). IN: a top-

p

candidate is a prefix of the gold first token; NOT-IN: none is. Strict EM in Table 11 .

Figure 7: Only sampled chain-of-thought, scored by pass@16, reaches the accuracy of First-Token Cueing. Accuracy of First-Token Cueing and five baselines (direct or CoT prompt, greedy or sampled decoding) for the Gemma-3 family (

L\geq 2

), English QA average. Annotations: Cueing

-

pass@16 (CoT sampling), pp.

Figure 8: Models continue a correct start more confidently than a type-matched incorrect one. Share of NOT-IN questions whose continuation has lower entropy after the correct first token than after the incorrect one, by the prior gap between the two starts (shaded: matched prior). Error bars: 95% Wilson intervals.

Model	$n$	Cor.	Inc.	Jacc.	Pref.	Base	Sel.	Cue
		Changed				EM
Llama-3.1 8B	2,332	74	78	0.76	66.7	15.3	25.3	30.1
Llama-3.1 70B	1,485	60	63	0.62	65.0	26.9	29.5	35.7
Qwen-2.5 7B	2,845	83	89	0.82	64.2	11.6	25.3	31.1
Gemma-3 12B	1,760	80	83	0.78	60.0	14.7	25.8	34.2

Table 2: Models follow either cue but continue the correct one more confidently. Crossed-cue control at matched prior (the two starts differ by at most one nat in log-probability; NOT-IN questions; %). Changed: answer differs from the Baseline under the correct (Cor.) or incorrect (Inc.) cue; Jacc.: overlap of the two changed sets; Pref.: the correct-cue continuation has lower mean entropy (chance 50; 95% Wilson intervals within

\pm 2.5

; binomial

p<10^{-16}

for every model); EM of the Baseline, of the continuation selected by this confidence (Sel.), and of the correct cue.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Year	Description	License
TriviaQA ( Joshi et al., 2017 )	2017	Open-domain trivia questions with Wikipedia-grounded short answers. We use the rc.wikipedia.nocontext subset, validation split.	Apache 2.0
NQ-Open ( Kwiatkowski et al., 2019 )	2019	Open-domain variant of Natural Questions: real Google search queries paired with short Wikipedia-grounded answers. Validation split.	CC BY-SA 3.0
WebQuestions ( Berant et al., 2013 )	2013	Real-world questions issued to the Google Suggest API, paired with Freebase-grounded short answers. Test split.	CC BY 4.0
MKQA ( Longpre et al., 2021 )	2021	Multilingual Knowledge Questions and Answers: NQ-Open queries translated and answered in 26 languages. We use 7 languages (en, es, ru, ar, ja, zh_cn, ko).	CC BY-SA 3.0

Appendix

Table 3: Datasets used in this study, with year of publication, description, split, and license. All datasets are used in accordance with their stated license.

Condition	Prompt	Max tokens	Temp.	Top- $p$	$n$
Baseline	Direct	50	0.0	–	1
First-Token Cueing	Direct + cue	50	0.0	–	1
Greedy CoT	CoT	1024	0.0	–	1
pass@16 (Direct sampling)	Direct	50	0.7	0.95	16
pass@ $k$ (CoT sampling)	CoT	1024	0.7	0.95	16

Appendix

Table 4: Decoding hyperparameters per condition. “Direct” uses Prompt A.4 ; “CoT” uses Prompt A.4 . “Direct + cue” denotes the direct-answer prompt with the gold answer’s first token prepended to the assistant response. pass@

k

(CoT sampling) shares the same

n=16

sample pool for

k\in\{4,16\}

	Sample			Reweighted to $L\geq 2$
	Agree.	$\kappa$	FP / FN	Agree.	Gain judge	Gain labels
EN	90.0	0.80	0 / 5	91.5	$+23.8$	$+30.4$
KO	92.0	0.84	1 / 3	90.6	$+18.8$	$+23.0$
JA	98.0	0.96	0 / 1	96.6	$+19.5$	$+12.7$
ZH	92.0	0.84	0 / 4	89.1	$+18.2$	$+26.2$
Baseline	95.0	0.90	1 / 4	93.2	–	–

Appendix

Table 5: The judge agrees with author-checked labels and errs toward rejecting correct responses. Llama-3.1 8B on MKQA (

L\geq 2

); 50 responses per language, half Baseline and half First-Token Cueing, stratified by the judge’s verdict. Agree.: agreement with the labels (%). FP: the judge accepts an incorrect response; FN: it rejects a correct one. Reweighted columns weight each verdict stratum by its share in the full

L\geq 2

subset; Gain: Cue

-

Baseline accuracy (points) under the judge and under the labels, averaged over languages in the last row.

Model	$n$	Errors	Corrected (%)	Broken (%)	Ratio
Gemma-3 270M	11,593	10,875	21.6	37.2	8.8
Gemma-3 1B	11,593	9,253	32.7	18.8	6.9
Gemma-3 4B	11,593	6,623	41.4	9.3	5.9
Gemma-3 12B	11,593	4,829	43.3	6.4	4.9
Gemma-3 27B	11,593	3,856	44.6	4.9	4.6
Llama-3.2 1B	12,253	8,160	44.2	12.4	7.1

Appendix

Table 6: Errors corrected and answers broken by First-Token Cueing on English QA (

L\geq 2

; TriviaQA, NQ-Open, and WebQuestions pooled; judge).

n

: questions; Errors: Baseline errors; Corrected: share of Baseline errors that the cue makes correct; Broken: share of Baseline-correct answers that the cue makes incorrect; Ratio: corrected over broken answers.

		Gemma-3					Llama 3.1/3.2				Qwen-2.5
Task	Metric	270M	1B	4B	12B	27B	1B	3B	8B	70B	0.5B	1.5B	3B	7B	14B	32B	72B
TriviaQA	Baseline	6.7	23.9	54.1	74.4	83.0	40.2	63.3	77.0	90.9	16.8	36.9	44.9	58.1	72.5	74.8	83.2
	Cueing	29.0	52.5	75.5	86.5	92.2	68.1	81.6	89.2	94.9	46.9	65.6	70.3	78.6	86.0	86.9	91.6
	$\Delta_{\mathrm{pp}}$	+22.3	+28.6	+21.4	+12.1	+9.2	+27.9	+18.3	+12.2	+4.0	+30.1	+28.7	+25.4	+20.4	+13.5	+12.2	+8.4
	$n$	6,560	6,560	6,560	6,560	6,560	7,075	7,075	7,075	7,075	7,157	7,157	7,157	7,157	7,157	7,157	7,157
NQ-Open	Baseline	3.3	11.7	24.2	33.7	43.9	20.5	33.0	40.0	53.9	9.0	16.8	20.3	27.8	37.3	37.5	44.6

Appendix

Table 7: Cueing improves over Baseline in every cell (

160

160

) of the headline subset, questions whose gold answer has at least two tokens under the model’s tokenizer. Baseline and First-Token Cueing judge accuracy (%), absolute improvement

\Delta_{\mathrm{pp}}

, and the number of questions

n

for all

16

models on the three English QA benchmarks and the seven MKQA languages.

Figure 9: Cueing exceeds Baseline at every gold answer length in every model. Baseline vs. First-Token Cueing accuracy by gold answer length (tokens) for all sixteen models, on all questions. Sample sizes appear below each pair.

Figure 10: The first token is the largest group at every answer length in 13 of the 16 models. Distribution of the shortest gold prefix

k

that makes a Baseline error correct. Each curve

L\geq\ell

restricts to gold answers of at least

\ell

tokens;

8{+}

collects

k\geq 8

, and none marks errors not recovered by any prefix up to the full gold answer. Sample sizes (

n

) appear in each panel’s legend.

Figure 11: A random first token lowers accuracy in every model. Average accuracy over TriviaQA, NQ-Open, and WebQuestions, Baseline (solid) vs. Random-Token (dashed), for all

16

models. Annotations give the absolute (pp) and relative (%) change.

		Gemma-3					Llama 3.1/3.2				Qwen-2.5
Task	Metric	270M	1B	4B	12B	27B	1B	3B	8B	70B	0.5B	1.5B	3B	7B	14B	32B	72B
TriviaQA	Baseline	8.5	26.9	56.6	75.7	83.9	41.5	64.3	77.2	90.9	18.8	39.0	46.9	59.5	73.5	75.6	83.8
	Random	0.5	5.4	13.5	19.6	33.9	6.4	14.5	53.1	40.9	3.1	5.8	19.2	27.0	57.3	48.9	63.3
	$\Delta_{\mathrm{pp}}$	$-8.0$	$-21.5$	$-43.1$	$-56.1$	$-50.0$	$-35.1$	$-49.8$	$-24.1$	$-50.0$	$-15.7$	$-33.2$	$-27.7$	$-32.4$	$-16.2$	$-26.7$	$-20.4$
NQ-Open	Baseline	3.9	12.7	25.5	35.2	44.8	20.9	33.3	40.4	54.0	9.6	17.4	21.1	28.9	38.0	38.0	45.3
	Random	0.3	3.8	7.2	9.7	18.4	4.0	9.6	27.5	29.2	2.1	3.0	8.4	14.1	29.3	24.2	35.5

Appendix

Table 8: A random first token lowers accuracy in every (model, task) cell (

48

48

), by

22.7

pp on average. Baseline and Random-Token accuracy (%) and the absolute change

\Delta_{\mathrm{pp}}

for all

16

models on the three English QA benchmarks.

	Exact Match (%)			Token F1
Model	Base	Cue	$\Delta$	Base	Cue	$\Delta$
Gemma-3 270M	3.10	25.44	+22.34	10.12	43.86	+33.73
Gemma-3 1B	10.47	30.72	+20.25	18.40	48.07	+29.67
Gemma-3 4B	27.42	46.56	+19.15	37.12	63.01	+25.90
Gemma-3 12B	39.71	55.94	+16.23	49.52	71.51	+21.99
Gemma-3 27B	43.66	55.95	+12.29	55.16	72.08	+16.91

Appendix

Table 9: Cueing improves exact match and token-level F1 for every model. Strict alias-aware EM and token-level F1 (max over aliases;

0

–

100

scale), Baseline vs. First-Token Cueing, averaged over TriviaQA, NQ-Open, and WebQuestions (all questions).

Language	Base	Cue	$\Delta$	Base	Cue	$\Delta$
	Exact Match (%)			Token F1
EN	18.3	39.2	$+20.9$	26.6	53.9	$+27.4$
ES	13.3	33.5	$+20.2$	19.2	46.9	$+27.7$
RU	8.1	17.0	$+8.9$	12.4	25.5	$+13.1$
AR	4.3	13.2	$+8.9$	7.8	23.0	$+15.3$
JA	3.7	9.5	$+5.8$	25.1	50.8	$+25.7$

Appendix

Table 10: Cueing improves exact match and token-level F1 in every MKQA language. Mean over the

16

models on questions whose gold answer has at least two tokens, with the official MKQA normalization (Chinese and Japanese F1 over characters). Cueing improves both metrics in all

112

(model, language) cells.

Model	Stratum	$n$	Base	Cue	Oracle	Cue $-$ Oracle
Gold answers of at least two tokens
Llama-3.1 8B	IN	7,620	62.2	75.6	77.3	$-1.7$
	NOT-IN	4,633	28.5	42.4	35.4	$+7.0$
Llama-3.1 70B	IN	8,149	73.5	81.4	82.5	$-1.1$
	NOT-IN	4,104	40.9	49.1	45.1	$+4.0$
All questions

Appendix

Table 11: The oracle comparison of Table 1 holds under strict EM. Cueing vs. the oracle over the model’s own first-token candidates (accuracy, %; three English benchmarks pooled), on questions with at least two gold tokens and on all questions.

Stratum	$\|C\|$	$n$ (8B / 70B)	8B	70B	8B	70B
			Cue $-$ Base		Cue $-$ Oracle
IN	1	3,221 / 5,044	$-0.2$	$+0.0$	$-0.3$	$-0.0$
	2	1,550 / 1,917	$+8.7$	$+9.4$	$-1.7$	$-1.0$
	3–4	1,034 / 943	$+18.9$	$+19.8$	$-3.6$	$-3.4$
	5–8	721 / 492	$+22.1$	$+22.8$	$-2.1$	$-3.5$
	9+	2,106 / 822	$+34.9$	$+32.1$	$-1.4$	$-2.0$

Appendix

Table 12: Stratification by candidate-set size

|C|

(pp; pooled over the three English QA benchmarks). The gross gain grows with

|C|

, but the excess over the oracle does not: it is

\leq 0

10/10

IN cells and

>0

10/10

NOT-IN cells, including

|C|=1

, where there is nothing to select among.

Stratum	Ambiguity	$n$ (8B / 70B)	8B	70B
			Cue $-$ Oracle
IN	1 (unique)	880 / 946	$-0.3$	$+0.0$
	2–5	1,834 / 1,972	$-0.2$	$-0.6$
	6–20	1,045 / 1,129	$-1.1$	$-0.9$
	21+	567 / 587	$-2.5$	$-2.7$
NOT-IN	1 (unique)	748 / 682	$+8.4$	$+8.1$

Appendix

Table 13: Stratification by prefix ambiguity: the number of distinct normalized gold answers in the dataset that share the gold first token (token length

\geq 3

; questions whose first token is the full answer excluded; pp, pooled over the three English QA benchmarks). The excess over the oracle peaks at intermediate ambiguity, not for unique prefixes.

Model	Dataset	$n$	Base	Gold cue	Plausible wrong cue
Llama-3.1 8B	TriviaQA	5,262	66.3	86.1	51.7
	NQ-Open	3,280	35.4	58.2	27.4
	WebQ	1,725	38.9	53.9	30.6
Llama-3.1 70B	TriviaQA	3,991	82.7	91.8	70.4
	NQ-Open	2,927	44.6	65.8	37.4
	WebQ	1,569	42.5	57.1	35.6

Appendix

Table 14: Plausible-but-incorrect first-token cue (judge accuracy, %): the highest-probability non-gold token of the model’s own top-

p

set, continued greedily;

n

covers questions with at least one non-gold candidate. It restricts the output space through the same mechanism as the gold cue, yet reduces accuracy below Baseline. Strict EM shows the same ordering (Table 16 ).

Dataset	$n$	Base	Output prefilling	Prefill $-$ Base	Cue $-$ Base
TriviaQA	7,993	77.2	78.0	$+0.8$	$+13.0$
NQ-Open	3,610	40.6	41.3	$+0.7$	$+20.8$
WebQ	2,032	47.5	46.8	$-0.7$	$+12.7$

Appendix

Table 15: Generic output prefilling ( Cappelletti et al., 2025 ) vs. gold First-Token Cueing on open-ended QA (Llama-3.1 8B, judge accuracy, %, all questions). Output prefilling prepends “Given the question, my answer is: ” (no gold information). The two interventions were run as separate passes, so their Baselines differ marginally (cueing-run Baselines:

77.2

40.4

47.0

); each delta is computed against its own run.

Model	Dataset	$n$	Base EM	Gold-cue EM	Plausible-wrong-cue EM
Llama-3.1 8B	TriviaQA	5,262	56.0	78.9	38.5
	NQ-Open	3,280	18.9	38.4	10.3
	WebQ	1,725	13.0	23.1	5.0
Llama-3.1 70B	TriviaQA	3,991	72.8	85.6	57.4
	NQ-Open	2,927	25.4	44.5	14.7
	WebQ	1,569	13.5	22.1	6.8

Appendix

Table 16: Plausible-but-incorrect first-token cue, strict EM (%).

Model	Thinking pass@16	$n$	Share of failures	Cue recovery
Llama-3.1 8B	recoverable	2,745	59.3%	59.9%
	not recoverable	1,883	40.7%	26.6%
Llama-3.1 70B	recoverable	1,442	47.2%	61.0%
	not recoverable	1,616	52.8%	33.8%

Appendix

Table 17: Cueing recovery vs. thinking-based recovery. Baseline failures (judge; three English QA benchmarks pooled; gold answers of at least two tokens) split by whether inference-time thinking (pass@16 over

16

CoT samples) recovers them, the criterion Calderon et al. (2026) use to mark facts as encoded but not directly recallable. Cueing recovery concentrates on the failures this criterion marks as encoded.

	Gemma-3					Llama 3.1/3.2				Qwen-2.5
First-token length	270M	1B	4B	12B	27B	1B	3B	8B	70B	0.5B	1.5B	3B	7B	14B	32B	72B
1 char	+9.5	+12.8	+14.5	+10.7	+8.2	+21.2	+17.0	+12.4	+6.0	+16.9	+17.3	+18.2	+16.7	+10.8	+9.8	+8.1
2 chars	+11.5	+15.4	+16.2	+13.0	+10.0	+24.2	+17.1	+13.6	+9.5	+20.5	+20.1	+17.7	+15.3	+10.1	+10.2	+10.1
3 chars	+17.8	+17.0	+18.5	+15.8	+12.3	+25.7	+19.6	+13.0	+9.5	+22.4	+24.9	+23.3	+19.7	+14.5	+12.6	+13.0
4+ chars	+31.5	+25.5	+22.3	+19.5	+14.7	+29.9	+23.2	+16.2	+12.1	+29.2	+31.2	+27.2	+23.4	+17.2	+17.0	+16.6

Appendix

Table 18: Cueing gain by the character length of the gold first token (strict EM, Cue

-

Baseline in pp, averaged over TriviaQA, NQ-Open, and WebQuestions). All

64

cells are positive, and the 4+-character bucket is the largest for every model.

	mean	first-token length (chars)				tok	tokens
Family	chars	1	2	3	4+	= ans.	/ answer
Gemma-3	3.84	19.9%	10.1%	22.9%	47.1%	15.0%	3.21
Llama-3	3.16	25.6%	16.2%	26.2%	32.1%	10.1%	3.41
Qwen-2.5	3.06	31.3%	14.6%	22.0%	32.1%	9.0%	3.60

Appendix

Table 19: Gold first-token statistics per tokenizer family, pooled over TriviaQA, NQ-Open, and WebQuestions (

n=13{,}635

). “tok = ans.” is the share of questions whose first token is already the complete answer.

	First-token chars	1-char first token (%)	Tokens per answer
Data	(G / L / Q)	(G / L / Q)	(G / L / Q)
English QA	3.84 / 3.16 / 3.06	19.9 / 25.6 / 31.3	3.21 / 3.41 / 3.60
MKQA EN	3.23 / 3.14 / 2.64	38.3 / 19.5 / 46.8	4.04 / 3.60 / 4.34
ES	3.01 / 2.88 / 2.39	39.5 / 21.9 / 49.3	4.23 / 3.96 / 4.69
RU	1.57 / 1.78 / 1.25	64.9 / 56.3 / 87.7	5.68 / 5.45 / 6.48
AR	1.63 / 1.99 / 1.70	58.5 / 38.1 / 54.3	5.49 / 5.08 / 5.82

Appendix

Table 20: Gold first-token statistics by tokenizer family (G: Gemma-3, L: Llama-3, Q: Qwen-2.5): mean length of the first token in characters, share of single-character first tokens, and mean gold answer length in tokens. English QA pools TriviaQA, NQ-Open, and WebQuestions; the other rows are MKQA languages. Plotted in Figure 5 .

	Llama-3.1 8B ( $4{,}608$ eligible)				Llama-3.1 70B ( $3{,}995$ )				Qwen-2.5 7B ( $5{,}481$ )				Gemma-3 12B ( $6{,}324$ )
Caliper	pairs	cov.	pref.	adv.	pairs	cov.	pref.	adv.	pairs	cov.	pref.	adv.	pairs	cov.	pref.	adv.
1 nat	873	37.9	65.6	+0.37	469	23.5	63.4	+0.34	1,109	40.5	63.4	+0.31	513	16.2	60.1	+0.08
2 nats	1,171	50.8	65.5	+0.37	710	35.5	64.5	+0.32	1,488	54.3	64.0	+0.31	843	26.7	59.4	+0.07
3 nats	1,371	59.5	66.7	+0.38	878	44.0	65.2	+0.31	1,681	61.3	64.4	+0.31	1,074	34.0	60.5	+0.07
5 nats	1,615	70.1	68.4	+0.38	1,174	58.8	68.3	+0.32	1,951	71.2	65.6	+0.30	1,372	43.4	60.8	+0.07
none	2,297	99.7	74.8	+0.41	1,989	99.6	76.5	+0.36	2,707	98.8	68.4	+0.31	3,147	99.5	65.8	+0.07

Appendix

Table 21: Crossed-cue control under relaxed prior matching. cov.: % of eligible questions in a pair; pref.: % of questions whose correct-cue continuation is more confident (no prior stratification); adv.: confidence advantage of the correct start at equal prior (regression intercept, nats of entropy). The advantage is stable as coverage grows.

	All	Greedy first char. wrong
Method	( $n=13{,}635$ )	( $n=6{,}055$ )
Greedy (Baseline)	50.0	12.1
Beam search (width 8)	53.0	16.9
Self-consistency (16 samples)	51.6	15.0
36 characters, total log-prob	52.2	17.2
36 characters, continuation log-prob	45.7	20.5

Appendix

Table 22: Gold-free search over first characters (Llama-3.1 8B, strict EM, %). Searching over all first characters with the model’s own scores gains about as much as beam search; the correct candidate is usually in the set but not selected.

Model	AUROC	Model	AUROC
Gemma-3 270M	0.633	Llama-3.1 70B	0.832
Gemma-3 1B	0.668	Qwen-2.5 0.5B	0.729
Gemma-3 4B	0.757	Qwen-2.5 1.5B	0.801
Gemma-3 12B	0.778	Qwen-2.5 3B	0.758
Gemma-3 27B	0.763	Qwen-2.5 7B	0.739
Llama-3.2 1B	0.773	Qwen-2.5 14B	0.704

Appendix

Table 23: First-token entropy predicts Baseline correctness (AUROC of negative first-token entropy for judge correctness; three English QA benchmarks pooled).

Rank of gold first token	Share of errors (%)	Fixed by the cue (%)
1 (the greedy first token)	7.4	5.5
2–5	21.6	65.5
6–50	30.0	50.7
51–1,000	27.4	37.5
$>1{,}000$	13.6	30.3

Appendix

Table 24: Recovery by the rank of the gold first token under the Baseline first-step distribution (Baseline errors, three English QA benchmarks; mean over

16

models). Errors whose gold first token was the model’s runner-up are fixed most often; when the greedy first token already is the gold token, the cue leaves the input unchanged.

		Gemma-3					Llama 3.1/3.2				Qwen-2.5
Task	Metric	270M	1B	4B	12B	27B	1B	3B	8B	70B	0.5B	1.5B	3B	7B	14B	32B	72B
TriviaQA	Baseline	8.5	26.9	56.6	75.7	83.9	41.5	64.3	77.2	90.9	18.8	39.0	46.9	59.5	73.5	75.6	83.8
	Cueing	40.3	59.9	79.5	88.7	93.4	71.4	83.5	90.2	95.4	51.7	68.9	73.1	80.5	87.2	88.1	92.3
	$\Delta_{\mathrm{pp}}$	+31.8	+33.0	+22.9	+12.9	+9.4	+29.9	+19.3	+13.0	+4.6	+33.0	+29.9	+26.2	+21.1	+13.7	+12.5	+8.6
	$\Delta\%$	+373.4	+122.5	+40.4	+17.0	+11.3	+72.2	+29.9	+16.9	+5.0	+175.4	+76.8	+55.9	+35.5	+18.7	+16.5	+10.2
NQ-Open	Baseline	3.9	12.7	25.5	35.2	44.8	20.9	33.3	40.4	54.0	9.6	17.4	21.1	28.9	38.0	38.0	45.3

Appendix

Table 25: Cueing also improves over Baseline in every cell (

160

160

) on all questions, including single-token gold answers. Baseline and First-Token Cueing judge accuracy (%), absolute improvement

\Delta_{\mathrm{pp}}

, and relative improvement

\Delta\%

for all

16

models on the three English QA benchmarks and the seven MKQA languages.

Figure 12: All questions (including single-token answers). Counterpart of Figure 3 : average open-domain QA accuracy (TriviaQA, NQ-Open, WebQuestions), Baseline (light) vs. First-Token Cueing (dark). Annotations give absolute (pp) and relative (%) gains.

Figure 13: All questions (including single-token answers). Counterpart of Figure 4 : Cueing $-$ Baseline accuracy gap (pp) on MKQA for $16$ models $\times$ $7$ languages; per-cell Baseline accuracy in parentheses. All $112$ cells are positive.

Factual Knowledge in Language Models

Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.

See all 195 papers

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode
Mar 14, 2026Valeria Ruscio, Keiran ThompsonHallucination in Language Models LLM Interpretability
Intuition Machines Inc.
Locating Answer-Correctness Signals in Frozen Large Language Models
Sep 29, 2026Yuansen Liu, Yixuan Tang, Anthony Kum Hoe TungLLM Answer Verification Language Model Error Detection
School of Computing, National University of Singapore
Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer
Oct 26, 2026Xinyi NiLLM Prompting Knowledge Conflicts in Language Models
Brandeis University

Mar 14, 2026J/K move · Enter open · S save

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode

Valeria Ruscio, Keiran Thompson

Intuition Machines Inc.

Hallucination in Language Models LLM Interpretability Factual Knowledge in Language Models Causal Interventions in Language Models

Open paper AskPDF arXiv

A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decodability and selection, we distinguish read from write at the first answer token. Read asks whether the gold token can be decoded from intermediate residual states under same-relation decoy controls. Write asks whether the final readout ranks that token first among content tokens. Under three different readers, with a randomized-label control, a substantial fraction of failures remain readable while another content token is selected. We explain this through the selection margin at the final readout, the difference between the answer logit and the logit of its strongest alternative, which is answer support minus alternative support, and can also be split into a context-averaged baseline linked to token frequency and an item-specific term. Setting the answer support to the level typical of successful generations is sufficient to recover first-token selection for the majority of failures in most of the models we study; the original alternative remains ahead in most remaining failures under this edit, and this outcome follows directly from the readout geometry. Removing the frequency direction alone shifts selection but rarely recovers the answer. Prompt variants of the same fact that succeed supply support that transfers to failing variants through the residual stream and through late MLP outputs, with less consistent effects through late attention. First-token recovery leaves most full answers wrong, which limits the recovery achieved by these edits and separates three things that are easily conflated, decodability, recoverability, and generation.

Abstract

Factual Knowledge in Language Models

Explore similar work

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode

Locating Answer-Correctness Signals in Frozen Large Language Models

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer