FlashAttention with Statistics: Low-Overhead Attention Features for Compact Hallucination Detectors

Andrei Volodichev; Maksim Savichev; Nikita Sviridov; Artem Yuliy; Dmitry Simakov; Alexey Zaytsev

cs.AIOct 15, 2026

FlashAttention with Statistics: Low-Overhead Attention Features for Compact Hallucination Detectors

Authors: Andrei Volodichev, Maksim Savichev, Nikita Sviridov, Artem Yuliy, Dmitry Simakov, Alexey Zaytsev

Organizations: Applied AI Institute · Yandex · Risk AI Research Lab

Abstract

Large language models can produce fluent answers that the supplied context does not support. A compact frozen proxy model can detect such answers by reading the context and another model's response, even from a proprietary generator. Attention-based detectors use summary statistics of the proxy's attention weights, but standard pipelines materialize the attention matrix, whose memory grows quadratically with sequence length. We compute these statistics without materializing attention, by accumulation in the FlashAttention pass, sparse lookups after it, or a second tiled pass over the scores, and classify them by this path into nearly free and costlier ones. On an H100, the statistics of five published detectors add 16% to the latency of a Qwen3-8B forward at 2K tokens, falling to 1.4% at 35K, and store 22.5 KiB per response token. On three benchmarks, probes trained on frozen Qwen3 proxies of 0.6B to 8B parameters prove strong detectors: a 4B proxy outperforms the strongest of nine zero-shot 4-9B LLM judges in F1 on two of the three, the proxies match or outperform specialized grounding checkers of up to 7B parameters, probing each proxy beats prompting it as a judge, and adding attention features significantly improves F1 in half of the 12 benchmark-proxy pairs, never significantly lowering it.

Figures & tables

Figure 1: Cost of extracting all implemented attention statistics for hallucination detection in every layer of Qwen3-8B and Qwen3-0.6B using our kernels. Top: model forward-pass latency. Bottom: relative overhead compared with the same forward using standard scaled dot-product attention (SDPA), without statistic extraction.

Figure 2: Attention with statistics. Blue : standard FlashAttention ( Dao et al., 2022 ; Dao, 2024 ) ; green : our additions. (a) We compute statistics for response rows only; prompt rows can go through any production kernel (§ 2.1 ).

s_{IJ}

: scores of query tile

I

against key tile

J

. (b) One tile of the pass: the exponentials that FlashAttention already computes also feed the T1 statistics, which are finalized after the last tile; the pass also stores the log-sum-exp (LSE) that T2 and T3 reuse (§ 2.2 ). (c) The four tiers, i.e. computation paths of a statistic, ordered by the extra work they need: none (T1, fused), a few positions per row (T2, lookup), a second pass over all scores (T3, rescore), or the whole stored map (T4, dense) (§ 3 ).

Tier	Scores	Stat. ops	Memory
T1 (fused) entropy, prompt mass, effective support, max weight	none	$O(N_{s}N_{\mathcal{R}})$	$N_{s}R$
T2 (lookup) $\log p_{ii}$ , $p_{i,i-1}$ , BOS weight	$bR$	$O(bR)$	$bR$
T3 (rescore) column sums, sink rates, threshold counts	$N_{\mathcal{R}}$	$O(N_{\mathcal{R}})$	$O(R+L)$
T4 (dense) MTopDiv, stored maps	$N_{\mathcal{R}}$	$\Omega(N_{\mathcal{R}})$	$N_{\mathcal{R}}$

Table 1: Work per head beyond the pass over the response block, by tier. Scores : extra attention scores evaluated, each costing

d

multiply-adds and one exponential. Stat. ops : statistic-specific elementwise operations, plus the statistic’s own algorithm for T4. Memory : numbers written per head besides the attention output and LSE.

Figure 3: What each benchmark configuration adds to one attention layer (Q2). Configuration

\leq

Tk enables the paths up to T

k

;

\leq

T3 reads the BOS column sum from the column sums instead of the lookup. (a) Overhead of each configuration over our statistics-off Base layer at

L{=}32\mathrm{K}

(whiskers: 16K–32K), and of Base relative to SDPA on the whole sequence (diamonds). (b) Latency from 1K to 100K tokens at

d=128

r=0.3

. (c) Overhead relative to SDPA cuDNN on the whole sequence, at

r=0.3

(solid) and

r=0.1

(dashed).

Figure 4: Added latency and peak memory over the forward without features, per sample, by prompt length (Q3). Lines: binned medians. The shortest prompts are RAGTruth QA, the others PHANTOM.

		RAGTruth QA				HaluBench				TRIVIA+
Features	$N$	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B
A 7	7	.573	.567	.503	.550	.615	.639	.631	.637	.531	.524	.545	.527
L 9	9	.440	.444	.467	.433	.633	.627	.642	.665	.545	.523	.518	.576
A 7 L 9	16	.566	.532	.541	.546	.642	.643	.656	.675	.555	.560	.543	.578
H 64	64	.557	.559	.566	.557	.697	.734	.742	.742	.627	.638	.634	.670
A 7 H 64	71	.637	.581	.547	.600	.698	.729	.744	.746	.613	.638	.639	.645

Table 2: Test F1 by feature set and Qwen3 proxy size. A 7 : cheap attention features; A 64 : all-head sink values; H 64 : compact hidden states; L 9 : logit features; combinations concatenate blocks (Section 5.2 ).

N

: number of classifier inputs. Bold : best in column; underlined : second best. Bottom: the best of nine LLM judges of 4–9B parameters, Qwen3.5-9B on every dataset (Table 13 ).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Definition (per head)	Tier	Shape	Kernel computation	Derived feature and use
Log-sum-exp	$\mathrm{LSE}_{i}=\log\sum_{j\in\mathcal{J}_{i}}e^{s_{ij}}$	Base	$N_{h}\times R$	$m_{i}+\log\ell_{i}$ after the last key tile, as in FlashAttention	none; normalizer reused by T2 and T3
Entropy	$H_{i}=-\sum_{j}p_{ij}\log p_{ij}$	T1	$N_{h}\times R$	$\mathrm{LSE}_{i}-A_{i}/\ell_{i}$ with $\alpha=1$ , $g=s$	$H_{i}/\log n_{i}$ . Use: ours (A 7 )
Effective support	$N_{\mathrm{eff},2,i}=\bigl(\sum_{j}p_{ij}^{2}\bigr)^{-1}$	T1	$N_{h}\times R$	$\ell_{i}^{2}/A_{i}$ with $\alpha=2$ , $g=1$	Use: ours (A 7 )
Maximum weight	$\max_{j}p_{ij}$	T1	$N_{h}\times R$	$1/\ell_{i}$ , no accumulator	Use: ours (A 7 )
Prompt mass	$\mu^{\mathrm{P}}_{i}=\sum_{j<P}p_{ij}$	T1	$N_{h}\times R$	$A_{i}/\ell_{i}$ with $\alpha=1$ , $g=\mathbf{1}[j<P]$	lookback ratio. Use: Lookback Lens ( Chuang et al., 2024 )
Response mass	$\mu^{\mathrm{R}}_{i}=\sum_{j\geq P}p_{ij}$	T1	$N_{h}\times R$	$1-\mu^{\mathrm{P}}_{i}$ , derived; no extra state	lookback ratio. Use: Lookback Lens

Appendix

Table 3: Statistics computed by the response-stage kernels. Tier: computation path (Section 3.1 ); Base: written by the statistics-off forward. Shape: kernel output per layer for one sequence, with

N_{h}

query heads,

R

response rows,

L

keys, and

S

fixed sink keys.

A_{i}

is the T1 accumulator of Equation 3 for the given

\alpha

and

g

after the last key tile, and

\ell_{i}

the running sum of the online softmax. Derived features are defined in the text; “adapted”: restricted to response rows.

Statistic	dense	fused	ratio
Entropy $H$ (T1)	$1.6\mathrm{e}{-2}$	$3.5\mathrm{e}{-6}$	$4.5\mathrm{e}{3}$
Effective support $N_{\mathrm{eff},2}$ (T1)	$5.3\mathrm{e}{0}$	$5.9\mathrm{e}{-3}$	$9.0\mathrm{e}{2}$
Maximum weight (T1)	$4.3\mathrm{e}{-3}$	$1.5\mathrm{e}{-7}$	$2.9\mathrm{e}{4}$
Prompt mass (T1)	$2.2\mathrm{e}{-3}$	$1.3\mathrm{e}{-7}$	$1.7\mathrm{e}{4}$
LSE (Base)	$1.1\mathrm{e}{-2}$	$1.1\mathrm{e}{-6}$	$1.0\mathrm{e}{4}$
$\log p_{ii}$ (T2)	$2.0\mathrm{e}{-2}$	$1.2\mathrm{e}{-6}$	$1.7\mathrm{e}{4}$

Appendix

Table 4: Mean absolute error on response rows against an fp64 reference for materialized attention (the dense bf16 path) and our bf16 kernels, and their ratio. Each entry averages the per-head error over heads and layers and reports the worst input type (random or PHANTOM) and length (2K, 8K, 32K). Our statistics are about three to four orders of magnitude closer to fp64; they have the lower error for every statistic, input, and length, in both mean and maximum error.

	$d=64$			$d=128$			$d=256$
Overhead (%)	$r{=}.1$	$r{=}.3$	$r{=}.5$	$r{=}.1$	$r{=}.3$	$r{=}.5$	$r{=}.1$	$r{=}.3$	$r{=}.5$
$\leq$ T1 (in pass) vs. Base	9–12	22	27–28	3–7	12–15	9	1–4	6–9	2–6
$\leq$ T2 (+ lookups) vs. Base	9–12	21–22	27–30	3–6	12–20	9–10	1–4	6–10	4–7
$\leq$ T3 (+ col. sums) vs. Base	25–26	64	87	16–19	39–44	53–58	19–26	39–43	54–57
Base vs. full cuDNN	6–11	12–19	16–20	4–9	7–15	10–17	10–13	20–25	28–34
Base vs. full FA2	0–3	$-4$ – $-1$	$-9$ – $-8$	$-2$ – $+1$	$-8$ – $-5$	$-13$ – $-12$	$-5$ – $-2$	$-16$ – $-11$	$-21$ – $-20$

Appendix

Table 5: The numbers behind Figure 3 a (Q2): latency overhead (%) of one attention layer, as a range over

L\in\{16\mathrm{K},32\mathrm{K}\}

. Upper rows: each configuration of Section 4 over the statistics-off split Base, with the prompt stage on SDPA cuDNN. Lower rows: Base over SDPA on the whole sequence, with the prompt stage on the same backend, and the cost of all row statistics (

\leq

T2 ) over full SDPA cuDNN, the fastest production kernel here. H100 NVL, bf16.

Figure 5: Figure 1 at every length, for Qwen3-8B (left) and Qwen3-0.6B (right, all 16 heads of all 28 layers), with the overhead on a log scale. Dotted line: the 2K cut of Figure 1 . Same inputs, backends, and binning; batch 1, H100 NVL, bf16.

Per forward	Floor	Ours
Latency (ms)	33	108
GPU busy (ms)	13	18
GPU kernels	2,005	5,173
Host–device syncs	1	361

Appendix

Table 6: Where the time of a short forward goes: Qwen3-8B on five TRIVIA+ inputs of 90–253 tokens, all 36 layers scored, FlashAttention-2, H100 NVL. GPU busy is the summed device time of all kernels (PyTorch profiler); syncs are counted with PyTorch’s synchronization debug mode.

Figure 6: Figure 1 for peak memory: the peak of each forward above the memory allocated before it, so the model weights are excluded, with all features (solid, points) and without (dashed). Same runs, inputs of at least 2K tokens, and binning; batch 1, H100 NVL, bf16.

Dataset	Train	Val	Test	Hal.	Grp.
RAGTruth QA	3921	971	875	160	150
HaluBench	4206	898	904	409	546
TRIVIA+	2263	316	645	224	358

Appendix

Table 7: Splits used in Section 5 . Hal.: hallucinated test answers; Grp.: test groups for the bootstrap.

AUROC
	RAGTruth		HaluBench
Baseline	QA	TRIVIA+	filtered	unfiltered
Words	.702	.574	.580	.727
Characters	.720	.598	.568	.725
TF–IDF	.726	.658	.663	.842
F1

Appendix

Table 8: Text-only baselines on the test splits of Table 7 : answer length in words or characters, and TF–IDF of the answer alone. The last column uses a split of HaluBench built the same way without the length filtering (2,231 test answers, 1,075 hallucinated). The 95% group-bootstrap intervals of AUROC are at most

\pm 0.05

Statistic	Definition (per answer token $t$ )	Shape
Answer-token probability	$q_{t}(y_{t})$	$R$
Answer-token rank	$1+\|\{v\in V:q_{t}(v)>q_{t}(y_{t})\}\|$	$R$
Top- $k$ probabilities	$q_{t}^{(r)}$ for each rank $r=1,\dots,50$	$R\times 50$
Entropy	$-\sum_{v\in V}q_{t}(v)\log q_{t}(v)$	$R$
Effective vocabulary size	$\bigl(\sum_{v\in V}q_{t}(v)^{2}\bigr)^{-1}$	$R$
Zipf slope	least-squares slope of $\log q_{t}^{(r)}$ against $\log r$ , $r=1,\dots,50$	$R$

Appendix

Table 9: Logit statistics computed per answer token from the proxy’s next-token distribution. Shape: output for one sequence with

R

answer tokens.

		RAGTruth QA				HaluBench				TRIVIA+
Features	$N$	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B
A 7	7	.852	.847	.838	.856	.627	.636	.652	.660	.642	.653	.666	.645
L 9	9	.764	.765	.793	.765	.654	.646	.661	.680	.684	.661	.696	.703
A 7 L 9	16	.852	.849	.859	.859	.663	.667	.680	.693	.686	.682	.706	.707
H 64	64	.864	.855	.869	.862	.792	.824	.846	.846	.775	.786	.807	.804
A 7 H 64	71	.871	.861	.866	.872	.795	.824	.846	.847	.777	.789	.806	.793

Appendix

Table 10: Test AUROC of the detectors of Table 2 . Bold : best in column; underlined : second best (ties marked alike). Bottom: the best of nine LLM judges of 4–9B parameters, Qwen3.5-9B on every dataset (Table 13 ).

		RAGTruth QA				HaluBench				TRIVIA+
	Addition	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B
$\Delta$ F1	H ${}_{64}{\to}$ A 7 H 64	+.080	+.022	$-$ .020	+.043	+.001	$-$ .005	+.002	+.004	$-$ .014	.000	+.005	$-$ .025
	L ${}_{9}{\to}$ A 7 L 9	+.126	+.088	+.074	+.114	+.009	+.015	+.014	+.009	+.011	+.038	+.025	+.002
	H 64 L ${}_{9}{\to}$ A 7 H 64 L 9	+.007	+.005	$-$ .016	+.030	+.006	$-$ .010	+.006	+.002	+.042	+.002	+.005	+.020
$p$ , Holm-12	H ${}_{64}{\to}$ A 7 H 64	.005	.557	.605	.017	1	1	1	1	1	1	1	1
	L ${}_{9}{\to}$ A 7 L 9	$\boldsymbol{<}$ .001	.004	.002	$\boldsymbol{<}$ .001	1	1	.503	1	1	.317	1	1

Appendix

Table 11: Change in test F1 when A 7 is added, with two-sided

p

-values from 100,000 paired bootstrap resamples of test groups (seed 42; models, regularization, and thresholds fixed), Holm-corrected over the 12 dataset–proxy pairs of an addition (Holm-12). A gain is significant if

\Delta>0

and the corrected

p<0.05

; bold : significant gain. No addition gives a significant loss.

		RAGTruth QA				HaluBench				TRIVIA+
	Addition	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B	0.6B	1.7B	4B	8B
$\Delta$ F1	H ${}_{64}{\to}$ A 64 H 64	+.044	$-$ .030	$-$ .001	+.037	+.027	+.012	+.035	+.050	+.009	+.002	+.040	+.001
$p$ , Holm-12	H ${}_{64}{\to}$ A 64 H 64	.201	.324	1	.269	.198	1	.027	$\boldsymbol{<}$ .001	1	1	.189	1

Appendix

Table 12: Change in test F1 when A 64 is added to H 64 (Section 5.4 ), with two-sided

p

-values from 100,000 paired bootstrap resamples of test groups as in Table 11 , Holm-corrected over the 12 dataset–proxy pairs (Holm-12). Bold : significant gain. No change is a significant loss.

		AUROC			F1
Judge	Prompt	RAGTruth	HaluB.	TRIVIA+	RAGTruth	HaluB.	TRIVIA+
Qwen3.5-9B	Strict few-shot	.909	.859	.786	.623	.755	.655
Gemma-4-E4B	Evidence	.878	.855	.774	.617	.751	.621
Granite-4.2-8B	Strict	.843	.848	.755	.547	.753	.645
Ministral-3-8B	Strict	.823	.817	.735	.517	.721	.613
Falcon-H1-7B	Strict few-shot	.752	.834	.747	.434	.726	.624

Appendix

Table 13: LLM judges on the test splits of Table 7 with F1 at the threshold selected on validation. Top: nine instruction-tuned LLMs prompted as judges. Middle: the four Qwen3 proxy backbones prompted as judges and two specialized grounding checkers. Bottom: our proxy detectors for reference.

Fit on	Method	RAGTruth	CovidQA	PubMedQA	FinanceBench	DROP	HaluEval	Mean
		Test domain
RAUQ (no fitting)		.58	.67	.66	.52	.66	.85	.66
RAGTruth	INTRYGUE	.89	.77	.55	.48	.41	.92	.62
	Sink rate	.80	.72	.53	.49	.56	.91	.64
	LLM-Check	.59	.58	.62	.53	.53	.75	.60
	TOHA	.68	.59	.44	.46	.61	.80	.58

Appendix

Table 14: Cross-domain transfer on HaluBench (Qwen3-8B reader): test AUROC of each method fit on the source of its row group and tested on each source. Shaded: same source. Other five : fit on the five sources other than the test domain (leave-one-source-out); All six : fit on all six sources (pooled). Mean: over the five other test domains for a single fitting source, over all six for the last two groups. RAUQ fits nothing and scores the same for every fitting source. Bold : best method of the group in each column (ties marked alike). Each test domain has 138–166 examples; the 95% bootstrap intervals span

\pm 0.09

at the median (

\pm 0.03

\pm 0.12

Artifact	Use	Licence
RAGTruth	detection	MIT
HaluBench	detection	CC BY-NC 2.0
TRIVIA+	detection	CC BY-NC-ND 4.0
PHANTOM	kernels	Apache 2.0
Qwen3 0.6B–8B	proxy, judge	Apache 2.0
Qwen3.5-9B	judge	Apache 2.0

Appendix

Table 15: Licences of the artifacts we use, as stated by their releases; Falcon LLM and GLM-4 are the custom licences of these models. Detection: Section 5 ; kernels: Section 4 ; judges and grounding checkers (checker): Appendix L.5 , where the models are cited.

LLM Hallucination Detection

Large language model hallucination detection identifies unsupported, incorrect, or fabricated claims in model outputs and traces.

See all 208 papers

OpenAttention: One Percent Is Enough with Tail Proxies
Oct 22, 2026Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim +8Efficient Attention Sparse Attention
Sogang University · Samsung Electronics · Pusan National University
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
Hallucination Detection via Activations of Open-Weight Proxy Analyzers
May 8, 2026Akshita Singh, Prabesh Paudel, Siddhartha RoyHallucination in Language Models LLM Reliability
Khoury College of Computer Sciences Northeastern University

Oct 22, 2026J/K move · Enter open · S save

OpenAttention: One Percent Is Enough with Tail Proxies

Hyunwoo Yu, Jongbeom Lee, Hyunbin Kim, Sungjun Park+7

Sogang University · Samsung Electronics · Pusan National University

Efficient Attention Sparse Attention LLM Inference Acceleration Long-Context Language Modeling

Open paper AskPDF arXiv

Sparse attention reads only a small subset of the key-value cache, and how to treat the unread tail remains a key challenge. Discarding the tail biases the output, and sampling it as previous approaches do fails at reads of 5% and below, where the tail likely holds heavy hitters the selector missed. Our key observation is that the simplest possible proxy, one token that folds the entire tail into its count-aware mean, already yields lower output error than a sample at every read behind a practical selector. We further show in closed form that at low reads sampling nearly doubles the weight error of discarding while a single proxy stays below it. Building on these findings, we explore approximating the dense attention distribution with a few proxies that target the tail. We propose OpenAttention, a pipeline of clustering, ErrorGate and top-k selection followed by a count-aware mean proxy. To implement clustering and selection both efficiently and effectively, we design training-free Query-Geometry Clustering (QGC) and a Query-Geometry Selector (QGS), which compress the keys while respecting the query directions. Additionally, unlike prior methods that rank clusters by centroid alone, ErrorGate allocates the exact read by the estimated weight error of each cluster. On RULER-32K HARD with Llama-3.1-8B, OpenAttention retains 99% of dense accuracy at a 1% read. It also achieves state-of-the-art accuracy across models, context lengths and datasets. Our customized attention kernel for OpenAttention runs the decode attention stage 10

\times

faster than dense attention at a 1% read.

Abstract

LLM Hallucination Detection

Explore similar work

OpenAttention: One Percent Is Enough with Tail Proxies

Block Sparse Flash Attention

Hallucination Detection via Activations of Open-Weight Proxy Analyzers