The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious

Ferdinand M. Schessl

cs.CLApr 15, 2026

The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious

Authors: Ferdinand M. Schessl

Organizations: Independent Researcher

Abstract

Turn-level metrics are widely used to evaluate properties of multi-turn human-LLM conversations, from safety and sycophancy to dialogue quality. However, consecutive turns within a conversation are not statistically independent -- a fact that virtually all current evaluation pipelines fail to correct for in their statistical inference. We systematically characterize the autocorrelation structure of 66 turn-level metrics across 202 multi-turn conversations (11,639 turn pairs, 5 German-speaking users, 4 LLM platforms) and demonstrate that naive pooled analysis produces severely inflated significance estimates: 42% of associations that appear significant under standard pooled testing fail to survive cluster-robust correction. The inflation varies substantially across categories rather than scaling linearly with autocorrelation: three memoryless families (embedding velocity, directional, differential) aggregate to 14%, while the seven non-memoryless families (thermo-cycle, frame distance, lexical/structural, rolling windows, cumulative, interaction, timestamp) aggregate to 33%, with individual category rates ranging from 0% to 100% depending on per-family effect size. We present a two-stage correction framework combining Chelton (1983) effective degrees of freedom with conversation-level block bootstrap, and validate it on a pre-registered hold-out split where cluster-robust metrics replicate at 57% versus 30% for pooled-only metrics. We provide concrete design principles, a publication checklist, and open-source code for the correction pipeline. A survey of ~30 recent papers at major NLP and AI venues that compute turn-level statistics in LLM evaluations finds that only 4 address temporal dependence at all, and 26 do not correct for it.

Figures & tables

Metric Family	$\bar{\rho}_{1}$	$n_{\mathrm{eff}}$	Red.
Cumulative accum.	0.928	435	$27\times$
Rolling aggr. ( $W{=}20$ )	0.910	549	$21\times$
Normalized cumul.	0.710	1,974	$6\times$
Lexical/structural	0.500	3,880	$3\times$
Frame distance	0.450	4,415	$3\times$
Impulse (1st diff.)	0.438	4,540	$3\times$

Table 1: Lag-1 autocorrelation

\bar{\rho}_{1}

(averaged across 202 conversations), effective sample size

n_{\mathrm{eff}}

(of

n{=}11{,}639

turn pairs), and reduction factor for ten representative metric families. Figure 1 visualizes the same families graphically. Cumulative and rolling metrics lose up to 98% of nominal degrees of freedom.

Figure 1: Mean lag-1 autocorrelation across metric families. Metrics above the dashed line (

\rho\approx 0.5

) are at severe risk of significance inflation.

Figure 2: Cluster-robust survival as a function of lag-1 autocorrelation. Each point represents one metric category that passed Stage 1 (pooled FDR). Point size is proportional to the number of tests. The logistic fit shows a gradual decline in survival with increasing

\rho

; the

\rho\approx 0.5

threshold marks a practical decision boundary for our two-stage protocol rather than a sharp cliff.

Community metric	Design class	$\boldsymbol{\rho}_{\textbf{predicted}}$
USR ( Mehri and Eskenazi, 2020 )	Rolling window	0.7–0.9
LlamaGuard ( Inan et al., 2023 )	Per-turn classifier	0.3–0.6
DynaEval ( Zhang et al., 2021 )	Graph-based window	0.4–0.7
Cumulative sentiment	Cumulative	$>$ 0.9
Per-turn perplexity	Per-turn scalar	0.2–0.4

Table 2: Predicted autocorrelation ranges for community metrics, derived from design-class membership. Empirical verification on standard benchmarks is a natural next step.

Category	#Sig p	#Rob.	IR (%)	$\boldsymbol{\bar{\rho}}$
Embed. velocity	3	2	33	0.06
Directional	1	1	0 0	0.21
Compression (NCD)	2	0	100	0.11
Differential ( $\Delta$ )	3	3	0 0	0.20
Thermo-cycle	4	2	50	0.35
Frame distance	3	2	33	0.45

Table 3: Cluster-robust survival by metric category for the malicious label (

n{=}202

conversations, 11,639 turn pairs, four LLM platforms). #Sig p : pooled BH-FDR

q{<}0.05

. #Rob. : survives Chelton + block-bootstrap at

\alpha{=}0.05

. IR : inflation rate.

\bar{\rho}

: mean lag-1 autocorrelation. Bottom row: totals across all three labels ( malicious , typx , rewiring ).

Figure 3: Waterfall diagram of the correction pipeline on the exploratory split. At each stage, the number of surviving metric–label associations decreases. The inflation rate (IR) is the gap between FDR-corrected and cluster-robust results. The permutation baseline (

540\pm 34

;

B{=}1{,}000

within-chat label shuffles) confirms that the 1,408 per-chat significances exceed chance by 2.6

\times

; the hold-out replication provides an independent 6.3

\times

confirmation (Sec 5.3 ).

Expl. status	$\boldsymbol{n}$	Repl.	Rate
Cluster-rob. ( $p_{\text{final}}{<}.05$ )	44	25	57%
Pooled-only (FDR, not rob.)	30	9	30%
Permutation (meta)	—	—	$p{<}10^{-4}$

Table 4: Hold-out replication rates by exploratory correction status. Repl. : metrics showing same-direction significance (

p<0.05

, cluster-robust) in the hold-out. Cluster-robust metrics replicate at nearly twice the rate of pooled-only metrics.

	$r$	$\bar{\rho}_{1}$	$n_{\mathrm{eff}}$	95% CI
Raw metric	0.08	0.057	10,394	$[.061,\;.099]$
EWMA (corrected)	0.16	0.70	2,033	$[.118,\;.202]$
EWMA (naive)	0.16	—	11,639	[.142, .178]

Table 5: The EWMA paradox resolved. Smoothing improves the effect (

r

0.08\to 0.16

) but raises autocorrelation (

\rho

0.057\to 0.70

). The naive CI (italicized) is nearly 60% too narrow. CIs are Chelton-corrected analytical intervals.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Code	Category	$r_{\text{obs}}$	$p_{\text{pooled}}$	$\bar{\rho}_{1}$	$n_{\text{eff}}$	$p_{\text{final}}$	Status
V1	Embedding velocity	+0.077	$6.5\times 10^{-13}$	0.057	10,394	$6.5\times 10^{-13}$	robust
V2	Embedding velocity (smoothed)	+0.164	$1.2\times 10^{-70}$	0.70	2,033	$1.1\times 10^{-13}$	robust
D1	Differential	$-$ 0.132	$6.7\times 10^{-43}$	0.20	3,721	$1.4\times 10^{-10}$	robust
C1	Compression-based	+0.098	$3.9\times 10^{-26}$	0.105	9,426	$2.2\times 10^{-21}$	robust
$\cdots$ (47 robust + 34 non-robust entries; full table as CSV in the code repository) $\cdots$
I1	Interaction term	+0.032	0.008	0.50	1,317	0.072	weak

Appendix

Table 6: Representative entries from the full results table (81 naive-significant tests across all labels).

r_{\text{obs}}

: pooled point-biserial correlation.

p_{\text{final}}=\max(p_{\text{Chelton}},p_{\text{boot}})

. Status: robust if

p_{\text{final}}<0.05

, weak otherwise.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability
Mar 12, 2026Kevin H. Guo, Chao Yan, Avinash Baidya +5LLM Reliability Multi-Turn Dialogue Evaluation
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models
Aug 3, 2026Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian +1LLM Evaluation Human-in-the-Loop Evaluation
Center for Tech Responsibility, Brown University · University of Pennsylvania

Mar 12, 2026J/K move · Enter open · S save

Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability

Kevin H. Guo, Chao Yan, Avinash Baidya, Katherine Brown+4

Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research

LLM Reliability Multi-Turn Dialogue Evaluation Clinical Language Model Evaluation Knowledge Conflicts in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) excel on static benchmarks, but their performance across multi-turn conversations, which better reflect real-world usage, remains understudied. Addressing this gap is critical in high-stakes settings like healthcare, where patients and clinicians are turning to LLM chatbots to address their medical inquiries. Here, we introduce the "stick-or-switch" (SoS) framework, which partitions a question-answer space into multiple sequential presentations to model two safety-centric behaviors: conviction (i.e., sticking to a correct answer selection or abstention against incorrect suggestions) and flexibility (i.e., switching to a correct suggestion when it is introduced). Evaluating 17 LLMs across three clinical benchmarks, we observe a pervasive conversation tax, where partitioning an answer-space into sequential presentations reduces end-to-end accuracy and abstention against incorrect suggestions by an average of up to 30%, reaching 65% in certain models. We also observe blind switching, where models transition an initial abstention to incorrect and correct suggestions at near-identical rates reaching 50%. Finally, we show that increasing model scale mitigates some of these conversational inefficacies while exacerbating others, such as a higher propensity to adopt an incorrect suggestion from an initial abstention. Together our findings demonstrate that the general proficiency captured by static benchmarks do not translate over multi-turn dialogues.

Abstract

LLM Evaluation

Explore similar work

Stop Listening to Me! How Multi-turn Conversations Can Degrade LLM Reliability

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models