Turn-level metrics are widely used to evaluate properties of multi-turn human-LLM conversations, from safety and sycophancy to dialogue quality. However, consecutive turns within a conversation are not statistically independent -- a fact that virtually all current evaluation pipelines fail to correct for in their statistical inference. We systematically characterize the autocorrelation structure of 66 turn-level metrics across 202 multi-turn conversations (11,639 turn pairs, 5 German-speaking users, 4 LLM platforms) and demonstrate that naive pooled analysis produces severely inflated significance estimates: 42% of associations that appear significant under standard pooled testing fail to survive cluster-robust correction. The inflation varies substantially across categories rather than scaling linearly with autocorrelation: three memoryless families (embedding velocity, directional, differential) aggregate to 14%, while the seven non-memoryless families (thermo-cycle, frame distance, lexical/structural, rolling windows, cumulative, interaction, timestamp) aggregate to 33%, with individual category rates ranging from 0% to 100% depending on per-family effect size. We present a two-stage correction framework combining Chelton (1983) effective degrees of freedom with conversation-level block bootstrap, and validate it on a pre-registered hold-out split where cluster-robust metrics replicate at 57% versus 30% for pooled-only metrics. We provide concrete design principles, a publication checklist, and open-source code for the correction pipeline. A survey of ~30 recent papers at major NLP and AI venues that compute turn-level statistics in LLM evaluations finds that only 4 address temporal dependence at all, and 26 do not correct for it.
Figures & tables
Metric Family
ρˉ1
neff
Red.
Cumulative accum.
0.928
435
27×
Rolling aggr. ( W=20 )
0.910
549
21×
Normalized cumul.
0.710
1,974
6×
Lexical/structural
0.500
3,880
3×
Frame distance
0.450
4,415
3×
Impulse (1st diff.)
0.438
4,540
3×
Table 1: Lag-1 autocorrelation ρˉ1 (averaged across 202 conversations), effective sample size neff (of n=11,639 turn pairs), and reduction factor for ten representative metric families. Figure 1 visualizes the same families graphically. Cumulative and rolling metrics lose up to 98% of nominal degrees of freedom.
Figure 1: Mean lag-1 autocorrelation across metric families. Metrics above the dashed line ( ρ≈0.5 ) are at severe risk of significance inflation.
Figure 2: Cluster-robust survival as a function of lag-1 autocorrelation. Each point represents one metric category that passed Stage 1 (pooled FDR). Point size is proportional to the number of tests. The logistic fit shows a gradual decline in survival with increasing ρ ; the ρ≈0.5 threshold marks a practical decision boundary for our two-stage protocol rather than a sharp cliff.
Community metric
Design class
ρpredicted
USR ( Mehri and Eskenazi, 2020 )
Rolling window
0.7–0.9
LlamaGuard ( Inan et al., 2023 )
Per-turn classifier
0.3–0.6
DynaEval ( Zhang et al., 2021 )
Graph-based window
0.4–0.7
Cumulative sentiment
Cumulative
> 0.9
Per-turn perplexity
Per-turn scalar
0.2–0.4
Table 2: Predicted autocorrelation ranges for community metrics, derived from design-class membership. Empirical verification on standard benchmarks is a natural next step.
Category
#Sig p
#Rob.
IR (%)
ρˉ
Embed. velocity
3
2
33
0.06
Directional
1
1
0 0
0.21
Compression (NCD)
2
0
100
0.11
Differential ( Δ )
3
3
0 0
0.20
Thermo-cycle
4
2
50
0.35
Frame distance
3
2
33
0.45
Table 3: Cluster-robust survival by metric category for the malicious label ( n=202 conversations, 11,639 turn pairs, four LLM platforms). #Sig p : pooled BH-FDR q<0.05 . #Rob. : survives Chelton + block-bootstrap at α=0.05 . IR : inflation rate. ρˉ : mean lag-1 autocorrelation. Bottom row: totals across all three labels ( malicious , typx , rewiring ).
Figure 3: Waterfall diagram of the correction pipeline on the exploratory split. At each stage, the number of surviving metric–label associations decreases. The inflation rate (IR) is the gap between FDR-corrected and cluster-robust results. The permutation baseline ( 540±34 ; B=1,000 within-chat label shuffles) confirms that the 1,408 per-chat significances exceed chance by 2.6 × ; the hold-out replication provides an independent 6.3 × confirmation (Sec 5.3 ).
Expl. status
n
Repl.
Rate
Cluster-rob. ( pfinal<.05 )
44
25
57%
Pooled-only (FDR, not rob.)
30
9
30%
Permutation (meta)
—
—
p<10−4
Table 4: Hold-out replication rates by exploratory correction status. Repl. : metrics showing same-direction significance ( p<0.05 , cluster-robust) in the hold-out. Cluster-robust metrics replicate at nearly twice the rate of pooled-only metrics.
r
ρˉ1
neff
95% CI
Raw metric
0.08
0.057
10,394
[.061,.099]
EWMA (corrected)
0.16
0.70
2,033
[.118,.202]
EWMA (naive)
0.16
—
11,639
[.142, .178]
Table 5: The EWMA paradox resolved. Smoothing improves the effect ( r : 0.08→0.16 ) but raises autocorrelation ( ρ : 0.057→0.70 ). The naive CI (italicized) is nearly 60% too narrow. CIs are Chelton-corrected analytical intervals.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Code
Category
robs
ppooled
ρˉ1
neff
pfinal
Status
V1
Embedding velocity
+0.077
6.5×10−13
0.057
10,394
6.5×10−13
robust
V2
Embedding velocity (smoothed)
+0.164
1.2×10−70
0.70
2,033
1.1×10−13
robust
D1
Differential
− 0.132
6.7×10−43
0.20
3,721
1.4×10−10
robust
C1
Compression-based
+0.098
3.9×10−26
0.105
9,426
2.2×10−21
robust
⋯ (47 robust + 34 non-robust entries; full table as CSV in the code repository) ⋯
I1
Interaction term
+0.032
0.008
0.50
1,317
0.072
weak
Appendix
Table 6: Representative entries from the full results table (81 naive-significant tests across all labels). robs : pooled point-biserial correlation. pfinal=max(pChelton,pboot) . Status: robust if pfinal<0.05 , weak otherwise.
Large language models (LLMs) excel on static benchmarks, but their performance across multi-turn conversations, which better reflect real-world usage, remains understudied. Addressing this gap is critical in high-stakes settings like healthcare, where patients and clinicians are turning to LLM chatbots to address their medical inquiries. Here, we introduce the "stick-or-switch" (SoS) framework, which partitions a question-answer space into multiple sequential presentations to model two safety-centric behaviors: conviction (i.e., sticking to a correct answer selection or abstention against incorrect suggestions) and flexibility (i.e., switching to a correct suggestion when it is introduced). Evaluating 17 LLMs across three clinical benchmarks, we observe a pervasive conversation tax, where partitioning an answer-space into sequential presentations reduces end-to-end accuracy and abstention against incorrect suggestions by an average of up to 30%, reaching 65% in certain models. We also observe blind switching, where models transition an initial abstention to incorrect and correct suggestions at near-identical rates reaching 50%. Finally, we show that increasing model scale mitigates some of these conversational inefficacies while exacerbating others, such as a higher propensity to adopt an incorrect suggestion from an initial abstention. Together our findings demonstrate that the general proficiency captured by static benchmarks do not translate over multi-turn dialogues.
Kevin H. Guo, Chao Yan, Avinash Baidya +5
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.
Benchmark suites assess model capability on controlled tasks; large-scale conversation corpora capture naturalistic use without user feedback; and in-interface feedback mechanisms record satisfaction without task purpose. Together, they leave a critical gap in LLM evaluation: no existing infrastructure routinely links interaction trajectories to user-defined outcomes. We introduce MonitrLLM, open-source infrastructure for community-centered LLM evaluations that links full conversation transcripts to user-reported task intent and outcome assessments, treating all three as primary evaluative signals rather than optional metadata. To demonstrate the value of this approach, we conducted a two-week feasibility pilot with 26 college students using ChatGPT, collecting 206 evaluation reports with full conversation transcripts. The findings from our pilot demonstrate the value of connecting conversation trajectories with user-reported outcomes. For instance, despite reporting high average satisfaction (4.19/5) with their LLM interactions, participants also experience a substantial 23.1% failure rate on their goal tasks. We also find that multi-turn conversations are reported as failing at 2.5 times the rate of single-turn exchanges, a pattern that reframes extended interaction as a signal of difficulty rather than engagement. We conclude by discussing the value of incorporating direct user feedback with observational data for robust LLM evaluations, and the possibilities for infrastructure that enables this goal.
Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian +1
Center for Tech Responsibility, Brown University · University of Pennsylvania