DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents

Sergio Burdisso; Esaú Villatoro-Tello; Dairazalia Sánchez-Cortés; Yanis Labrak; Jordi Luque; Milos Cernak; Petr Motlicek

cs.CLOct 15, 2026

DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents

Authors: Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés, Yanis Labrak, Jordi Luque, Milos Cernak, Petr Motlicek

Organizations: Idiap Research Institute, Martigny, Switzerland · Telefónica Innovación Digital, Barcelona, Spain · Logitech, Lausanne, Switzerland · Brno University of Technology, Brno, Czech Republic

Abstract

Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a different failure mode has been overlooked: procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process. We introduce DialFork, the first benchmark to measure this failure mode on raw dialogue text. DialFork is built by forking conformant dialogues at controlled turn positions and injecting one of four hallucination types, yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction. We define three evaluation tasks that characterize procedural deviation at three granularities: how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails). For the first two, we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content. At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders, showing that semantic similarity is no substitute for procedural similarity. We release DialFork under open licenses at https://github.com/idiap/dialfork, hoping the community will measure conversational process alongside content.

Figures & tables

Figure 1: Construction of DialFork by controlled forking: the simulation of a conformant dialogue $D^{+}$ is re-run with the same seed, reproducing its prefix, and a hallucination instruction is injected just before the fork turn. Earlier forks deviate more, inducing a ground-truth ordering by procedural (dialog-flow) similarity.

Type	$(\mathcal{S},\mathcal{A})$	$\delta$	$\mathcal{O}$	What is broken	Example (simplified): first two turns after the fork
Shared opening before the fork					User: I’d like to book a viewing. System: Your name? User: Mark.
Conformant	✓	✓	✓	— (original)	System: Which apartment do you want to view? User: Center Apts. …
WIC	$\boldsymbol{\times}$	$\boldsymbol{\times}$	✓	Workflow identity	System: Let’s search apartments! budget? User: No, just a viewing! …
SSPC	✓	$\boldsymbol{\times}$	✓	Step selection	System: Your viewing is booked! User: What?! Which one?! …
GR	✓	✓	$\boldsymbol{\times}$	Task goal	System: Want to try co-living instead? User: No, no, I already know …
CSC	✓	✓	$\sim$	Goal conflict	System: Just to confirm, your name is Mark? User: Yes, Mark. …

Table 1: Procedural hallucination types and the components of

\mathcal{W}=(\mathcal{S},\mathcal{A},\delta,\mathcal{O})

they corrupt (✓: preserved;

\times

: replaced;

\sim

: partially overridden). Simplified example (a real one is in Appendix G ): after the shared opening, each row shows only the first two turns from the fork turn

t_{k}

on; every version then continues (…) as a full dialogue. Gray: intended workflow; blue: corrupted.

Figure 2: Per-task composition of DialFork . (a) Number of dialogues per STAR domain (inner ring) and task (outer ring; abbreviations as in (b)). Since every conformant dialogue yields 16 forks, the proportions are the same for conformant and forked dialogues. (b) Mean dialogue length per task, conformant vs. forked, grouped by domain in the same order and colors as (a).

Figure 3: Human validation of DialFork (Appendix N ). (a) Mean blind conformance rating per fork position

k

for WIC , SSPC , and GR forks (bars, with 95% bootstrap CIs) and for all four types (markers), and for conformant dialogues. (b) Distance, in assistant turns, between the annotated onset of the injected behavior and the fork turn

t_{k}

Figure 4: DialFork statistics. (a) Dialogue length per split (box: interquartile range; whiskers: 5th–95th percentiles; line: median; dot: mean). (b) Mean number of turns per fork position

k

, split into turns generated before the fork, under the intended workflow, and after it, under the corrupted policy (averaged over the four hallucination types).

Method	WIC	SSPC	GR	CSC	Avg.
Random	0.000	0.000	0.000	0.000	0.000
Turn-count diff.	$-$ 0.076	$-$ 0.340	0.278	$-$ 0.208	$-$ 0.087
Word-count diff.	0.222	0.208	0.583	$-$ 0.208	0.201
FBD	0.972	0.792	0.986	0.903	0.913
PRD	0.931	0.764	0.986	0.833	0.878
FlowKL	0.847	0.903	0.944	0.833	0.882

Table 2: Task 1 (Set-to-Set Distance): Kendall’s

\tau

between predicted and ground-truth fork-position rankings, averaged across the 24 tasks. Top block: controls; bold : best per column; shading: darker is better. All three distance metrics are significantly above chance (

p<0.001

); differences among them are not significant (

p\geq 0.43

; Appendix J ).

Encoder	Method	WIC	SSPC	GR	CSC	Avg.
Random		0.000	0.000	0.000	0.000	0.000
Turn-count diff.		0.025	0.251	0.226	0.002	0.126
Word-count diff.		0.220	0.242	0.390	0.040	0.223
Shared prefix		0.688	0.668	0.712	0.662	0.682
LaBSE	Mean	0.816	0.745	0.895	0.579	0.759
	Full	0.755	0.705	0.813	0.428	0.675

Table 3: Task 2 (Dialog-to-Dialog Distance): Kendall’s

\tau

between predicted and ground-truth fork rankings, averaged across all conformant dialogues. Top block: controls. Bold : best per encoder; bold+underline : best overall; shading: darker is better. Mean/Full = mean-turn/full-dialog embedding. FlowScore ’s improvements over every other method are statistically significant for every encoder (one-sided Wilcoxon, Holm-corrected,

p<0.001

; Table 7 ).

Baselines
	Hallucination MAE $\downarrow$				Hallucination		Conformance
Method	CSC	GR	SSPC	WIC	Avg. MAE $\downarrow$	Cov. $\uparrow$	MAE $\downarrow$	Cov. $\downarrow$	Score $\uparrow$
Random	6.31	7.92	5.74	6.44	6.60	100.00	7.50	100.00	0.94
First turn	5.64	5.64	5.64	5.64	5.64	100.00	14.12	100.00	0.00
Last turn	12.01	15.57	10.26	12.22	12.52	100.00	1.00	100.00	0.00
Streaming

Table 4: Task 3 (Turn Prediction): results on the Task 3 test set; MAEs are in turns. Bold : best within a regime; bold+underline : best overall; shading: darker is better.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dialogues	#Dialogs	%Total	#Turns	#Words
	Coverage		Dialog Length
Conformant	2,134	5.88%	15.52	186.94
WIC forked	8,536	23.53%	20.02	258.49
SSPC forked	8,536	23.53%	19.31	187.47
GR forked	8,536	23.53%	24.79	342.23
CSC forked	8,536	23.53%	20.58	247.64

Appendix

Table 5: Dataset composition: conformant dialogues and forks of each hallucination type, with their number, share of the dataset, and average length in turns (#Turns) and words (#Words).

Task	#	MT	MW	#	MT	MW	#	MT	MW	Train	Test
	Conformant			Forks			Total			Task 3 split
apartment_schedule (ase)	206	18.65	235.68	3,296	22.36	294.07	3,502	22.15	290.64	103	1,751
apartment_search (as)	30	11.87	271.57	480	17.07	300.99	510	16.77	299.26	15	255
bank_balance (bb)	46	13.52	125.39	736	18.78	214.28	782	18.47	209.05	23	391
bank_fraud_report (bfr)	135	13.73	166.71	2,160	19.90	232.26	2,295	19.54	228.40	67	1,156
doctor_followup (df)	42	9.38	110.36	672	13.85	173.54	714	13.59	169.83	21	357

Appendix

Table 6: Per-task detailed statistics. Task labels are shown as task_name (abbr). In each block, # = number of dialogues, MT = mean turns, MW = mean words. Conformant = non-hallucinated; Forks = aggregated forked variants; Total combines both. Task 3 split: number of dialogues in the Task 3 train and test sets (§ A.1 ).

Figure 5: Flow-chart representation of the schema graph for the doctor_schedule task from STAR ( Mosig et al., 2020 ) . The corresponding JSON specification is given in Listing 1 .

Figure 6: Directed flow graph induced from DialFork ’s conformant dialogues $\mathcal{D}^{+}$ of the doctor_schedule task. Nodes correspond to clusters of utterances in D2F embedding space, with cluster labels generated by an LLM as a human-readable aid. Edge widths are proportional to their transition probability. User nodes are colored blue and system nodes white.

Baseline	Model	$N$	Mean $\Delta$	95% CI	$W$	$p_{W}$ (Holm)	$t$	$p_{t}$ (Holm)
Mean-turn emb.	LaBSE	8536	0.089	[0.083, 0.094]	3,567,916	<0.001	32.94	<0.001
Mean-turn emb.	Qwen3-Emb.	8536	0.166	[0.159, 0.173]	7,337,009	<0.001	46.89	<0.001
Mean-turn emb.	SBERT	8536	0.120	[0.114, 0.126]	4,799,006	<0.001	41.42	<0.001
Mean-turn emb.	BGE-M3	8536	0.103	[0.098, 0.109]	4,247,414	<0.001	36.03	<0.001
Mean-turn emb.	D2F	8536	0.079	[0.074, 0.084]	3,277,998	<0.001	30.18	<0.001
Mean-turn emb.	DSE	8536	0.098	[0.092, 0.104]	3,910,352	<0.001	33.80	<0.001

Appendix

Table 7: Task 2 significance of FlowScore vs. each baseline, per encoder (both tests one-sided, FlowScore

>

baseline). Columns:

N

, number of paired units (conformant dialogue

\times

hallucination type); Mean

\Delta

, mean per-unit difference in Kendall

\tau

( FlowScore

-

baseline); 95% CI, paired-bootstrap confidence interval of Mean

\Delta

(10,000 resamples);

W

, Wilcoxon signed-rank statistic;

p_{W}

, its p-value;

t

, paired

t

-test statistic;

p_{t}

, its p-value. P-values are Holm-Bonferroni corrected across encoders within each test. Wilcoxon is the primary test; the

t

-test is shown for comparison. Bold marks significance (

\alpha=0.05

Embedding	Method	CSC	GR	SSPC	WIC	Avg. $\downarrow$	Cov. $\uparrow$	MAE $\downarrow$	Cov. $\downarrow$	Score $\uparrow$
		Hallucination MAE $\downarrow$				Hallucination		Conformance
LaBSE	Surprisal Spike	4.21	1.56	1.30	1.68	2.19	93.24	6.33	12.02	73.69
	w/ NC	4.13	1.61	1.33	1.71	2.20	93.56	6.58	15.47	72.45
	Improbable Transition	4.13	0.93	1.25	1.63	1.98	93.86	6.52	12.49	73.60
	w/ NC	3.94	0.92	1.29	1.60	1.94	94.80	6.55	16.31	73.23
D2F	Surprisal Spike	4.82	1.99	1.70	2.42	2.73	90.92	6.72	10.81	70.91

Appendix

Table 9: Streaming results on the Task 3 test set of the 3-gram LM detectors when using each embedding model to cluster turns into nodes. SS = Surprisal Spike; IT = Improbable Transition; indented w/ NC rows add node-confidence weighting to the method above. Best values within each embedding model are bold; best overall (non-baseline) are bold+underline; shading: darker is better.

RQ	Question	Validates	Measure	Statistic	Result
RQ1	Does fork position track perceived procedural deviation?	Fork order $k$ (Tasks 1–2)	Step 1: 0–100 conformance slider, before any cue	Kendall $\tau_{b}$ and probability of superiority within each (annotator, task); conformant dialogues as anchor	Yes : strong rank correlation with $k$ ( $\tau_{b}{=}0.64$ ) for all the types that break the workflow
RQ2	Does the injected behavior appear?	Type and variant labels	Step 2: yes / partly / no, against the true instruction or a foil	Confirmation rate per type (Wilson CIs) vs. foil confirmation rate	Yes : 91% confirmed (87–96% per type), vs. 8% for foils on conformant dialogues
RQ3	Does it start at the intended turn?	Fork turn $t_{k}$ (Task 3)	Step 2: first assistant turn where it starts, before the fork is shown	$\|$ onset $\,-\,t_{k}\|$ distribution; share exact and within $\pm 1$ turn	Yes : 67% exact, 77% within one turn
RQ4	Is the pre-fork part conformant?	Seeded replay of the prefix	Step 3: yes / mostly / no, for forks with $t_{k}>1$	Conformance rate (Wilson CI)	Yes : 95% followed or mostly followed

Appendix

Table 10: Research questions of the human study, the label or claim each one validates, how it is measured (step of the annotation item, § N.4 ), the statistic reported, and the result (§ N.7 ).

Panel of Figure 7 — Figure 7: The annotation tool, shown on a demonstration instance with placeholder annotators ( doctor_schedule task). The workflow steps stay visible on the left of every conversation; only the assistant turns are judged.

Figure 8: Human study results, one panel per research question. (a) RQ1: mean step-1 conformance rating per fork position $k$ over WIC , SSPC , and GR forks (bars) and over all four types (markers; CSC keeps every component of the workflow, see text), and for conformant dialogues, with 95% bootstrap CIs. (b) RQ2: share of forked dialogues in which the true injected instruction was confirmed (yes or partly), per hallucination type, vs. the share of foil instructions confirmed, with 95% Wilson CIs. (c) RQ3: absolute difference between the annotated onset and the fork turn $t_{k}$ , over confirmed true instructions. (d) RQ4: answers to the prefix question per fork position ( $k{=}1$ forks have no pre-fork part).

	People	LLM judge
	study	study	all dialogues
Rating, $k{=}1/2/3/4$	14/19/41/58	15/16/28/55	11/15/27/53
Rating, conformant	94	98	97
Instruction confirmed	91%	85%	84%
Foil confirmed	8%	0%	2.0%
Onset at $t_{k}$ / within $\pm 1$	67% / 77%	72% / 82%	71% / 84%

Appendix

Table 11: The calibrated LLM judge ( Qwen3.6-35B ) on the study’s conversations and on the whole corpus, next to the annotators. Ratings are means over WIC , SSPC and GR forks (§ N.7 ); confirmation and onset are over forks shown the true instruction; foils are those shown on conformant dialogues.

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
May 1, 2026Sailesh Panda, Pritam Kadasi, Abhishek Upperwal +1LLM Evaluation Instruction Following
Indian Institute of Technology Gandhinagar · Soket AI
Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
May 30, 2026Yufeng WangAI Agent Reliability LLM Agent Evaluation
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA

May 1, 2026J/K move · Enter open · S save

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh

Indian Institute of Technology Gandhinagar · Soket AI

LLM Evaluation Instruction Following LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful long-horizon procedural execution.

Abstract

Hallucination in Language Models

Explore similar work

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench