Organizations: Idiap Research Institute, Martigny, Switzerland · Telefónica Innovación Digital, Barcelona, Spain · Logitech, Lausanne, Switzerland · Brno University of Technology, Brno, Czech Republic
Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a different failure mode has been overlooked: procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process. We introduce DialFork, the first benchmark to measure this failure mode on raw dialogue text. DialFork is built by forking conformant dialogues at controlled turn positions and injecting one of four hallucination types, yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction. We define three evaluation tasks that characterize procedural deviation at three granularities: how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails). For the first two, we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content. At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders, showing that semantic similarity is no substitute for procedural similarity. We release DialFork under open licenses at https://github.com/idiap/dialfork, hoping the community will measure conversational process alongside content.
Figures & tables
Figure 1: Construction of DialFork by controlled forking: the simulation of a conformant dialogue D+ is re-run with the same seed, reproducing its prefix, and a hallucination instruction is injected just before the fork turn. Earlier forks deviate more, inducing a ground-truth ordering by procedural (dialog-flow) similarity.
Type
(S,A)
δ
O
What is broken
Example (simplified): first two turns after the fork
Shared opening before the fork
User: I’d like to book a viewing. System: Your name? User: Mark.
Conformant
✓
✓
✓
— (original)
System: Which apartment do you want to view? User: Center Apts. …
WIC
×
×
✓
Workflow identity
System: Let’s search apartments! budget? User: No, just a viewing! …
SSPC
✓
×
✓
Step selection
System: Your viewing is booked! User: What?! Which one?! …
GR
✓
✓
×
Task goal
System: Want to try co-living instead? User: No, no, I already know …
CSC
✓
✓
∼
Goal conflict
System: Just to confirm, your name is Mark? User: Yes, Mark. …
Table 1: Procedural hallucination types and the components of W=(S,A,δ,O) they corrupt (✓: preserved; × : replaced; ∼ : partially overridden). Simplified example (a real one is in Appendix G ): after the shared opening, each row shows only the first two turns from the fork turn tk on; every version then continues (…) as a full dialogue. Gray: intended workflow; blue: corrupted.
Figure 2: Per-task composition of DialFork . (a) Number of dialogues per STAR domain (inner ring) and task (outer ring; abbreviations as in (b)). Since every conformant dialogue yields 16 forks, the proportions are the same for conformant and forked dialogues. (b) Mean dialogue length per task, conformant vs. forked, grouped by domain in the same order and colors as (a).
Figure 3: Human validation of DialFork (Appendix N ). (a) Mean blind conformance rating per fork position k for WIC , SSPC , and GR forks (bars, with 95% bootstrap CIs) and for all four types (markers), and for conformant dialogues. (b) Distance, in assistant turns, between the annotated onset of the injected behavior and the fork turn tk .
Figure 4: DialFork statistics. (a) Dialogue length per split (box: interquartile range; whiskers: 5th–95th percentiles; line: median; dot: mean). (b) Mean number of turns per fork position k , split into turns generated before the fork, under the intended workflow, and after it, under the corrupted policy (averaged over the four hallucination types).
Method
WIC
SSPC
GR
CSC
Avg.
Random
0.000
0.000
0.000
0.000
0.000
Turn-count diff.
− 0.076
− 0.340
0.278
− 0.208
− 0.087
Word-count diff.
0.222
0.208
0.583
− 0.208
0.201
FBD
0.972
0.792
0.986
0.903
0.913
PRD
0.931
0.764
0.986
0.833
0.878
FlowKL
0.847
0.903
0.944
0.833
0.882
Table 2: Task 1 (Set-to-Set Distance): Kendall’s τ between predicted and ground-truth fork-position rankings, averaged across the 24 tasks. Top block: controls; bold : best per column; shading: darker is better. All three distance metrics are significantly above chance ( p<0.001 ); differences among them are not significant ( p≥0.43 ; Appendix J ).
Encoder
Method
WIC
SSPC
GR
CSC
Avg.
Random
0.000
0.000
0.000
0.000
0.000
Turn-count diff.
0.025
0.251
0.226
0.002
0.126
Word-count diff.
0.220
0.242
0.390
0.040
0.223
Shared prefix
0.688
0.668
0.712
0.662
0.682
LaBSE
Mean
0.816
0.745
0.895
0.579
0.759
Full
0.755
0.705
0.813
0.428
0.675
Table 3: Task 2 (Dialog-to-Dialog Distance): Kendall’s τ between predicted and ground-truth fork rankings, averaged across all conformant dialogues. Top block: controls. Bold : best per encoder; bold+underline : best overall; shading: darker is better. Mean/Full = mean-turn/full-dialog embedding. FlowScore ’s improvements over every other method are statistically significant for every encoder (one-sided Wilcoxon, Holm-corrected, p<0.001 ; Table 7 ).
Hallucination MAE ↓
Hallucination
Conformance
Method
CSC
GR
SSPC
WIC
Avg. MAE ↓
Cov. ↑
MAE ↓
Cov. ↓
Score ↑
Baselines
Random
6.31
7.92
5.74
6.44
6.60
100.00
7.50
100.00
0.94
First turn
5.64
5.64
5.64
5.64
5.64
100.00
14.12
100.00
0.00
Last turn
12.01
15.57
10.26
12.22
12.52
100.00
1.00
100.00
0.00
Streaming
Table 4: Task 3 (Turn Prediction): results on the Task 3 test set; MAEs are in turns. Bold : best within a regime; bold+underline : best overall; shading: darker is better.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Coverage
Dialog Length
Dialogues
#Dialogs
%Total
#Turns
#Words
Conformant
2,134
5.88%
15.52
186.94
WIC forked
8,536
23.53%
20.02
258.49
SSPC forked
8,536
23.53%
19.31
187.47
GR forked
8,536
23.53%
24.79
342.23
CSC forked
8,536
23.53%
20.58
247.64
Appendix
Table 5: Dataset composition: conformant dialogues and forks of each hallucination type, with their number, share of the dataset, and average length in turns (#Turns) and words (#Words).
Conformant
Forks
Total
Task 3 split
Task
#
MT
MW
#
MT
MW
#
MT
MW
Train
Test
apartment_schedule (ase)
206
18.65
235.68
3,296
22.36
294.07
3,502
22.15
290.64
103
1,751
apartment_search (as)
30
11.87
271.57
480
17.07
300.99
510
16.77
299.26
15
255
bank_balance (bb)
46
13.52
125.39
736
18.78
214.28
782
18.47
209.05
23
391
bank_fraud_report (bfr)
135
13.73
166.71
2,160
19.90
232.26
2,295
19.54
228.40
67
1,156
doctor_followup (df)
42
9.38
110.36
672
13.85
173.54
714
13.59
169.83
21
357
Appendix
Table 6: Per-task detailed statistics. Task labels are shown as task_name (abbr). In each block, # = number of dialogues, MT = mean turns, MW = mean words. Conformant = non-hallucinated; Forks = aggregated forked variants; Total combines both. Task 3 split: number of dialogues in the Task 3 train and test sets (§ A.1 ).
Figure 5: Flow-chart representation of the schema graph for the doctor_schedule task from STAR ( Mosig et al., 2020 ) . The corresponding JSON specification is given in Listing 1 .
Figure 6: Directed flow graph induced from DialFork ’s conformant dialogues D+ of the doctor_schedule task. Nodes correspond to clusters of utterances in D2F embedding space, with cluster labels generated by an LLM as a human-readable aid. Edge widths are proportional to their transition probability. User nodes are colored blue and system nodes white.
Baseline
Model
N
Mean Δ
95% CI
W
pW (Holm)
t
pt (Holm)
Mean-turn emb.
LaBSE
8536
0.089
[0.083, 0.094]
3,567,916
<0.001
32.94
<0.001
Mean-turn emb.
Qwen3-Emb.
8536
0.166
[0.159, 0.173]
7,337,009
<0.001
46.89
<0.001
Mean-turn emb.
SBERT
8536
0.120
[0.114, 0.126]
4,799,006
<0.001
41.42
<0.001
Mean-turn emb.
BGE-M3
8536
0.103
[0.098, 0.109]
4,247,414
<0.001
36.03
<0.001
Mean-turn emb.
D2F
8536
0.079
[0.074, 0.084]
3,277,998
<0.001
30.18
<0.001
Mean-turn emb.
DSE
8536
0.098
[0.092, 0.104]
3,910,352
<0.001
33.80
<0.001
Appendix
Table 7: Task 2 significance of FlowScore vs. each baseline, per encoder (both tests one-sided, FlowScore > baseline). Columns: N , number of paired units (conformant dialogue × hallucination type); Mean Δ , mean per-unit difference in Kendall τ ( FlowScore − baseline); 95% CI, paired-bootstrap confidence interval of Mean Δ (10,000 resamples); W , Wilcoxon signed-rank statistic; pW , its p-value; t , paired t -test statistic; pt , its p-value. P-values are Holm-Bonferroni corrected across encoders within each test. Wilcoxon is the primary test; the t -test is shown for comparison. Bold marks significance ( α=0.05 ).
Hallucination MAE ↓
Hallucination
Conformance
Embedding
Method
CSC
GR
SSPC
WIC
Avg. ↓
Cov. ↑
MAE ↓
Cov. ↓
Score ↑
LaBSE
Surprisal Spike
4.21
1.56
1.30
1.68
2.19
93.24
6.33
12.02
73.69
w/ NC
4.13
1.61
1.33
1.71
2.20
93.56
6.58
15.47
72.45
Improbable Transition
4.13
0.93
1.25
1.63
1.98
93.86
6.52
12.49
73.60
w/ NC
3.94
0.92
1.29
1.60
1.94
94.80
6.55
16.31
73.23
D2F
Surprisal Spike
4.82
1.99
1.70
2.42
2.73
90.92
6.72
10.81
70.91
Appendix
Table 9: Streaming results on the Task 3 test set of the 3-gram LM detectors when using each embedding model to cluster turns into nodes. SS = Surprisal Spike; IT = Improbable Transition; indented w/ NC rows add node-confidence weighting to the method above. Best values within each embedding model are bold; best overall (non-baseline) are bold+underline; shading: darker is better.
RQ
Question
Validates
Measure
Statistic
Result
RQ1
Does fork position track perceived procedural deviation?
Fork order k (Tasks 1–2)
Step 1: 0–100 conformance slider, before any cue
Kendall τb and probability of superiority within each (annotator, task); conformant dialogues as anchor
Yes : strong rank correlation with k ( τb=0.64 ) for all the types that break the workflow
RQ2
Does the injected behavior appear?
Type and variant labels
Step 2: yes / partly / no, against the true instruction or a foil
Confirmation rate per type (Wilson CIs) vs. foil confirmation rate
Yes : 91% confirmed (87–96% per type), vs. 8% for foils on conformant dialogues
RQ3
Does it start at the intended turn?
Fork turn tk (Task 3)
Step 2: first assistant turn where it starts, before the fork is shown
∣ onset −tk∣ distribution; share exact and within ±1 turn
Yes : 67% exact, 77% within one turn
RQ4
Is the pre-fork part conformant?
Seeded replay of the prefix
Step 3: yes / mostly / no, for forks with tk>1
Conformance rate (Wilson CI)
Yes : 95% followed or mostly followed
Appendix
Table 10: Research questions of the human study, the label or claim each one validates, how it is measured (step of the annotation item, § N.4 ), the statistic reported, and the result (§ N.7 ).
Figure 7: The annotation tool, shown on a demonstration instance with placeholder annotators ( doctor_schedule task). The workflow steps stay visible on the left of every conversation; only the assistant turns are judged.
Figure 8: Human study results, one panel per research question. (a) RQ1: mean step-1 conformance rating per fork position k over WIC , SSPC , and GR forks (bars) and over all four types (markers; CSC keeps every component of the workflow, see text), and for conformant dialogues, with 95% bootstrap CIs. (b) RQ2: share of forked dialogues in which the true injected instruction was confirmed (yes or partly), per hallucination type, vs. the share of foil instructions confirmed, with 95% Wilson CIs. (c) RQ3: absolute difference between the annotated onset and the fork turn tk , over confirmed true instructions. (d) RQ4: answers to the prefix question per fork position ( k=1 forks have no pre-fork part).
People
LLM judge
study
study
all dialogues
Rating, k=1/2/3/4
14/19/41/58
15/16/28/55
11/15/27/53
Rating, conformant
94
98
97
Instruction confirmed
91%
85%
84%
Foil confirmed
8%
0%
2.0%
Onset at tk / within ±1
67% / 77%
72% / 82%
71% / 84%
Appendix
Table 11: The calibrated LLM judge ( Qwen3.6-35B ) on the study’s conversations and on the whole corpus, next to the annotators. Ratings are means over WIC , SSPC and GR forks (§ N.7 ); confirmation and onset are over forks shown the true instruction; foils are those shown on conformant dialogues.
Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful long-horizon procedural execution.
Do LLM agents act on the reasoning they state? This question of process fidelity is central to LLM-based social simulation, yet hard to measure where no reference for correct behavior exists. We study it in a controlled setting: a Texas Poker simulator with a verifiable reference action for every decision by splitting the faithfulness gap into two steps: reasoning-to-conclusion (does the stated decision follow from the agent's own reasoning?) and conclusion-to-action (does the agent execute what it states?). The two steps behave very differently. Conclusion-to-action is reliable: inconsistency is 0.7% for Claude Haiku 4.5 and 1.4% for DeepSeek-Reasoner once the conclusion is read from an explicit tag, whereas free-text conclusion extraction reports 22-26%. Reasoning-to-conclusion is where fidelity frays, but not through a single dominant failure. In a step-level diagnostic the agent's errors split roughly evenly between bad inputs, borderline cases, and rule misapplication deriving a conclusion that contradicts the agent's own restated rule from inputs it estimated correctly. This composition is model-dependent: rule misapplication accounts for a third of Haiku's interpretable errors but only 8% of DeepSeek's. The one robust signal is directional: when an agent does misapply its own stated rule, it almost always (99.5% for Haiku) errs in the risk-averse direction. The override is partly hedging behavior, not a capability limit: instructing the agent to apply the rule mechanically halves the misapplication rate (13.9% to 6.8% of decisions) and raises adherence by eight points. Process-fidelity evaluation should therefore elicit machine-checkable conclusions and probe for directional biases rather than assume a single upstream failure mode, lest it conflate measurement noise with model behavior.
Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated against human annotation. We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains, nine production LLMs, and a 100-label human-validated subset. We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation. Substring-based judging agrees with human annotation at kappa=0.049 (chance-level); a three-LLM ensemble reaches kappa=0.432 (moderate) with a conservative bias. Under validated evaluation, a parameter-level injection propagates to a wrong final answer with human-calibrated probability approximately 0.62 (range 0.46-0.73 across models). Rejection (catching bad parameters) and recovery (correcting after acceptance) are independent model capabilities (Spearman rho=0.126, p=0.747). A tuned runtime interceptor reduces hallucination on GPT-4o-mini by 23.0 percentage points under a concurrent n=600 control, but shows no significant effect on Gemini-2.0-Flash, whose aggressive parameter rejection eliminates the target failure mode. All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.