Organizations: Hong Kong University of Science and Technology (Guangzhou) · University of Maryland, College Park · Peking University · Huawei Technologies Ltd.
Large Language Models (LLMs) are increasingly deployed through conversational interfaces, yet strong single-turn performance does not imply reliability in multi-turn settings. Multi-turn assistance requires a model to maintain an evolving representation of user requirements and to reason over it before complying with later requests. Failures can therefore be subtle: models may fail to recognize a changed constraint or hidden conflict, or recognize it without enforcing it in the final response. Existing benchmarks primarily test fully specified prompts, long-context recall, or controlled domain environments, leaving open-domain requirement reconciliation undermeasured. We introduce TurnFlow, an interactive benchmark for evaluating whether models preserve, revise, and act on user requirements across turns. TurnFlow combines adaptive user simulation with verifiable judging, allowing conversations to unfold naturally while keeping hidden requirements and pass/fail criteria controlled. In our evaluation, nine LLMs achieve overall pass rates from 42.86% to 59.59% on 490 scenarios. Performance varies substantially across challenge types: models perform well on Stateful Multi-Constraint Following but remain considerably less reliable on Scope Creep and Goal Shift. These results reveal persistent limitations in reasoning over evolving user requirements across turns.
Figures & tables
Figure 1: TurnFlow pass rates across five challenge categories. Each colored point represents one of the nine evaluated models, and black horizontal bars indicate the mean pass rate across models within each category.
Benchmark
Domain Diversity
Reactive
User State
Challenge Diversity
Diagnostics
MT-Bench ( Zheng et al., 2023 )
△
✗
✗
△
✗
MTBench-101 ( Bai et al., 2024 )
△
✗
✗
✓
△
MultiChallenge ( Deshpande et al., 2025 )
✓
✗
△
✓
✓
Lost-in-Multi-Turn ( Laban et al., 2025 )
✓
✗
✓
✗
✓
τ2 -bench ( Barres et al., 2025 )
✗
✓
✓
△
✓
SOTOPIA ( Zhou et al., 2024 )
△
✓
✓
△
✓
Table 1: Comparison with representative multi-turn and agent-conversation benchmarks. ✓ indicates a primary design target, △ partial coverage, and ✗ not a primary target. “Domain diversity” refers to the coverage of multiple domains; “Reactive” to user or environment responses conditioned on the tested model’s behavior; “User State” refers to having explicit user, task, or environment state that controls the interaction; “Challenge Diversity” refers to the coverage of multiple challenge types; and “Diagnostics” refers to having built-in axes, labels, element-level criteria, or ablations that help analyze where failures come from rather than only reporting an aggregate score.
Figure 2: TurnFlow Workflow. Open-domain dialogue are filtered and categorized into 5 challenge types and 9 domains. Given a dialogue log and its corresponding coverage type, the scenario synthesizer extracts a structured scenario template, generates or repairs a candidate scenario specification, and applies LLM-based and rule-based checks to assess structural validity, task content, difficulty, and information leakage. Accepted candidates are serialized as scenario objects containing the user persona and opening anchor, dialogue state and staged aims, constraints and hidden information, and required and forbidden rubric elements. Each scenario is subsequently consumed by the reactive user simulator to guide the conversation and referenced by the response verifier to evaluate the final response. The detailed conversation of the reactive user simulator is illustrated in Figure 3 .
Figure 3: A specific example (Challenge Type: Scope Creep) illustrating the reactive user simulator (BRIDGE) workflow in TurnFlow. Specifically, the conversation progressively expands a beginner-oriented Excel tracker into an enterprise-grade system. The user simulator advances through a sequence of staged aims before issuing the final probe. Although the evaluated model recognizes the accumulated complexity, it still provides a month-level implementation plan, leading the rubric-based response verifier to return fail as shown in Figure 2 .
Figure 4: Scenario distribution across the 9 dialog domain, stacked by challenge type. Bar height is the per-domain total; coloured segments show challenge-type contributions. Per-(challenge, domain) counts are reported in Appendix B .
Model
Passed
Challenge category
Overall
/490
Stateful
Scope
Collision
Override
Goal Shift
Kimi-K3
292
93.00
44.30
55.00
48.45
54.39
59.59
ANTH-O-4.6
288
90.00
27.85
79.00
60.82
33.33
58.78
DeepSeek-V4.1-Flash
259
91.00
34.18
54.00
54.64
29.82
52.86
OAI-5.5
258
96.00
25.32
57.00
73.20
12.28
52.65
MiniMax-M3
247
92.00
22.78
56.00
60.82
19.30
50.41
Table 2: Multi-turn pass rates (%) on TurnFlow. All models are evaluated on the same 490 scenarios. Overall is the percentage of scenarios passed. The five categories contain 100, 79, 100, 97, and 114 scenarios, respectively. Rows are ordered by Overall; bold marks the highest observed score in each column.
Figure 5: A trajectory passes Fidelity-AC only if it delivers the intended final trap at the right point, faithfully discloses task-critical content, and keeps the task presentation free of self-solving, contradiction, or premise drift. BRIDGE substantially improves trajectory fidelity over prompt-only and preassigned-action user strategies. We sampled 40 scenarios for this evaluation. Per-challenge counts are reported in Appendix H .
Figure 6: The primary bars represent stable fail cases in the multi-turn setting that become stable pass cases in single-turn setting. Gray ticks denote multi-turn stable pass cases that remain stable pass under the same collapsing procedure. We sample 80 scenarios per model in total. A category-level breakdown is provided in Appendix E .
Figure 7: (a) Pass rates under FTA, Replay, BRIDGE, and Fixed Disclosure on the same 200 matched diagnostic cells. Fixed Disclosure bypasses the LLM user simulator and presents the prescribed scenario information through deterministic templates. (b) Cross-replay transfers the source model’s frozen user turns, while Cross-FTA transfers its complete dialogue prefix. Forward pass lift and reverse fail lift measure target verdicts that flip toward the source verdict in the two directions; control flip applies the corresponding reuse procedure using a same-model trajectory. FTA denotes final-turn answer only.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Check family
Failure mode targeted
Action
Schema
Missing runnable fields or empty criteria
reject or regenerate
Reference
Aim references unresolved hidden-state keys
reject or regenerate
Derivability
Criteria depend on external or implicit facts
reject
Leakage and triviality
Final answer exposed too early or made trivial
reject or regenerate
Challenge specific
Scenario does not instantiate the target mechanism
reject or regenerate
Appendix
Table 3: Scenario-level quality-control checks applied before simulation. These checks validate the benchmark item itself; model-output reliability is evaluated separately by the judge and simulator diagnostics.
Domain
Stateful
Scope
Collision
Override
Goal Shift
Total
Career
0
6
2
13
72
93
Food
81
0
0
7
0
88
Digital
0
43
10
24
3
80
Finance
0
9
33
6
10
58
Household
0
2
15
24
6
47
Travel
4
2
9
13
10
38
Appendix
Table 4: Per-(challenge, domain) scenario counts on the stable pool used in the main results. Domain abbreviations follow Figure 4 .
Model
Challenge category
Avg.
Stateful
Scope
Collision
Override
Goal Shift
ANTH-O-4.6
89.0
19.0
69.0
63.9
14.0
51.0
DeepSeek-V4-Flash
61.0
5.1
51.0
52.6
1.8
34.3
DeepSeek-V4-Pro
70.4
11.4
44.0
53.6
5.2
36.9
GDM-3.1-Pro
59.0
1.3
45.0
54.6
0.9
32.2
GLM-5
72.0
24.1
48.0
55.7
1.8
40.3
Appendix
Table 5: Original multi-turn pass rates (%) on TurnFlow. Avg. denotes the unweighted mean of the five category-level pass rates, following the original reporting convention. Bold marks the highest reported score in each column.
Figure 8: Per-category single-turn recovery diagnostic. The first bar shows MT-stable-fail cases that become stable PASS after single-turn collapse; the second shows MT-stable-pass controls that remain stable PASS. Row labels report recovery/control sample sizes.
Challenge
Task-specific fact key
Question/template slot
Stateful
Active entities, corrected or removed constraints, safe options, and stale-constraint traps.
Does the reply flag required active constraints and avoid warning about corrected, cleared, or out-of-scope constraints?
Scope Creep
Embedded risks behind positive framing and forbidden escalation actions.
Does the reply address each risk concretely, avoid trap actions, and avoid green-lighting the expanded plan?
Constraint Collision
Hard cap, candidate items, within-cap and over-cap options, user-stated distractor choice.
Does the reply recommend a valid option and avoid endorsing, purchasing, or exempting invalid options?
For each override item, does the reply set aside the preference, name a non-preference method, and avoid trap alternatives?
Goal Shift
Original strategic decision, scenario-defined execution goal, and grounds for blocking continuation.
Return a BLOCK / ALLOW action label together with a verbatim reason_span .
Appendix
Table 6: Summary of task-specific extraction targets. Each scenario supplies its own packaged judge definition and output schema. Deterministic reducers compute final outcomes from the structured outputs.
Challenge
Sampled
4-model
10-model
Retained
Stateful
20
19
18
94.7%
Scope Creep
20
18
18
100.0%
Constraint Collision
20
20
19
95.0%
Preference Override
20
19
19
100.0%
Goal Shift
20
20
19
95.0%
Overall
100
96
93
96.9%
Appendix
Table 7: Expanded-panel stability sensitivity. Of the 96 scenarios stable under the four-model audit criterion, 93 remain stable after expanding to ten models. This tolerance-one diagnostic is distinct from the strict zero-flip production filter.
Realism measure
Average
Turn-level naturalness (1–5)
4.4
Conversational coherence (1–5)
4.6
Persona/style consistency (1–5)
4.4
Turns flagged as overly complete
4.1%
Turns flagged as overly polite
2.5%
Turns flagged as template-like
5.8%
Appendix
Table 8: Human realism audit over 40 BRIDGE trajectories.
Scenario-validity measure
Acceptance
Constraint/disclosure plausibility
90%
Final-challenge derivability
93%
Required/forbidden criterion validity
89%
All three criteria satisfied
87%
Appendix
Table 9: Majority-vote acceptance rates in the 100-scenario validity audit.
Challenge
Prompt-only
Preassigned
BRIDGE
Stateful Multi-Constraint Following
0/8
0/8
6/8
Scope Creep
1/8
3/8
6/8
Constraint Collision
0/8
0/8
8/8
Preference Override
2/8
5/8
7/8
Goal Shift
0/8
6/8
8/8
Overall
3/40
14/40
35/40
Appendix
Table 10: Full BRIDGE trajectory-fidelity diagnostic. Values count trajectories passing Fidelity-AC under Prompt-only, Preassigned-Action, and BRIDGE. A pass requires correct final-probe delivery, faithful disclosure of task-critical content, and an uncorrupted task presentation.
Simulator
OAI-5.5
GDM-3.1-Pro
DS-Pro
ANTH-O
GDM-3-Flash
38%
31%
34%
43%
OAI-5.4-mini
36%
30%
33%
43%
DeepSeek-V4-Flash
34%
29%
33%
44%
Appendix
Table 11: Pass rates on the same stratified 100-scenario subset under three BRIDGE backbones. DS-Pro denotes DeepSeek-V4-Pro, and ANTH-O denotes ANTH-O-4.6.
Figure 9: Cross-verifier majority-verdict agreement over 588 cases (241 pass and 347 fail). Model–model pairs appear in the top block; the bottom block compares each extractor with human judgments on a 50-scenario subset. Replies and deterministic verdict rules are fixed.
Model
Baseline
Tracking
Summary
Self-correct
DeepSeek-V4-Pro
41.3
36.5
38.1
46.0
GDM-3.1-Pro
29.8
27.7
29.8
25.5
OAI-5.5
43.6
45.5
50.9
43.6
Qwen3.6-Plus
33.9
37.5
32.1
33.9
Overall
37.2
36.8
37.7
37.3
Appendix
Table 12: Pass rate (%) on the paired 200-scenario intervention subset. These subset baselines are not directly comparable with the full-pool results in Table 2 . Bold marks the best condition within each model row; ties retain both entries.
Intervention
Representative evidence
Diagnostic implication
State tracking
In T19_408, preserving all stated constraints turns a general DIY preference into a hard rule and leads the model to recommend a DIY 200-amp electrical-panel replacement rather than a licensed electrician.
Tracking can preserve active state while failing to distinguish hard requirements from soft preferences.
Self-summary
OAI-5.5 gains most from summaries, but in T20_262 the recap treats the sale plan as settled and optimizes execution rather than reconsidering whether the sale is necessary.
Summaries increase state salience but can consolidate an obsolete execution frame.
Self-correction
On Scope Creep, self-correction revises 60% of initial drafts but recovers none of the failures.
Final-answer editing cannot repair a path-level error after the model has accepted an invalid premise.
Appendix
Table 13: Representative mechanisms behind the uneven intervention effects.