Ambiguity is a core challenge in human--AI interaction, where models must decide when to make assumptions and when to seek clarification. Existing approaches to ambiguity evaluation largely focus on coverage over multiple interpretations using static metrics, offering limited insight into how efficiently a model converges toward a single user-intended interpretation through interaction. We introduce a preference-grounded simulated user framework for controlled, multi-turn evaluation of ambiguity resolution, grounding simulator responses strictly in user disambiguation preferences to avoid hallucination and over-disclosure. Alongside this, we define three observable dissatisfaction signals --- incorrect assumptions, repeating clarifications, and irrelevant clarifications --- that characterize interaction quality beyond task accuracy. Our analysis reveals a clear trade-off: models that frequently seek clarification tend to over-clarify and prolong interactions, while assumption-driven models are more prone to misaligning with user intent. We demonstrate that system prompts optimized on these dissatisfaction signals achieve a Pareto-optimal balance across all three metrics, reducing overall error rates by up to 7.65%, underscoring the value of our framework in designing agents that ask precisely and assume wisely.
Figures & tables
Figure 1: Demonstrates different ambiguity resolution styles and signals of user dissatisfaction during interaction. Users may choose different resolution strategies towards a given ambiguity. Solution-driven models often make incorrect assumptions while clarification-seeking models ask repeating or irrelevant clarifications.
Figure 2: Preference-Grounded User Simulator that rectifies or clarifies based on preference set P .
Model
Acc(%) ↑
|Conv| ↓
IncAssmp ↓
RepClrf ↓
IrrelClrf ↓
CIR(%) ↑
AmbigQA
Phi-4
34.34
2.03
1.13
0.02
0.02
21.54
DeepSeek-R1-Distill
30.79
2.05
1.07
0.06
0.04
23.36
Claude-Opus-4.6
65.11
1.85
0.88
0.05
0.02
30.50
GPT-5.4
63.47
1.90
0.81
0.12
0.02
31.84
Ambrosia
Table 1: Reports task success, measured by accuracy Acc(%), alongside interaction efficiency through Clean Interaction Rate CIR(%) and average conversation length ∣Conv∣ , after the model aligns with the user’s intended interpretation of an ambiguous query. ↑ and ↓ denote metrics where higher and lower values are preferable, respectively. Closed-weight models generally achieve stronger overall performance and efficiency.
Strategy
Acc(%) ↑
|Conv| ↓
IncAssmp ↓
RepClrf ↓
IrrelClrf ↓
CIR(%) ↑
AmbigQA
None
63.47
1.90
0.81
0.12
0.02
31.84
Self-Ask
65.72
1.96
0.85
0.14
0.02
30.25
Intent-Sim
62.02
1.85
0.86
0.07
0.02
29.60
DisPO
64.68
2.10
0.68
0.15
0.11
39.49
Clrf
59.80
4.20
0.28
1.83
0.91
24.61
Table 2: Interaction efficiency for different ambiguity resolution strategies. DisPO achieves a Pareto-optimal trade-off between incorrect assumptions and repeated or irrelevant clarifications.
Figure 3: Variation of dissatisfaction signals as extent of ambiguity ∣P∣ increases going vertically down. Individual bars report average incorrect assumptions ( blue ), repeating clarifications ( red ), and irrelevant clarifications ( green ). None incurs incorrect assumptions that worsen monotonically with ∣P∣ , while Clrf observes reduction in IrrelClrf with ∣P∣ . DisPO achieves a Pareto-optimal profile across all three metrics and ambiguity depth.
GPT-5
Phi-4
Metric
Base.
Ours
Base.
Ours
Acc (%)
63.21 ± 6.89
61.71 ± 0.93
45.39 ± 14.48
62.16 ± 0.44
∣ Conv ∣
1.98 ± 0.02
2.23 ± 0.31
2.56 ± 0.05
3.02 ± 0.02
Pref. Viol. (%)
5.26
1.01
7.05
4.87
Hallucin. (%)
0.00
0.00
11.00
0.00
Overdisc. (%)
3.06
1.68
2.67
0.33
Table 3: User simulator comparison.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Std Dev
Variance
Acc(%)
0.92
0.85
∣ Conv ∣
0.0515
0.003375
IncAssmp
0.0540
0.003579
RepClrf
0.0208
0.000561
IrrelClrf
0.0478
0.003311
CIR(%)
2.76
9.77
Appendix
Table 4: Standard deviation and variance of GPT-5.4 with DisPO over five independent runs, averaged across AmbigQA, Ambrosia, and nvBench2.0.
Table 5: Within-query sensitivity of GPT-5.4 across ambiguity-resolution strategies. Values are the mean per-query standard deviation across interpretations; higher values indicate more variation across user preference sets P .
User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user's intent, a helpful assistant should surface such ambiguity by asking a clarifying question. Doing so requires two abilities: recognizing that a query is ambiguous, and acting on that recognition by seeking clarification instead of answering directly. To study these abilities, we evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings: standard question answering, explicit ambiguity judgment, and behavioral analysis, where a judge model classifies responses as direct answers, refusals, or clarifying questions. We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers. Retrieved context further widens this gap by improving answerability while making models even less likely to ask clarifying questions.
Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a multi-turn benchmark that evaluates clarification as policy behavior rather than isolated question quality. RegretBench provides a hidden-intent formulation of ambiguity, supports free-form interaction grounded in semantic-state tracking, and introduces a regret-based objective that measures how much value a model loses relative to a reference clarification policy. Experiments on open-domain QA and product recommendation scenarios show that final success alone is insufficient, as models with similar accuracy can differ substantially in efficiency, robustness to user behaviors, and stopping decisions. By jointly measuring intent resolution, interaction cost, ineffective clarification, and regret, RegretBench reveals whether models clarify usefully and efficiently. Our results show that effective clarification requires more than plausible questions: models must ask the right question at the right time and stop once the user's intended meaning is clear.
Minh Ngoc Ta, My Anh Tran Nguyen, Duong D. Nguyen +2
MBZUAI · BKAI Research Center, Hanoi University of Science and Technology · INSAIT, Sofia University "St. Kliment Ohridski"
Current benchmarks for language models primarily evaluate execution on fully specified tasks. However, real user tasks are often ambiguous. Users arrive with incomplete, exploratory, or even inconsistent goals, requiring the assistant to first determine the intended task before carrying it out. We study this problem as task alignment: the ability to align with a user on their intended task. We introduce a general framework for converting specified tasks into underspecified interactions, formalized as a POMDP in which the model must infer a latent task from partial and evolving user intent. We validate our user simulator post hoc with a human user study. Across shopping, coding, and professional work settings, we find that while models often perform well once the task is specified, models still struggle with task alignment: current models act prematurely, interact ineffectively, and fail to resolve ambiguous requests. Models on average recover the user's intended task only 22-32% of the time under ambiguity. In a human study in the same setting, humans reach 48%, outperforming all evaluated models. We show that post-training with supervised fine-tuning and reinforcement learning improves task alignment, but models still lag behind humans in resolving uncertainty through interaction. Together, our results suggest that current models still lack key interaction abilities required for reliable agency.