Ambiguity is a core challenge in human--AI interaction, where models must decide when to make assumptions and when to seek clarification. Existing approaches to ambiguity evaluation largely focus on coverage over multiple interpretations using static metrics, offering limited insight into how efficiently a model converges toward a single user-intended interpretation through interaction. We introduce a preference-grounded simulated user framework for controlled, multi-turn evaluation of ambiguity resolution, grounding simulator responses strictly in user disambiguation preferences to avoid hallucination and over-disclosure. Alongside this, we define three observable dissatisfaction signals --- incorrect assumptions, repeating clarifications, and irrelevant clarifications --- that characterize interaction quality beyond task accuracy. Our analysis reveals a clear trade-off: models that frequently seek clarification tend to over-clarify and prolong interactions, while assumption-driven models are more prone to misaligning with user intent. We demonstrate that system prompts optimized on these dissatisfaction signals achieve a Pareto-optimal balance across all three metrics, reducing overall error rates by up to 7.65%, underscoring the value of our framework in designing agents that ask precisely and assume wisely.
Figures & tables
Figure 1: Demonstrates different ambiguity resolution styles and signals of user dissatisfaction during interaction. Users may choose different resolution strategies towards a given ambiguity. Solution-driven models often make incorrect assumptions while clarification-seeking models ask repeating or irrelevant clarifications.
Figure 2: Preference-Grounded User Simulator that rectifies or clarifies based on preference set P .
Model
Acc(%) ↑
|Conv| ↓
IncAssmp ↓
RepClrf ↓
IrrelClrf ↓
CIR(%) ↑
AmbigQA
Phi-4
34.34
2.03
1.13
0.02
0.02
21.54
DeepSeek-R1-Distill
30.79
2.05
1.07
0.06
0.04
23.36
Claude-Opus-4.6
65.11
1.85
0.88
0.05
0.02
30.50
GPT-5.4
63.47
1.90
0.81
0.12
0.02
31.84
Ambrosia
Table 1: Reports task success, measured by accuracy Acc(%), alongside interaction efficiency through Clean Interaction Rate CIR(%) and average conversation length ∣Conv∣ , after the model aligns with the user’s intended interpretation of an ambiguous query. ↑ and ↓ denote metrics where higher and lower values are preferable, respectively. Closed-weight models generally achieve stronger overall performance and efficiency.
Strategy
Acc(%) ↑
|Conv| ↓
IncAssmp ↓
RepClrf ↓
IrrelClrf ↓
CIR(%) ↑
AmbigQA
None
63.47
1.90
0.81
0.12
0.02
31.84
Self-Ask
65.72
1.96
0.85
0.14
0.02
30.25
Intent-Sim
62.02
1.85
0.86
0.07
0.02
29.60
DisPO
64.68
2.10
0.68
0.15
0.11
39.49
Clrf
59.80
4.20
0.28
1.83
0.91
24.61
Table 2: Interaction efficiency for different ambiguity resolution strategies. DisPO achieves a Pareto-optimal trade-off between incorrect assumptions and repeated or irrelevant clarifications.
Figure 3: Variation of dissatisfaction signals as extent of ambiguity ∣P∣ increases going vertically down. Individual bars report average incorrect assumptions ( blue ), repeating clarifications ( red ), and irrelevant clarifications ( green ). None incurs incorrect assumptions that worsen monotonically with ∣P∣ , while Clrf observes reduction in IrrelClrf with ∣P∣ . DisPO achieves a Pareto-optimal profile across all three metrics and ambiguity depth.
GPT-5
Phi-4
Metric
Base.
Ours
Base.
Ours
Acc (%)
63.21 ± 6.89
61.71 ± 0.93
45.39 ± 14.48
62.16 ± 0.44
∣ Conv ∣
1.98 ± 0.02
2.23 ± 0.31
2.56 ± 0.05
3.02 ± 0.02
Pref. Viol. (%)
5.26
1.01
7.05
4.87
Hallucin. (%)
0.00
0.00
11.00
0.00
Overdisc. (%)
3.06
1.68
2.67
0.33
Table 3: User simulator comparison.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Std Dev
Variance
Acc(%)
0.92
0.85
∣ Conv ∣
0.0515
0.003375
IncAssmp
0.0540
0.003579
RepClrf
0.0208
0.000561
IrrelClrf
0.0478
0.003311
CIR(%)
2.76
9.77
Appendix
Table 4: Standard deviation and variance of GPT-5.4 with DisPO over five independent runs, averaged across AmbigQA, Ambrosia, and nvBench2.0.
Table 5: Within-query sensitivity of GPT-5.4 across ambiguity-resolution strategies. Values are the mean per-query standard deviation across interpretations; higher values indicate more variation across user preference sets P .