Detect Before You Disambiguate: Perturbation-Based Input Ambiguity Detection in LLMs

Tiejin Chen; Xiaoou Liu; Michael Clifford; Ahmadreza Moradipari; Yitao Chen; Kyungtae Han; Nejib Ammar; Hua Wei

cs.CLOct 26, 2026

Detect Before You Disambiguate: Perturbation-Based Input Ambiguity Detection in LLMs

Authors: Tiejin Chen, Xiaoou Liu, Michael Clifford, Ahmadreza Moradipari, Yitao Chen, Kyungtae Han, Nejib Ammar, Hua Wei

Organizations: Arizona State University · Toyota InfoTech Labs

Abstract

Large language models (LLMs) often answer ambiguous inputs as if they had a single intended meaning. This is problematic because input ambiguity is an upstream source of uncertainty: when a prompt admits multiple valid interpretations, the key question is whether the model should answer directly at all. Existing input uncertainty methods often rely on LLM-generated statements or clarifications, but such transformations may already disambiguate the input before the ambiguity is measured. We propose a label-preserving perturbation framework for input ambiguity detection in which perturbations remain close to the original input while preserving its label. We instantiate this principle with two detectors: a black-box rule-based perturbation library and a white-box importance-guided method using model-internal signals. Experiments across four datasets in open-domain, conversational, and visual QA show that our methods outperform output-side uncertainty baselines and LLM-paraphrase-based perturbation baselines. Our results highlight the importance of detecting ambiguity before attempting to disambiguate it. Our code is available purple{here}.

Figures & tables

Figure 1: Overview of the proposed ambiguity detection framework. Given an input question and perturbation principles, the method generates label-preserving, low-capacity, and semantics-blind perturbations via white-box token-importance methods and black-box rule-based transformations. The perturbed questions are then sampled to obtain multiple model answers, whose pairwise semantic agreement is used to compute an ambiguity signal: low agreement indicates likely ambiguity, while high agreement suggests the question is likely unambiguous.

Model	Method	ABG-CoQA	AmbigQA-w-Ctx	AQuA	VQ-FA	Avg
		Text QA		Vision QA
Qwen3-4B (Qwen2.5-VL-3B)	Direct Prompt Kadavath et al. (2022)	0.542	0.525	0.359	0.525	0.488
	EigV(Agree) Lin et al. (2024b)	0.521	0.498	0.644	0.562	0.556
	EigV(Disagree) Lin et al. (2024b)	0.543	0.503	0.647	0.587	0.570
	Semantic Entropy Kuhn et al. (2023)	0.493	0.491	0.674	0.477	0.534
	Ours (Black-Box)	0.615	0.630	0.755	0.563	0.641

Table 1: Ambiguity-detection AUROC across three model families and four QA benchmarks. Bold marks the best score within each model block per column, and underline marks the second best. Vision columns use the vision-language counterpart of the text model (Qwen2.5-VL-3B for Qwen, InternVL3-8B for the Llama row, and the same Gemma-3-27B-it checkpoint for Gemma).

Figure 2: Performance comparing different importance methods in two datasets on Llama. The results show that though the performance of importance methods is stable, Vanilla Gradient shows a slightly better result.

Figure 3: AUROC comparison across individual perturbation rules and their combined cross-variant setting. While several single perturbations improve over the original input, combining all perturbation variants achieves the strongest ambiguity detection performance.

Config	Group	Original	UAR	$\Delta$ acc
AmbigQA-w-Ctx	ambiguous	12.19	26.52	$+14.33$
AmbigQA-w-Ctx	non_ambig.	16.28	34.35	$+18.07$
ABG-CoQA	ambiguous	22.76	34.15	$+11.39$
ABG-CoQA	non_ambig.	17.06	25.21	$+8.15$

Table 2: UAR vs. the original baseline on Qwen3-4B, greedy decoding. UAR re-prompts the model with detector-derived evidence; Original is the model’s first answer to the unperturbed question.

Figure 4: Comparison of performance and absolute value of uncertainty on ambiguous questions from AquA. The results show that directly using LLM-paraphrasing can actually reduce the uncertainty.

Model	Dataset	0.6B	4B
Qwen	ABG-CoQA	0.9960	0.9989
	AmbigQA-w-Ctx	0.9781	0.9842
	AQuA	0.9107	0.9191
	VQ-FA	0.8843	0.8923
Llama/InternVL	ABG-CoQA	0.9952	0.9985
	AmbigQA-w-Ctx	0.9749	0.9835

Table 3: Cosine similarity between original and perturbed inputs using Qwen3 embedding models.

Model	Dataset	Ours	LLM
Qwen3-4B	ABG-CoQA	0.615	$0.598$
Qwen3-4B	AmbigQA	0.630	$0.617$
Qwen2.5-VL-3B	AQuA	0.755	$0.715$
Qwen2.5-VL-3B	VQ-FA	0.563	$0.538$

Table 4: Detection AUROC for two realizations of the same rule set. Our perturbations outperform LLM-based ones, while remaining semantics-blind and cheaper.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Performance comparing more white-box importance methods on Abg-CoQA and Qwen3. The results show that our choice is better than simple attention or predictive entropy.

Method	Qwen	Llama/InternVL
Direct Prompt	0.488	0.531
ProCoT	0.574	0.585
Ours (Black-Box)	0.641	0.621
Ours (White-Box)	0.620	0.629

Appendix

Table 5: Mean ambiguity-detection AUROC across the four datasets. ProCoT provides a stronger ambiguity-aware prompting baseline than Direct Prompt, while our perturbation-based detectors achieve higher average AUROC for both model families.

Dataset	Group	Original	Hint	SelfSample	UAR
AmbigQA-w-Ctx	ambiguous	12.19	12.36	20.92	26.52
	non-ambig.	16.28	16.79	25.19	34.35
ABG-CoQA	ambiguous	22.76	24.39	28.46	34.15
	non-ambig.	17.06	17.92	20.28	25.21

Appendix

Table 6: Exact Match comparison between UAR and controlled re-prompting baselines on Qwen3-4B.

# Rules	1	3	5	7	9	11
AUROC	0.587	0.607	0.600	0.618	0.630	0.630

Appendix

Table 7: Ambiguity-detection AUROC under different black-box perturbation budgets on AmbigQA with Qwen3-4B.

LLM Uncertainty Estimation

Large language model uncertainty estimation measures when outputs may be wrong, ambiguous, or unreliable to guide safer decisions.

See all 403 papers

From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs
Sep 3, 2026Omer Nahum, Niv Nayman, Jonathan Fhima +4LLM Uncertainty Estimation Aleatoric Uncertainty
Data and Decision Sciences, Technion, Israel · Amazon Web Services · Computer Vision Group, University of Bern, Switzerland
Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
May 24, 2026Jinyan Su, Claire CardieLLM Evaluation Clarification Question Generation
1Cornell University
What Is a Clarification? Estimating Model-Dependent Input Ambiguity
Oct 14, 2026Jingqin Tang, Tim Tomov, Sebastian Schmidt +2LLM Uncertainty Estimation
School of Computation, Information & Technology, Technical University of Munich · Munich Data Science Institute · Munich Center for Machine Learning

Sep 3, 2026J/K move · Enter open · S save

From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs

Omer Nahum, Niv Nayman, Jonathan Fhima, Alon Zolfi+3

Data and Decision Sciences, Technion, Israel · Amazon Web Services · Computer Vision Group, University of Bern, Switzerland

LLM Uncertainty Estimation Aleatoric Uncertainty

Open paper AskPDF arXiv

A key challenge in reliable LLM deployment is recognizing when uncertainty reflects irreducible variability in the task rather than limitations in the model's knowledge. In language tasks, a central source of such aleatoric uncertainty is input ambiguity or underspecification, where multiple interpretations remain plausible. Existing decomposition methods estimate aleatoric uncertainty by generating multiple clarifications of the input, querying the model for an answer under each clarification, and comparing the resulting answers. We argue that answers are not necessary for identifying ambiguity: they are often redundant, add avoidable cost, and can mislead through epistemic leakage. We support this claim theoretically, and propose a clarification-only approach that estimates this ambiguity-induced component directly from the space of plausible interpretations, without answers to the clarified inputs. Using ambiguity detection as an operational evaluation across three benchmarks, this direct approach improves AUROC (63.34 vs. 60.85), reduces computational cost by 4-26x in output tokens and 2.2-3.5x in API calls, and yields estimates with substantially lower correlation with epistemic uncertainty. Overall, our results suggest that ambiguity-induced aleatoric uncertainty is better estimated from the interpretation space than from the response space.

Abstract

LLM Uncertainty Estimation

Explore similar work

From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs

Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

What Is a Clarification? Estimating Model-Dependent Input Ambiguity