Ambiguity in prescriptive text such as laws, contracts, rules, policies, or recommendations can be both desirable (e.g., if it enables a law to adapt to changing context, such as new technologies) and undesirable (e.g., if it leads to arbitrary discretion or conflict over the law's interpretation). To help navigate these trade-offs, it would be valuable to have methods for quantifying ambiguity that are predictive of such outcomes, but existing measures capture how text is perceived in isolation rather than how it is interpreted in context. In this paper, we propose quantifying ambiguity as the degree of disagreement among a set of stakeholders ("raters") on how alternative prescriptive texts ("policies") apply in each of a set of relevant scenarios ("cases"). Using two policy domains, we present preliminary analysis suggesting that, even when using simulated raters, this approach (i) correlates with real human disagreement, and (ii) is robust to spurious text features. We then apply the measure to existing deliberation datasets to describe the association between policy ambiguity and support for those policies.
Figures & tables
Figure 1: Visual intuition for the DELTA framework. Given context on a policy domain, a set of alternative policies, a set of cases to which the policies might apply, and a set of raters , each rater judges whether each policy permits and whether it prohibits each case, and the two answers are averaged into a single judgment. For each policy, disagreement is computed across raters for each case and then averaged across cases, yielding a score that captures the ambiguity of the policy relative to the chosen cases and population of raters. Cases and raters may be human-provided or, as in LLM-DELTA, generated and simulated with LLMs.
Figure 2: Each cell in the heatmap reports the average Spearman correlation between LLM-DELTA scores computed from a random subset of i cases and j raters and reference scores computed using the full evaluation design.
Figure 3: Spearman correlations between four computational ambiguity measures and the human-DELTA scores for the first dataset of 20 university free-speech policies, with bootstrapped 95% confidence intervals over 2000 iterations.
Figure 4: Correlations between LLM-DELTA and human-DELTA on dataset 1, with raw scores shown on the left and rankings shown on the right. Each point represents one policy.
Figure 5: Each point represents one policy in the adversarial dataset. Both LLM-as-a-judge ambiguity ratings and disagreement ratings rank the policies in nearly the reverse order of LLM-DELTA.
Figure 6: LLM-DELTA and bridging show null or negative Pearson correlation across four Pol.is datasets, contrary to the intuition that ambiguity fosters broader support.
Figure 7: Collective statements are ranked from least to most ambiguous according to DELTA scores. Lines show smoothed shares of participant agreement ratings on the 1–7 scale. 95% confidence bands are obtained using 300 bootstrapping iterations.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: The consent form used in the human subject study. Note that payment details had already been disclosed via the Prolific interface.
Figure A.2: Instructions given to participants in the human subject study at the beginning.
Figure A.3: An example of the rating task assigned to participants in the human subject study.
Figure A.4: An example of an attention check during the human subject study. Each participant was shown two such checks in total, and failing any led to the exclusion of their entire response from analysis.
Policy A
Policy B
surface-ambig, operationally unambig
surface-unambig, operationally ambig
Case
Group 1
Group 2
Group 1
Group 2
Case A ( C=Y , H favours G1)
Y
Y
Y
N
Case B ( C=N , H favours G1)
N
N
Y
N
Case C ( C=Y , H favours G2)
Y
Y
N
Y
Case D ( C=N , H favours G2)
N
N
N
Y
Appendix
Table B.1: Target judgment matrix. Each case takes a value on the consensus axis C (shared by both groups) and on the conflict axis H (interpreted differently by the two groups); the notation “ H favours G1” (resp. “G2”) denotes the value of H on which Group 1’s (resp. Group 2’s) reading of Policy B leads to Y. Y denotes that a group judges the case to be permitted by, or consistent with, the policy; N denotes the opposite.
Dataset
n
Cleaned
High-eng.
A
B
Uber
199
72
63
410
398
Consump.
522
275
126
540
134
Housing
613
314
91
335
705
Energy
1,041
515
133
206
984
Appendix
Table C.2: Summary statistics by dataset. n denotes the number of statements in the raw export. “Cleaned” denotes the number remaining after the moderation filter and manual curation; these are the statements DELTA scores. “High-eng.” denotes the number of high-engagement statements used in the bridging analysis, defined as those on which at least 20% of the participants in each of groups A and B registered a response (agree, disagree, or pass). “A” and “B” denote the sizes of the two opinion groups algorithmically identified by Pol.is, estimated as the largest number of votes the group cast on any single statement.
Figure C.6: Spearman rank correlation between bridging, min(pA,pB) , and DELTA as a function of the minimum engagement cutoff: statements are retained only if at least the given fraction of each opinion group’s participants responded to them. Shaded bands are Fisher- z95% confidence intervals; lines truncate where fewer than 20 statements remain. The dashed vertical line marks the 0.2 cutoff used in Figure 6 . The climate–energy correlation is negative at every cutoff and significantly so across a band of intermediate cutoffs; no dataset shows a stable positive correlation at any cutoff.
Metric
Value
Raw release (nine questions)
Deliberation questions
9
Unique deliberation rounds
344
Total participants
1,720
Total collective statements
2,693
Draft statements
1,367
Appendix
Table D.3: Summary statistics for the raw release and the cleaned dataset. The raw block spans all nine questions; cleaning excludes the Internet question, rounds that did not complete both iterations, and round-iterations with no rateable winning statement. Where a question’s statement count is below twice its round count, one iteration of one round produced no rateable winner.
Figure E.7: Prompt to generate cases, three at a time, from an ordered pair of policies. Placeholders in braces are filled from the dataset’s experiment configuration.
Figure E.8: Top: prompt to generate rater identities, ten at a time in opposing pairs. Bottom: prompt to expand a rater description, which takes the current description and adds one sentence to it; it is applied four times, starting from the generated identity, to reach the final five-sentence description.
Figure E.9: The judgment prompt for the Pol.is and Habermas datasets. The final question takes one of two framings, asked in separate queries: the positive framing “Does the intention of the policy proposal support/is compatible with the principles and beliefs described in the opinion statement?” and the negative framing “Does the intention of the policy proposal contradict/stand in opposition to the principles and beliefs described in the opinion statement?”
Figure E.10: The judgment prompts for the university speech dataset (top) and the adversarial dataset (bottom). As in fig. E.9 , each judgment is asked under two framings in separate queries. University speech: “Would this person say the speech in the incident is permitted under the policy?” and “Would this person say the speech in the incident is prohibited under the policy?” Adversarial: “Would this person say the proposed implementation is consistent with the constitutional aspiration?” and “Would this person say the proposed implementation is inconsistent with the constitutional aspiration?”
Figure E.11: Prompt for the LLM-as-a-judge ambiguity baseline.
Figure E.12: Prompt for the LLM-as-a-judge disagreement baseline, which mirrors the DELTA mechanism (part 1 of 3).
Figure E.13: Prompt for the LLM-as-a-judge disagreement baseline, which mirrors the DELTA mechanism (part 2 of 3).
Figure E.14: Prompt for the LLM-as-a-judge disagreement baseline, which mirrors the DELTA mechanism (part 3 of 3).
Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two τ2-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.
LLM-based agents are increasingly deployed for routine but consequential tasks in real-world domains, where their behavior is governed by inherently ambiguous domain policies that admit multiple valid interpretations. Despite the prevalence of such ambiguities in practice, existing agent benchmarks largely assume unambiguous, well-specified policies, leaving a critical evaluation gap. We introduce DRIP-R, a benchmark that systematically exploits real-world retail policy ambiguities to construct scenarios in which no single correct resolution exists. DRIP-R comprises a curated set of policy-ambiguous return scenarios paired with a realistic customer personas, a full-duplex conversational simulation with tool-calling capabilities and a multi-judge evaluation framework covering policy adherence, dialogue quality, behavioral alignment, and resolution quality. Our experiments show that frontier models fundamentally disagree on identical policy-ambiguous scenarios, confirming that ambiguity poses a genuine and systematic challenge to LLM decision-making.
Hsuvas Borkakoty, Sebastian Pohl, Cheng Wang +2
Interdisciplinary Transformation University, Austria · Amazon, Berlin, Germany
Legal contracts contain ambiguities that expose enterprises to financial and legal risks. Some ambiguities allow flexible interpretation without triggering disputes, while others lead to significant legal conflicts. This makes identification alone insufficient, and interpretable rationale analysis essential. We propose LAURA, a post-training framework for interpretable ambiguous clause identification. LAURA leverages knowledge distillation with an IRAC-Unlearning prompting technique to transfer knowledge from a teacher LLM to an open-weight student model (<=1B parameters), which is then trained using a joint objective combining classification and rationale generation losses. The framework supports both legal and non-legal stakeholders in making informed decisions about which ambiguities require further attention. Extensive experiments across 7 baselines and 7 open-weight models demonstrate that LAURA with Flan-T5 (250M) delivers state-of-the-art interpretability over all interpretable baselines while matching the identification performance of the best-performing opaque baseline.
Amrita Singh, Aditya Joshi, Jiaojiao Jiang +1
School of Computer Science and Engineering University of New South Wales (UNSW), Sydney