cs.CLOct 26, 2026
SaveA Contextual Measure of Ambiguity in Prescriptive Text
Organizations: Massachusetts Institute of Technology · AI & Democracy Foundation
Abstract
Ambiguity in prescriptive text such as laws, contracts, rules, policies, or recommendations can be both desirable (e.g., if it enables a law to adapt to changing context, such as new technologies) and undesirable (e.g., if it leads to arbitrary discretion or conflict over the law's interpretation). To help navigate these trade-offs, it would be valuable to have methods for quantifying ambiguity that are predictive of such outcomes, but existing measures capture how text is perceived in isolation rather than how it is interpreted in context. In this paper, we propose quantifying ambiguity as the degree of disagreement among a set of stakeholders ("raters") on how alternative prescriptive texts ("policies") apply in each of a set of relevant scenarios ("cases"). Using two policy domains, we present preliminary analysis suggesting that, even when using simulated raters, this approach (i) correlates with real human disagreement, and (ii) is robust to spurious text features. We then apply the measure to existing deliberation datasets to describe the association between policy ambiguity and support for those policies.
Figures & tables
Figure 1: Visual intuition for the DELTA framework. Given context on a policy domain, a set of alternative policies, a set of cases to which the policies might apply, and a set of raters , each rater judges whether each policy permits and whether it prohibits each case, and the two answers are averaged into a single judgment. For each policy, disagreement is computed across raters for each case and then averaged across cases, yielding a score that captures the ambiguity of the policy relative to the chosen cases and population of raters. Cases and raters may be human-provided or, as in LLM-DELTA, generated and simulated with LLMs.
Figure 2: Each cell in the heatmap reports the average Spearman correlation between LLM-DELTA scores computed from a random subset of cases and raters and reference scores computed using the full evaluation design.
Figure 3: Spearman correlations between four computational ambiguity measures and the human-DELTA scores for the first dataset of 20 university free-speech policies, with bootstrapped 95% confidence intervals over 2000 iterations.
Figure 4: Correlations between LLM-DELTA and human-DELTA on dataset 1, with raw scores shown on the left and rankings shown on the right. Each point represents one policy.
Figure 5: Each point represents one policy in the adversarial dataset. Both LLM-as-a-judge ambiguity ratings and disagreement ratings rank the policies in nearly the reverse order of LLM-DELTA.
Figure 6: LLM-DELTA and bridging show null or negative Pearson correlation across four Pol.is datasets, contrary to the intuition that ambiguity fosters broader support.
Figure 7: Collective statements are ranked from least to most ambiguous according to DELTA scores. Lines show smoothed shares of participant agreement ratings on the 1–7 scale. 95% confidence bands are obtained using 300 bootstrapping iterations.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: The consent form used in the human subject study. Note that payment details had already been disclosed via the Prolific interface.
Figure A.2: Instructions given to participants in the human subject study at the beginning.
Figure A.3: An example of the rating task assigned to participants in the human subject study.
Figure A.4: An example of an attention check during the human subject study. Each participant was shown two such checks in total, and failing any led to the exclusion of their entire response from analysis.
| Policy A | Policy B | |||
| surface-ambig, operationally unambig | surface-unambig, operationally ambig | |||
| Case | Group 1 | Group 2 | Group 1 | Group 2 |
| Case A ( , favours G1) | Y | Y | Y | N |
| Case B ( , favours G1) | N | N | Y | N |
| Case C ( , favours G2) | Y | Y | N | Y |
| Case D ( , favours G2) | N | N | N | Y |
Table B.1: Target judgment matrix. Each case takes a value on the consensus axis (shared by both groups) and on the conflict axis (interpreted differently by the two groups); the notation “ favours G1” (resp. “G2”) denotes the value of on which Group 1’s (resp. Group 2’s) reading of Policy B leads to Y. Y denotes that a group judges the case to be permitted by, or consistent with, the policy; N denotes the opposite.
| Dataset | Cleaned | High-eng. | A | B | |
|---|---|---|---|---|---|
| Uber | 199 | 72 | 63 | 410 | 398 |
| Consump. | 522 | 275 | 126 | 540 | 134 |
| Housing | 613 | 314 | 91 | 335 | 705 |
| Energy | 1,041 | 515 | 133 | 206 | 984 |
Table C.2: Summary statistics by dataset. denotes the number of statements in the raw export. “Cleaned” denotes the number remaining after the moderation filter and manual curation; these are the statements DELTA scores. “High-eng.” denotes the number of high-engagement statements used in the bridging analysis, defined as those on which at least 20% of the participants in each of groups A and B registered a response (agree, disagree, or pass). “A” and “B” denote the sizes of the two opinion groups algorithmically identified by Pol.is, estimated as the largest number of votes the group cast on any single statement.
Figure C.6: Spearman rank correlation between bridging, , and DELTA as a function of the minimum engagement cutoff: statements are retained only if at least the given fraction of each opinion group’s participants responded to them. Shaded bands are Fisher- confidence intervals; lines truncate where fewer than statements remain. The dashed vertical line marks the cutoff used in Figure 6 . The climate–energy correlation is negative at every cutoff and significantly so across a band of intermediate cutoffs; no dataset shows a stable positive correlation at any cutoff.
| Metric | Value |
|---|---|
| Raw release (nine questions) | |
| Deliberation questions | 9 |
| Unique deliberation rounds | 344 |
| Total participants | 1,720 |
| Total collective statements | 2,693 |
| Draft statements | 1,367 |
Table D.3: Summary statistics for the raw release and the cleaned dataset. The raw block spans all nine questions; cleaning excludes the Internet question, rounds that did not complete both iterations, and round-iterations with no rateable winning statement. Where a question’s statement count is below twice its round count, one iteration of one round produced no rateable winner.
Figure E.7: Prompt to generate cases, three at a time, from an ordered pair of policies. Placeholders in braces are filled from the dataset’s experiment configuration.
Figure E.8: Top: prompt to generate rater identities, ten at a time in opposing pairs. Bottom: prompt to expand a rater description, which takes the current description and adds one sentence to it; it is applied four times, starting from the generated identity, to reach the final five-sentence description.
Figure E.9: The judgment prompt for the Pol.is and Habermas datasets. The final question takes one of two framings, asked in separate queries: the positive framing “Does the intention of the policy proposal support/is compatible with the principles and beliefs described in the opinion statement?” and the negative framing “Does the intention of the policy proposal contradict/stand in opposition to the principles and beliefs described in the opinion statement?”
Figure E.10: The judgment prompts for the university speech dataset (top) and the adversarial dataset (bottom). As in fig. E.9 , each judgment is asked under two framings in separate queries. University speech: “Would this person say the speech in the incident is permitted under the policy?” and “Would this person say the speech in the incident is prohibited under the policy?” Adversarial: “Would this person say the proposed implementation is consistent with the constitutional aspiration?” and “Would this person say the proposed implementation is inconsistent with the constitutional aspiration?”
Figure E.11: Prompt for the LLM-as-a-judge ambiguity baseline.
Figure E.12: Prompt for the LLM-as-a-judge disagreement baseline, which mirrors the DELTA mechanism (part 1 of 3).
Figure E.13: Prompt for the LLM-as-a-judge disagreement baseline, which mirrors the DELTA mechanism (part 2 of 3).
Figure E.14: Prompt for the LLM-as-a-judge disagreement baseline, which mirrors the DELTA mechanism (part 3 of 3).
More in this topic
Annotator Disagreement
Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.
See all 91 papers