From Opinions to Commitments: Unstable Assignments and Silent Choices in AI-Mediated Proposals
Organizations: Computational Social Science, ETH Zurich, Zurich, Switzerland
Abstract
Large language models (LLMs) are increasingly being used to turn citizens' opinions into collective proposals. These proposals can make assignments: who should act, what must be done, how much one should spend and by when. However, if the assignments change when the same opinions are processed again, publishing a single proposal can hide alternatives that participants never get to see. We examined this issue in four LLM mediators (three open-weight and one commercial) using opinions from DeliberationBank and Polis. For each task, a mediator drafted a proposal from thirty opinions five times, varying either the random seed or the order of the opinions. In 17% of five-run sets from the open-weight mediators and 20% from the commercial one, some runs contained assignments while others did not. Reordering the opinions changed the assignments almost as often as replacing them with opinions from different citizens. Three human annotators confirmed that the detected differences between runs were real, although the LLM judge overstated how often they occurred. Substantial variation remained under four safeguards: temperature zero, a commercial mediator, draft selection, and draft merging. The mediator's objective also shaped representation: prompts seeking agreement covered minority positions much less often than prompts asking to describe differences. A single proposal can therefore obscure choices in both the assignments it proposes and the views it excludes. We propose audits that make alternative assignments visible and disclose the objective behind each proposal.
Figures & tables
| Step | Value | Finding | Requirement |
|---|---|---|---|
| Summarize | Pluralism | Agreement prompts cover minority positions less than differences prompts (Fig. 6 ). | Summarize only with contributors’ consent; state the objective, report coverage and retain unresolved disagreements. |
| Suggest | Human agency | Assignments appeared mainly when the prompt asked for concrete detail (9–13% to 33–41% of statements) and included terms no opinion mentioned (Fig. 2 ). | Generate assignments only when participants request them. |
| Present | Transparency, equal treatment | The same opinions, run again or reordered, produce different assignments (Figs. 2 , 3 ). | Show alternative assignments; test and disclose sensitivity to opinion order; record runs and the source of each assignment. |
| Present | Contestability | Reordering drafts changes selections; merges retain terms found in only one draft (Section 5.2 ). | Open draft selection and merging to review and challenge. |
| Commit | Human agency | In 17 of 23 annotated pairs with an assignment, runs differed in parties or terms or conflicted (Fig. 4 b). | Treat assignments as suggestions until participants endorse them and the assigned parties accept them. |
| Audit | Reliable measurement | Judges report differences that human annotators do not (Fig. 4 a). | Validate audit results against human annotations. |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Study | Focus of evaluation | Variation considered |
|---|---|---|
| Bakker et al. [7] | Approval of consensus statements | Candidates ranked by a reward model |
| Tessler et al. [8] | Endorsement and inclusion of minority critiques | Rounds of human critique |
| Zhu et al. [17] | Representativeness of 18 summarizers | Input size; position of an opinion |
| Fish et al. [21] ; Low et al. [22] | Representation; actionability | Selection among candidates; revision by participants |
| Najera et al. [23] ; Majumdar et al. [24] | Agreement between models; fairness of LLM votes | Different models; repeated votes |
| Cooper et al. [30] | Arbitrariness of classifier decisions | Retraining replicates |
| Models | Role |
|---|---|
| Qwen3 8B; Gemma 3 12B; Mistral Small 3.2 24B | Main open-weight mediators for the primary analyses on both datasets; each also merges its own drafts (RQ2). |
| GPT-5.6 Terra | Main commercial mediator on both datasets. |
| Qwen3 8B; Gemma 3 12B; Llama 3.1 8B; Phi-4 14B | Mediators for DeliberationBank samples 2 and 3 (RQ3). |
| Llama 3.1 8B; Phi-4 14B | Rankers that select one draft from five (RQ2). |
| GPT-6.1 Sol; GPT-5.6 Luna | Main and secondary LLM judges for statements used in the analyses. |
| Claude Haiku 5.5 | Judge from another developer, used to check the main judges. |
| Clause | Text |
|---|---|
| Find agreement | Your statement must set out what these citizens can agree on, in terms that people on opposing sides would both accept. |
| Set out differences | Your statement must set out where these citizens differ, as they actually differ. Do not look for agreement and do not propose a reconciliation. |
| Concrete | Be concrete: say who pays, how much, who qualifies, what happens if the rule is broken, and by when. |
| Plain language | Write in plain, direct language that a general newspaper reader could follow without any difficulty or specialist knowledge. |
| Component | Design arithmetic | Statements |
|---|---|---|
| DB: order and new-opinion reruns | configurations | 3,600 |
| DB: additional Mistral seed outputs | 300 | |
| Polis: open-model seed and order | 1,800 | |
| Terra: seed and order, both datasets | 1,200 | |
| DB: four-model grid, three samples | 4,320 | |
| DB: Mistral identical-prompt repetitions | 180 saved responses | 180 |
| Stratum | Statements | Pairs | Annotated statements | Annotated pairs | Control pairs |
|---|---|---|---|---|---|
| DeliberationBank: shuffled order, new sample; GPT-5.6 Terra | 4,200 | 1,680 | 24 | 20 | 4 |
| DeliberationBank: random seed | 900 | 360 | 18 | 10 | 0 |
| Polis: random seed, shuffled order | 2,400 | 960 | 16 | 16 | 2 |
| Total | 7,500 | 3,000 | 58 | 46 | 6 |
| Reader | Anyone assigned? (58) | Terms specified? (58) | How do two runs relate? (46) |
|---|---|---|---|
| Annotator 1 | 1.00 [1.00, 1.00] | 0.86 [0.67, 1.00] | 0.94 [0.81, 1.00] |
| Annotator 2 | 0.69 [0.50, 0.86] | 0.78 [0.58, 0.94] | 0.50 [0.30, 0.69] |
| Annotator 3 | 1.00 [1.00, 1.00] | 0.95 [0.82, 1.00] | 0.89 [0.73, 1.00] |
| GPT-6.1 Sol | 0.96 [0.87, 1.00] | 0.94 [0.83, 1.00] | 0.39 [0.19, 0.61] |
| GPT-5.6 Luna | 0.62 [0.43, 0.81] | 0.94 [0.82, 1.00] | 0.44 [0.22, 0.66] |
| Pool | Stratum (GPT-5.6 Luna label) | Compared | Annotated | Same input | Weight |
|---|---|---|---|---|---|
| DB: order, new sample and Terra | different parties | 7 | 1 | 1 | 7.0 |
| different terms | 358 | 6 | 5 | 59.7 | |
| conflicting assignments | 63 | 3 | 1 | 21.0 | |
| nobody assigned | 1,086 | 3 | 1 | 362.0 | |
| added assignment | 104 | 3 | 3 | 34.7 | |
| same assignments | 62 | 4 | 4 | 15.5 |
| Reference | Estimate | Interval |
|---|---|---|
| Annotator 1 | 69 | 47–89 |
| Annotator 2 | 62 | 37–80 |
| Annotator 3 | 77 | 57–98 |
| Majority | 71 | 49–91 |
| Prompt | Annotated pairs | Propose an assignment | Differing assignments |
|---|---|---|---|
| Agreement | 1 | 0 | 0 |
| Agreement + concrete | 13 | 7 | 4 |
| Differences | 2 | 0 | 0 |
| Differences + concrete | 3 | 0 | 0 |
| Legislative compromise | 17 | 16 | 13 |
| Irreducible conflict | 5 | 0 | 0 |
| Temperature | Mixed, order (%) | Mixed, new sample (%) | Ratio, mixed | , order ( ) | , new sample ( ) | Ratio, |
|---|---|---|---|---|---|---|
| 0 | 16.1 | 18.9 | 0.85 (0.76–1.00) | 0.43 (36) | 0.29 (35) | 0.80 (0.73–0.88) |
| 0.7 | 17.2 | 21.7 | 0.79 (0.68–0.91) | 0.37 (33) | 0.32 (30) | 0.92 (0.79–1.08) |
| Both | 16.7 | 20.3 | 0.82 (0.74–0.93) | 0.40 (69) | 0.30 (65) | 0.85 (0.79–0.93) |
| Data | Configuration | None assign | Some assign | All five: Jaccard [95% CI] ( ) |
|---|---|---|---|---|
| DB | Seed ( 0.7) | 1.00 (117) | 0.29 (32) | 0.42 [0.30, 0.53] (31) |
| DB | Order ( 0) | 1.00 (115) | 0.37 (29) | 0.43 [0.32, 0.52] (36) |
| DB | Order ( 0.7) | 1.00 (116) | 0.35 (31) | 0.37 [0.27, 0.46] (33) |
| DB | Resample ( 0) | 1.00 (111) | 0.36 (34) | 0.29 [0.21, 0.36] (35) |
| DB | Resample ( 0.7) | 1.00 (111) | 0.33 (39) | 0.32 [0.18, 0.42] (30) |
| DB, GPT-5.6 Terra | Seed ( 0.7) | 1.00 (27) | 0.45 (12) | 0.27 [0.16, 0.37] (21) |
| Data | Rerun type | Pairs | Eligible | Any change (%) | Differing (%) | Conflicting (%) | Differing, all (%) |
|---|---|---|---|---|---|---|---|
| DB | Seed ( 0.7) | 360 | 148 | 97 | 78 [71, 84] | 5 | 32 |
| DB | Order ( 0) | 360 | 135 | 98 | 83 [76, 89] | 7 | 31 |
| DB | Order ( 0.7) | 360 | 142 | 98 | 77 [72, 84] | 6 | 30 |
| DB | Resample ( 0) | 360 | 141 | 97 | 80 [73, 88] | 8 | 31 |
| DB | Resample ( 0.7) | 360 | 143 | 96 | 79 [74, 83] | 8 | 31 |
| DB, GPT-5.6 Terra | Seed ( 0.7) | 120 | 80 | 98 | 88 [79, 96] | 1 | 58 |
| Data | Prompt | Outputs | Assigned (%) | Sets | Mixed | Pairs | Eligible | Differing |
|---|---|---|---|---|---|---|---|---|
| DB | Agreement | 550 | 18.7 | 110 | 40 | 220 | 46 | 21 |
| DB | Agreement + concrete | 550 | 65.8 | 110 | 37 | 220 | 181 | 162 |
| DB | Differences | 550 | 0.0 | 110 | 0 | 220 | 0 | 0 |
| DB | Differences + concrete | 550 | 0.5 | 110 | 2 | 220 | 12 | 9 |
| DB | Legislative compromise | 550 | 91.1 | 110 | 27 | 220 | 220 | 220 |
| DB | Irreducible conflict | 550 | 3.5 | 110 | 13 | 220 | 123 | 60 |
| Data | Mediators | Configuration | Groups | Sol | Luna |
|---|---|---|---|---|---|
| DB | Three open | Order (T=0) | 36 | 0.43 (0.31–0.52) | 0.38 (0.27–0.48) |
| DB | Three open | Order (T=0.7) | 32 | 0.37 (0.26–0.46) | 0.36 (0.24–0.47) |
| DB | Three open | Seed (T=0.7) | 30 | 0.42 (0.29–0.53) | 0.39 (0.25–0.49) |
| DB | Terra | Order (T=0) | 17 | 0.33 (0.19–0.47) | 0.26 (0.13–0.41) |
| DB | Terra | Seed (T=0.7) | 21 | 0.27 (0.16–0.37) | 0.27 (0.16–0.38) |
| Polis | Three open | Order (T=0) | 35 | 0.32 (0.18–0.48) | 0.30 (0.16–0.46) |
| Judge | Anyone assigned | Differing assignments | DB decoys | Polis decoys |
|---|---|---|---|---|
| Sol | 57/58 | 31/46 | 1/4,000 | 4/4,000 |
| Luna | 47/58 | 35/46 | — | — |
| Haiku | 57/58 | 34/46 | 0/4,000 | 17/4,000 |
| Terra minus open mediators | Sol | Luna |
|---|---|---|
| DeliberationBank | −0.029 [−0.068, 0.010] | −0.032 [−0.090, 0.023] |
| Polis | −0.019 [−0.067, 0.038] | −0.031 [−0.086, 0.029] |
| Data | Comparison | All pairs | Differing/all (%) | Eligible pairs | Differing/eligible (%) |
|---|---|---|---|---|---|
| DB | Two random candidates | 180 | 57 | 106 | 97 |
| DB | Winners, Llama 3.1 ranks | 90 | 57 | 52 | 98 |
| DB | Winners, Phi-4 ranks | 90 | 51 | 52 | 88 |
| DB | Two merged statements | 90 | 57 | 55 | 93 |
| Polis | Two random candidates | 180 | 63 | 117 | 97 |
| Polis | Winners, Llama 3.1 ranks | 90 | 64 | 58 | 100 |
| Data | Ranker | Parsed calls | First position | Paired pools | Consistency |
|---|---|---|---|---|---|
| DB | Llama 3.1 8B | 360 | 51 (46–56) | 180 | 32 (27–37) |
| DB | Phi-4 14B | 358 | 34 (29–40) | 178 | 47 (41–54) |
| Polis | Llama 3.1 8B | 360 | 57 (55–60) | 180 | 31 (24–37) |
| Polis | Phi-4 14B | 359 | 64 (58–70) | 179 | 40 (35–45) |
| Sample | Judge | Statements | Agreement gap | Differences gap | Contrast (95% CI) | LOO range | |
|---|---|---|---|---|---|---|---|
| DB sample 1 | Sol | 9 | 800 | 34.5 | 3.4 | 37.9 (19.7–55.5) | 32.4–43.4 |
| DB sample 2 | Sol | 9 | 160 | 22.0 | 6.0 | 28.0 (7.8–51.3) | 20.7–32.7 |
| DB sample 3 | Sol | 8 | 800 | 28.4 | 9.9 | 38.3 (14.6–57.5) | 33.2–48.3 |
| Polis | Sol | 9 | 800 | 20.1 | 1.7 | 21.8 (14.8–29.7) | 19.0–23.3 |
| DB sample 1 | Luna | 9 | 800 | 40.9 | 3.0 | 37.9 (22.2–53.7) | 32.9–42.8 |
| DB sample 2 | Luna | 9 | 160 | 23.0 | 6.0 | 29.0 (15.8–44.9) | 23.0–32.4 |
| Sample | Objective | Four prompts | Six prompts |
|---|---|---|---|
| DB sample 1 | Find agreement | +26.1 (+8.6–+44.3) | +20.5 (+4.0–+37.7) |
| DB sample 1 | Set out differences | 5.0 ( 17.2–+8.5) | 2.0 ( 13.6–+10.9) |
| Polis | Find agreement | +21.4 (+16.4–+25.9) | +24.9 (+18.3–+30.8) |
| Polis | Set out differences | 1.2 ( 11.7–+8.2) | +0.0 ( 9.8–+7.9) |
| DB sample 2 | Find agreement | +21.6 (+8.3–+34.4) | +19.7 (+9.0–+30.9) |
| DB sample 2 | Set out differences | 3.8 ( 18.5–+8.2) | 2.4 ( 16.0–+9.2) |
| Sample | LLM judge | OR (95% CI) | , bias-reduced | Same direction | |
|---|---|---|---|---|---|
| DB sample 2 | Sol | 0.34 (0.15–0.75) | 0.0038 | 0.0101 | 4 of 4 |
| DB sample 3 | Sol | 0.21 (0.08–0.56) | 0.0009 | 0.0030 | 4 of 4 |
| DB sample 2 | Luna | 0.27 (0.11–0.66) | 0.0021 | 0.0082 | 4 of 4 |
| DB sample 3 | Luna | 0.22 (0.12–0.40) | 4 of 4 |
| Data | Arm | Higher support | Minority | Gap | Words | Assigning (%) | Decoys (%) |
|---|---|---|---|---|---|---|---|
| DeliberationBank | Baseline | 61 | 48 | 12.6 | 88 | 29 | 0.07 |
| List of views | 59 | 48 | 11.3 | 91 | 31 | 0.07 | |
| List + instruction | 67 | 54 | 12.6 | 114 | 33 | 0.07 | |
| Instruction | 69 | 53 | 16.6 | 112 | 34 | 0.07 | |
| Polis | Baseline | 29 | 19 | 10.4 | 99 | 37 | 0.40 |
| List of views | 29 | 18 | 11.0 | 102 | 35 | 0.20 |
LLM Decision-Making
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
See all 279 papers