When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
Organizations: College of Computing, Georgia Institute of Technology, Atlanta, GA 30332 · Scheller College of Business, Georgia Institute of Technology, Atlanta, GA 30308
Abstract
When language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than times as often as correct ones, where is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol.
Figures & tables
| Domain | Source | Rule | Items |
|---|---|---|---|
| Medicine | PubMedQA | Drop Maybe | 890 |
| Science | SciFact | Binary labels | 299 |
| Law | ContractNLI | Drop Neutral | 800 |
| Total | 1,989 |
| Model | [95% CI] | [95% CI] | |
|---|---|---|---|
| GPT-4o-mini | 249 | [ , ] | [ , ] |
| Claude-Sonnet-4 | 249 | [ , ] | [ , ] |
| Gemini-2.5-Flash | 249 | [ , ] | [ , ] |
| GPT-5.6-terra | 249 | [ , ] | [ , ] |
| Grok-4.5 | 249 | [ , ] | [ , ] |
| Gemini-3.6-Flash | 207 | [ , ] | [ , ] |
| Model | Expert, may be wrong | Expert | Expert + track record |
|---|---|---|---|
| Claude-Sonnet-4 | 1.4 | 7.8 | 8.8 |
| GPT-4o-mini | 2.2 | 7.4 | 8.2 |
| Gemini-2.5-Flash | 4.4 | 4.6 | 9.0 |
| GPT-5.6-terra | 9.2 | 13.6 | 15.2 |
| Grok-4.5 | 4.0 | 8.0 | 12.0 |
| Field | Recorded result |
|---|---|
| Endpoint / collection | GPT-4o-mini / July 2026 |
| Protocol / coverage | No padding, identical rationale; affected inputs excluded |
| Paired items / initial accuracy | 244 / 91.8% |
| Expert wrong-revision / correction rates | 27.7% / 70.0% |
| / (pp) | / |
| [95% item CI] (pp) | [ , ] |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Odds ratio [95% CI] | Risk ratio [95% CI] | -only / -only |
|---|---|---|---|
| GPT-4o-mini | 1.74 [1.57, 1.93] | 1.60 [1.47, 1.76] | 154 / 17 |
| Claude-Sonnet-4 | 1.16 [1.08, 1.25] | 1.12 [1.06, 1.19] | 112 / 60 |
| Gemini-2.0-Flash | 0.84 [0.75, 0.94] | 0.86 [0.78, 0.95] | 56 / 93 |
| Llama-3.1-8B | 1.38 [1.22, 1.55] | 1.29 [1.17, 1.42] | 247 / 143 |
| Mistral-7B | 1.47 [1.35, 1.61] | 1.38 [1.29, 1.50] | 117 / 18 |
| GPT-5.6-terra | 2.16 [1.57, 3.34] | 2.00 [1.50, 3.00] | 17 / 0 |
| Model | Pass-1 accuracy | acc. | Initially correct | Initially wrong | |
|---|---|---|---|---|---|
| GPT-4o-mini | 0.870 | 1.67 | |||
| Claude-Sonnet-4 | 0.853 | 1.12 | |||
| Gemini-2.0-Flash | 0.897 | 2.21 | |||
| Llama-3.1-8B | 0.830 | 4.01 | |||
| Mistral-7B | 0.877 | 5.75 | |||
| GPT-5.6-terra | 0.892 | 1.77 |
| Model | Pass-1 acc. | harmful | ||
|---|---|---|---|---|
| GPT-4o-mini | 2.8/10.0/16.9 | 7.7/21.6/28.2 | 87.0 | 80.0 |
| Claude-Sonnet-4 | 5.7/22.0/23.3 | 5.2/16.1/26.1 | 85.3 | 83.9 |
| Gemini-2.0-Flash | 5.1/12.6/10.2 | 9.4/20.2/22.6 | 89.7 | 79.8 |
| Llama-3.1-8B | 7.8/10.1/15.5 | 39.2/57.7/62.0 | 83.0 | 54.8 |
| Mistral-7B | 2.7/8.0/11.4 | 16.7/48.8/65.6 | 87.7 | 55.3 |
| GPT-5.6-terra | 3.6/5.8/12.6 | 22.2/14.8/22.2 | 89.2 | 82.4 |
| Model | Correct Wrong | Wrong Correct | Harmful (%) | |
|---|---|---|---|---|
| GPT-4o-mini | 1986 | 292 | 73 | 80.0 |
| Claude-Sonnet-4 | 1984 | 395 | 76 | 83.9 |
| Gemini-2.0-Flash | 1987 | 182 | 46 | 79.8 |
| Llama-3.1-8B | 1989 | 255 | 210 | 54.8 |
| Mistral-7B | 1980 | 198 | 160 | 55.3 |
| GPT-5.6-terra | 250 | 28 | 6 | 82.4 |
| Model | Initial Yes | Initial No | True Yes | True No |
|---|---|---|---|---|
| GPT-4o-mini | ( ) | ( ) | ( ) | (0.021) |
| Claude-Sonnet-4 | (0.15) | ( ) | ( ) | (0.021) |
| Gemini-2.0-Flash | ( ) | (0.001) | (0.002) | (0.73) |
| Llama-3.1-8B | ( ) | ( ) | ( ) | (0.004) |
| Mistral-7B | ( ) | ( ) | ( ) | ( ) |
| GPT-5.6-terra | ( ) | (0.12) | ( ) | (0.031) |
| Model | Stratum | |||
|---|---|---|---|---|
| GPT-4o-mini | Initial Yes | 5.0 (1215) | 16.8 (1214) | 26.7 (1214) |
| GPT-4o-mini | Initial No | 0.9 (773) | 3.1 (772) | 5.3 (772) |
| GPT-4o-mini | True Yes | 3.8 (1378) | 13.7 (1377) | 23.0 (1377) |
| GPT-4o-mini | True No | 2.5 (610) | 6.6 (609) | 7.9 (609) |
| Claude-Sonnet-4 | Initial Yes | 9.2 (1139) | 33.0 (1138) | 34.5 (1137) |
| Claude-Sonnet-4 | Initial No | 0.7 (846) | 5.2 (848) | 9.3 (847) |
| Model | Items | ||||
| GPT-4o-mini | 1,986 | 3.4 | 11.5 | 18.4 | |
| Claude-Sonnet-4 | 1,982 | 5.6 | 21.1 | 23.7 | |
| Gemini-2.0-Flash | 1,983 | 5.5 | 13.4 | 11.5 | |
| Llama-3.1-8B | 1,987 | 13.1 | 18.1 | 23.4 | |
| Mistral-7B | 1,977 | 4.4 | 13.0 | 18.0 | |
| GPT-5.6-terra | 250 | 5.6 | 6.8 | 13.6 |
| Domain | Pass-1 acc. | Pass-2 acc. | Reversal (%) | |
|---|---|---|---|---|
| Medicine | 4,435 | 0.873 | 0.793 | 22.7 |
| Science | 1,495 | 0.860 | 0.828 | 8.8 |
| Law | 3,996 | 0.859 | 0.770 | 18.7 |
| Domain | Pass-1 acc. | Pass-2 acc. | Reversal (%) | |
|---|---|---|---|---|
| Medicine | 426 | 0.904 | 0.873 | 5.4 |
| Science | 157 | 0.898 | 0.854 | 8.3 |
| Law | 392 | 0.885 | 0.837 | 9.4 |
| Model | (%) | (%) | (pp) | ||
|---|---|---|---|---|---|
| Claude-Sonnet-4 | 250 | 7.6 | 9.6 | 0.062 | |
| Gemini-2.5-Flash | 250 | 1.6 | 2.4 | 0.73 | |
| GPT-4o-mini | 250 | 3.6 | 6.4 | 0.016 |
| (Expert) | -gated (Mitigation) | ||||
|---|---|---|---|---|---|
| Model | Domain | C W | W C | C W | W C |
| GPT-4o-mini | Medicine | 48 | 8 | 27 | 6 |
| Science | 7 | 1 | 3 | 3 | |
| Law | 25 | 4 | 11 | 4 | |
| Claude-Sonnet-4 | Medicine | 24 | 8 | 8 | 8 |
| Science | 6 | 1 | 2 | 4 | |
| Model | Gated | Invalid | Length | |||
|---|---|---|---|---|---|---|
| GPT-5.6-terra | 249 | 249 | 249 | 249 | 0 | 0 |
| Grok-4.5 | 249 | 249 | 249 | 249 | 0 | 0 |
| Gemini-3.6-Flash | 233 | 221 | 217 | 214 | 47 | 617 |
| Claude-Opus-4.8 | 248 | 248 | 248 | 248 | 0 | 0 |
| Model | Expert answer | Expert + rationale | Difference ( ) | accuracy |
|---|---|---|---|---|
| Llama-3.3-70B | 7.9 | 43.4 | ( ) | |
| GPT-4o-mini | 7.4 | 33.5 | ( ) | |
| Claude-Sonnet-4 | 7.8 | 32.1 | ( ) | |
| Llama-3.1-8B | 27.8 | 45.4 | ( ) | |
| Gemini-2.5-Flash | 4.6 | 17.3 | ( ) | |
| GPT-5.6-terra | 13.6 | 10.4 | (0.077) |
| Model | Target | No gate | Gate | (pp) | ||
|---|---|---|---|---|---|---|
| Claude-Sonnet-4 | Matching | 26 | 57.7 | 61.5 | 1.00 | |
| Claude-Sonnet-4 | Conflicting | 223 | 24.7 | 20.2 | 0.013 | |
| Gemini-2.5-Flash | Matching | 33 | 33.3 | 27.3 | 0.62 | |
| Gemini-2.5-Flash | Conflicting | 216 | 15.7 | 16.2 | 1.00 | |
| GPT-4o-mini | Matching | 21 | 66.7 | 66.7 | 1.00 | |
| GPT-4o-mini | Conflicting | 228 | 28.1 | 24.6 | 0.077 |
| Model | Contrast | Effect [95% CI] (pp) | ||
|---|---|---|---|---|
| GPT-4o-mini | Source label | 249 | [ , ] | 0.012 |
| GPT-4o-mini | Gate | 249 | [ , ] | 0.077 |
| Claude-Sonnet-4 | Source label | 249 | [ , ] | 0.004 |
| Claude-Sonnet-4 | Gate | 249 | [ , ] | 0.049 |
| Gemini-2.5-Flash | Source label | 249 | [ , ] | 0.027 |
| Gemini-2.5-Flash | Gate | 249 | [ , ] | 1.00 |
| Model | Context | No gate | Gate | (pp) | ||
|---|---|---|---|---|---|---|
| GPT-5.6-terra | Answer only | 250 | 13.6 | 10.8 | 0.14 | |
| GPT-5.6-terra | Rationale | 249 | 10.4 | 10.8 | 1.00 | |
| Grok-4.5 | Answer only | 250 | 8.0 | 4.0 | 0.006 | |
| Grok-4.5 | Rationale | 249 | 12.0 | 10.4 | 0.39 | |
| Gemini-3.6-Flash | Answer only | 213 | 5.2 | 2.4 | 0.070 | |
| Gemini-3.6-Flash | Rationale | 190 | 5.8 | 8.4 | 0.18 |
| Model | Expert, may be wrong | Anonymous | Expert | Expert + track record |
|---|---|---|---|---|
| Claude-Sonnet-4 | 1.4 | 7.4 | 7.8 | 8.8 |
| GPT-4o-mini | 2.2 | 3.0 | 7.4 | 8.2 |
| Gemini-2.5-Flash | 4.4 | 4.8 | 4.6 | 9.0 |
| GPT-5.6-terra | 9.2 | 6.8 | 13.6 | 15.2 |
| Grok-4.5 | 4.0 | 5.2 | 8.0 | 12.0 |
| Model | Disagreement | Agreement | Interaction [95% CI] | |
|---|---|---|---|---|
| Claude-Sonnet-4 | 500 | [ , ] | ||
| Gemini-2.5-Flash | 500 | [ , ] | ||
| GPT-4o-mini | 500 | [ , ] | ||
| Llama-3.1-8B | 498 | [ , ] | ||
| Llama-3.3-70B | 495 | [ , ] | ||
| Claude-Opus-4.8 | 249 | [ , ] |
| Model | |||||
|---|---|---|---|---|---|
| Claude-Sonnet-4 | 500 | 6.2 | 7.8 | 0.096 | |
| Gemini-2.5-Flash | 500 | 4.0 | 4.6 | 0.66 | |
| GPT-4o-mini | 500 | 4.4 | 7.4 | ||
| Claude-Opus-4.8 | 249 | 2.8 | 2.8 | 1.00 | |
| Gemini-3.6-Flash | 218 | 2.8 | 5.1 | 0.23 | |
| GPT-5.6-terra | 250 | 10.0 | 13.6 | 0.049 |
| Model identifier | ( ) | accuracy ( ) | |||
|---|---|---|---|---|---|
| openai/gpt-5.6-terra | 5.6 | 6.8 | 13.6 | ( ) | ( ) |
| x-ai/grok-4.5 | 1.6 | 5.2 | 8.0 | (0.092) | (0.012) |
| google/gemini-3.6-flash | 3.1 | 2.7 | 5.3 | (0.18) | (0.15) |
| anthropic/claude-opus-4.8 | 2.8 | 3.6 | 2.8 | (0.63) | (1.00) |
| Model | acc. | |||||
|---|---|---|---|---|---|---|
| Claude-Opus-4.8 | 249/249/249 | 249 | 3.6 | 2.8 | ||
| Gemini-3.6-Flash | 224/221/226 | 213 | 2.8 | 5.2 | ||
| GPT-5.6-terra | 250/250/250 | 250 | 6.8 | 13.6 | ||
| Grok-4.5 | 250/250/250 | 250 | 5.2 | 8.0 |
| Model | Dups. | |||
| GPT-4o-mini | 1,988 | 1,986 | 1,986 | 666 |
| Claude-Sonnet-4 | 1,985 | 1,986 | 1,984 | 0 |
| Gemini-2.0-Flash | 1,986 | 1,984 | 1,987 | 0 |
| Llama-3.1-8B | 1,988 | 1,988 | 1,989 | 93 |
| Mistral-7B | 1,981 | 1,980 | 1,980 | 0 |
| GPT-5.6-terra | 250 | 250 | 250 | 0 |
| Asset | Use in this work | License / terms noted by source |
|---|---|---|
| PubMedQA | Medicine items after dropping Maybe labels | MIT license in the public dataset distribution; original dataset paper cited. |
| SciFact | Scientific claim-verification items after dropping neutral/insufficient-evidence labels | CC BY-NC 2.0 according to the dataset card and license metadata; original paper cited. |
| ContractNLI | Contract-reasoning items after dropping Neutral labels | Creative Commons Attribution 4.0 International according to the dataset site and included license file; original paper cited. |
| Llama-3.1-8B- and Llama-3.3-70B-Instruct | Open-weight behavioral evaluation; preference scoring for 3.1-8B | Meta Llama Community License and Acceptable Use Policy. |
| Mistral-7B-Instruct-v0.3 | Open-weight behavioral evaluation and preference scoring | Apache 2.0 according to the Mistral model card. |
| GPT, Claude, Gemini, and Grok APIs | Proprietary-model behavioral evaluation | Accessed through provider/API terms; no weights or provider-owned model artifacts are redistributed. |
| Model | Mismatch | Mean |
|---|---|---|
| Llama-3.1-8B | 63.2% | |
| Mistral-7B | 26.8% |
| Model | Acc | Rev. | Harm. |
|---|---|---|---|
| GPT-4o-mini | 0.792 / 0.826 / 0.718 | 10.4 / 7.0 / 18.6 | 78.8 / 68.6 / 86.0 |
| Gemini-2.0-Flash | 0.842 / 0.836 / 0.822 | 8.6 / 8.2 / 10.0 | 69.8 / 73.2 / 76.0 |
| Claude-Sonnet-4 | 0.728 / 0.824 / 0.716 | 19.3 / 8.1 / 21.9 | 79.7 / 60.0 / 79.1 |
| Model | Acc | Rev. | Harm. |
|---|---|---|---|
| GPT-4o-mini | 0.784 / 0.830 | 10.5 / 5.4 | 85.4 / 76.2 |
| Gemini-2.0-Flash | 0.838 / 0.838 | 6.9 / 6.9 | 80.8 / 80.8 |
| Claude-Sonnet-4 | 0.719 / 0.821 | 17.0 / 6.0 | 85.0 / 64.3 |
| Cohort | Pool | Filler | Pass 1 / Pass 2 |
|---|---|---|---|
| Main (Jan., reported) | 1,989 | Active | Same-run (reported) |
| Proprietary/open controls (July, reported) | 500 | Active | Stored initial answers |
| Four-model / (July, reported) | 250 | Active | Stored initial answers |
| Clean three-model (July, reported) | 250 | None | Exact times unavailable |
| Clean four-model (Sept.) | 250 | None | July / September |
| Temporal GPT-4o-mini (July, reported) | 224 | Original | Same-run (reported) |
| Model | [95% CI] | [95% CI] | |
|---|---|---|---|
| GPT-4o-mini | 1986 | [ , ] | [ , ] |
| Claude-Sonnet-4 | 1984 | [ , ] | [ , ] |
| Gemini-2.0-Flash | 1984 | [ , ] | [ , ] |
| Llama-3.1-8B | 1987 | [ , ] | [ , ] |
| Mistral-7B | 1978 | [ , ] | [ , ] |
| GPT-5.6-terra | 250 | [ , ] | [ , ] |
| Model | Sample | [item CI] | [document CI] | |
|---|---|---|---|---|
| GPT-4o-mini | All | 1986 | [ , ] | [ , ] |
| Coverage exclusion | 1938 | [ , ] | [ , ] | |
| Medicine/Science | 1188 | [ , ] | [ , ] | |
| Claude-Sonnet-4 | All | 1984 | [ , ] | [ , ] |
| Coverage exclusion | 1936 | [ , ] | [ , ] | |
| Medicine/Science | 1186 | [ , ] | [ , ] |
| Model | Sample | [item CI] | [document CI] | |
|---|---|---|---|---|
| GPT-4o-mini | All | 249 | [ , ] | [ , ] |
| Coverage exclusion | 244 | [ , ] | [ , ] | |
| Claude-Sonnet-4 | All | 249 | [ , ] | [ , ] |
| Coverage exclusion | 244 | [ , ] | [ , ] | |
| Gemini-2.5-Flash | All | 249 | [ , ] | [ , ] |
| Coverage exclusion | 244 | [ , ] | [ , ] |
| Model | Cohort | Medicine | Science | Law |
|---|---|---|---|---|
| Claude-Sonnet-4 | main | 887/887 | 299/203 | 798/76 |
| Gemini-2.0-Flash | main | 887/887 | 298/203 | 799/76 |
| Llama-3.1-8B | main | 889/889 | 298/202 | 800/76 |
| Mistral-7B | main | 879/879 | 299/203 | 800/76 |
| GPT-4o-mini | main | 889/889 | 299/203 | 798/76 |
| Claude-Sonnet-4 | clean_api | 109/109 | 40/39 | 100/89 |
| Model | [95% CI] | [95% CI] | |
|---|---|---|---|
| Claude-Sonnet-4 | 500 | [ , ] | [ , ] |
| Gemini-2.5-Flash | 500 | [ , ] | [ , ] |
| GPT-4o-mini | 500 | [ , ] | [ , ] |
| Llama-3.1-8B | 498 | [ , ] | [ , ] |
| Llama-3.3-70B | 495 | [ , ] | [ , ] |
| Claude-Opus-4.8 | 249 | [ , ] | [ , ] |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 862 papers