cs.AINov 12, 2026

When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision

Authors: Yupeng Tang, Mingfeng Lin

Organizations: College of Computing, Georgia Institute of Technology, Atlanta, GA 30332 · Scheller College of Business, Georgia Institute of Technology, Atlanta, GA 30308

Abstract

When language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than a/(1−a)a/(1-a) times as often as correct ones, where aa is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol.

Figures & tables

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation

    Sep 22, 2026Zeyu He, Zhuqian Zhou, Kirk Vanacore +2LLM-Assisted AnnotationHuman-in-the-Loop Annotation

  2. Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity

    Jun 1, 2026Jiaming Qu, Lucheng Fu, Yibo HuMulti-Agent LLM SystemsNormative Conformity in Language Models