When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
Authors: Yupeng Tang, Mingfeng Lin
Organizations: College of Computing, Georgia Institute of Technology, Atlanta, GA 30332 · Scheller College of Business, Georgia Institute of Technology, Atlanta, GA 30308
When language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than a/(1−a) times as often as correct ones, where a is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol.
Figures & tables
Figure 1: Main-audit design. One initial answer is shared across neutral reflection, anonymous disagreement, and expert-labeled disagreement. The diagram’s P is the reversal effect PR ; the accuracy effect PAcc compares correctness between the expert and anonymous branches (Section 3 ).
Domain
Source
Rule
Items
Medicine
PubMedQA
Drop Maybe
890
Science
SciFact
Binary labels
299
Law
ContractNLI
Drop Neutral
800
Total
1,989
Table 1: Benchmark composition. Counts are unique evidence-question items.
Figure 2: Reversal direction under expert disagreement ( A3 ). Leftward orange bars show Correct → Wrong; rightward green bars show Wrong → Correct. Both use all valid A3 trials as the denominator; their difference gives Δ accuracy (pp). The lower group uses the 250-item pool and a different prompt protocol (Table 26 ), so the two groups should not be compared directly.
Figure 3: Selectivity of revision relative to break-even. For each model, b/h is the reversal rate of initially wrong answers divided by that of initially correct answers, and a/(1−a) is the break-even value implied by its initial accuracy (Section 3.2 ). Revision raises accuracy only to the right of 1. Arrows run from anonymous to expert-labeled disagreement; Gemini-3.6-Flash reversed none of its initial errors under anonymous disagreement, so only its expert point is shown. The shaded group uses the 250-item pool and a different prompt protocol.
Figure 4: Expert-minus-anonymous effects on reversal ( PR , left) and accuracy ( PAcc , right) on the same paired items within each model, with 95% paired item-bootstrap intervals. The shaded group uses 250 control-pool items and a different prompt protocol.
Model
N
PR [95% CI]
PAcc [95% CI]
GPT-4o-mini
249
+4.82 [ +1.61 , +8.43 ]
−4.82 [ −8.43 , −1.61 ]
Claude-Sonnet-4
249
+5.62 [ +2.01 , +9.24 ]
−3.21 [ −6.83 , +0.40 ]
Gemini-2.5-Flash
249
+4.42 [ +0.80 , +8.03 ]
−2.81 [ −6.43 , +0.80 ]
GPT-5.6-terra
249
+2.01 [ −0.40 , +4.42 ]
−1.20 [ −4.02 , +1.20 ]
Grok-4.5
249
+1.20 [ −1.20 , +4.02 ]
−1.20 [ −4.02 , +1.20 ]
Gemini-3.6-Flash
207
+2.42 [ +0.00 , +4.83 ]
−1.45 [ −3.86 , +0.97 ]
Table 2: No-padding identical-rationale contrasts (pp): expert minus anonymous, with 95% paired item-bootstrap intervals. The two branches of each pair were collected together; rows below the rule come from the separate run of the four newer models. Coverage and document-cluster sensitivity are in Appendix H .
Model
Expert, may be wrong
Expert
Expert + track record
Claude-Sonnet-4
1.4
7.8
8.8
GPT-4o-mini
2.2
7.4
8.2
Gemini-2.5-Flash
4.4
4.6
9.0
GPT-5.6-terra
9.2
13.6
15.2
Grok-4.5
4.0
8.0
12.0
Table 3: Reversal rates (%) in the padded control suite at three asserted levels of source reliability. Evidence and the opposing answer are fixed; Table 21 adds the anonymous reference.
Field
Recorded result
Endpoint / collection
GPT-4o-mini / July 2026
Protocol / coverage
No padding, identical rationale; affected inputs excluded
Paired items / initial accuracy
244 / 91.8%
Expert wrong-revision / correction rates
27.7% / 70.0%
PR / PAcc (pp)
+4.92 / −4.92
PAcc [95% item CI] (pp)
−4.92 [ −8.61 , −1.64 ]
Table 4: Example audit card: GPT-4o-mini on the no-padding identical-rationale suite, after dropping items with incomplete excerpts (244 of 249 paired items kept). Estimates are specific to this model and protocol.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Logging and auxiliary diagnostics for the main two-pass audit. The initial answer is reused; generated explanations are logged but not carried between passes. Dashed paths denote auxiliary logs and teacher-forced scores, available for the local open-weight runs only. In the diagram, P(Yes/No) stands for the decision-string scores s of Equation 15 (Appendix D ).
Model
Odds ratio [95% CI]
Risk ratio [95% CI]
A3 -only / A1 -only
GPT-4o-mini
1.74 [1.57, 1.93]
1.60 [1.47, 1.76]
154 / 17
Claude-Sonnet-4
1.16 [1.08, 1.25]
1.12 [1.06, 1.19]
112 / 60
Gemini-2.0-Flash
0.84 [0.75, 0.94]
0.86 [0.78, 0.95]
56 / 93
Llama-3.1-8B
1.38 [1.22, 1.55]
1.29 [1.17, 1.42]
247 / 143
Mistral-7B
1.47 [1.35, 1.61]
1.38 [1.29, 1.50]
117 / 18
GPT-5.6-terra
2.16 [1.57, 3.34]
2.00 [1.50, 3.00]
17 / 0
Appendix
Table 5: Relative effects of the expert label: odds and risk ratios for A3 versus A1 , with 50,000 paired item resamples. These ratios include the one Llama pair with mismatched initial labels; the additive estimates in Table 32 exclude it.
Model
Pass-1 accuracy
Δ acc. A3
Initially correct A1→A3
Initially wrong A1→A3
S3
GPT-4o-mini
0.870
−11.0
10.0→16.9
21.6→28.2
1.67
Claude-Sonnet-4
0.853
−16.1
22.0→23.3
16.1→26.1
1.12
Gemini-2.0-Flash
0.897
−6.8
12.6→10.2
20.2→22.6
2.21
Llama-3.1-8B
0.830
−2.3
10.1→15.5
57.7→62.0
4.01
Mistral-7B
0.877
−1.9
8.0→11.4
48.8→65.6
5.75
GPT-5.6-terra
0.892
−8.8
5.8→12.6
14.8→22.2
1.77
Appendix
Table 6: Accuracy and selectivity under expert disagreement (the rates behind Figure 3 ). Pass-1 accuracy is a proportion; Δ acc. is in percentage points; reversal rates are percentages. S3=Pr(R∣C1=0,A3)/Pr(R∣C1=1,A3) ; larger means more selective revision of initial errors. Rates use valid trials per condition. Rows below the rule use the 250-item pool (Table 26 ).
Model
R∣C1=1
R∣C1=0
Pass-1 acc.
A3 harmful
GPT-4o-mini
2.8/10.0/16.9
7.7/21.6/28.2
87.0
80.0
Claude-Sonnet-4
5.7/22.0/23.3
5.2/16.1/26.1
85.3
83.9
Gemini-2.0-Flash
5.1/12.6/10.2
9.4/20.2/22.6
89.7
79.8
Llama-3.1-8B
7.8/10.1/15.5
39.2/57.7/62.0
83.0
54.8
Mistral-7B
2.7/8.0/11.4
16.7/48.8/65.6
87.7
55.3
GPT-5.6-terra
3.6/5.8/12.6
22.2/14.8/22.2
89.2
82.4
Appendix
Table 7: Reversal rates (%) conditional on Pass-1 correctness. Each cell is A0/A1/A3 . The final columns give Pass-1 accuracy and the harmful fraction among A3 reversals.
Model
N
Correct → Wrong
Wrong → Correct
Harmful (%)
GPT-4o-mini
1986
292
73
80.0
Claude-Sonnet-4
1984
395
76
83.9
Gemini-2.0-Flash
1987
182
46
79.8
Llama-3.1-8B
1989
255
210
54.8
Mistral-7B
1980
198
160
55.3
GPT-5.6-terra
250
28
6
82.4
Appendix
Table 8: Expert-disagreement transition counts on all valid A3 trials.
Model
Initial Yes
Initial No
True Yes
True No
GPT-4o-mini
+9.88 ( 3.1×10−26 )
+2.20 ( 9.1×10−4 )
+9.37 ( 3.5×10−27 )
+1.31 (0.021)
Claude-Sonnet-4
+1.50 (0.15)
+4.13 ( 1.8×10−7 )
+3.06 ( 9.7×10−4 )
+1.64 (0.021)
Gemini-2.0-Flash
−4.09 ( 4.7×10−7 )
+2.81 (0.001)
−2.47 (0.002)
−0.49 (0.73)
Llama-3.1-8B
+4.16 ( 1.5×10−4 )
+8.71 ( 2.7×10−4 )
+4.93 ( 1.3×10−5 )
+5.91 (0.004)
Mistral-7B
+2.05 ( 3.1×10−4 )
+11.66 ( 1×10−18 )
+3.27 ( 1×10−7 )
+8.96 ( 2.6×10−13 )
GPT-5.6-terra
+10.48 ( 2.4×10−4 )
+3.17 (0.12)
+9.09 ( 9.8×10−4 )
+4.65 (0.031)
Appendix
Table 9: Paired prestige effects by initial answer and true label. Cells give P in percentage points and exact McNemar p ; tests are exploratory and unadjusted.
Model
Stratum
A0
A1
A3
GPT-4o-mini
Initial Yes
5.0 (1215)
16.8 (1214)
26.7 (1214)
GPT-4o-mini
Initial No
0.9 (773)
3.1 (772)
5.3 (772)
GPT-4o-mini
True Yes
3.8 (1378)
13.7 (1377)
23.0 (1377)
GPT-4o-mini
True No
2.5 (610)
6.6 (609)
7.9 (609)
Claude-Sonnet-4
Initial Yes
9.2 (1139)
33.0 (1138)
34.5 (1137)
Claude-Sonnet-4
Initial No
0.7 (846)
5.2 (848)
9.3 (847)
Appendix
Table 10: Reversal percentages by initial answer and true label, with valid N in parentheses. Initial-Yes reversal is Yes → No; initial-No reversal is No → Yes.
Model
Items
A0
A1
A3
P=A3−A1
GPT-4o-mini
1,986
3.4
11.5
18.4
+6.9
Claude-Sonnet-4
1,982
5.6
21.1
23.7
+2.6
Gemini-2.0-Flash
1,983
5.5
13.4
11.5
−1.9
Llama-3.1-8B
1,987
13.1
18.1
23.4
+5.2
Mistral-7B
1,977
4.4
13.0
18.0
+5.0
GPT-5.6-terra
250
5.6
6.8
13.6
+6.8
Appendix
Table 11: Reversal rates (%) on the items with valid outputs under all three conditions, computed before the stricter initial-label check of Appendix H .
Domain
N
Pass-1 acc.
Pass-2 acc.
Reversal (%)
Medicine
4,435
0.873
0.793
22.7
Science
1,495
0.860
0.828
8.8
Law
3,996
0.859
0.770
18.7
Appendix
Table 12: Domain-stratified results under A3 , pooled over valid de-duplicated trials from the five full-benchmark models.
Domain
N
Pass-1 acc.
Pass-2 acc.
Reversal (%)
Medicine
426
0.904
0.873
5.4
Science
157
0.898
0.854
8.3
Law
392
0.885
0.837
9.4
Appendix
Table 13: Domain-stratified A3 results for GPT-5.6-terra, Grok-4.5, Gemini-3.6-Flash, and Claude-Opus-4.8. These records use the 250-item pool, not the original full benchmark.
Model
N
A1 (%)
A3 (%)
P (pp)
p
Claude-Sonnet-4
250
7.6
9.6
+2.0
0.062
Gemini-2.5-Flash
250
1.6
2.4
+0.8
0.73
GPT-4o-mini
250
3.6
6.4
+2.8
0.016
Appendix
Table 14: July API temperature-zero check within the padded control suite: A1/A3 use the same paired items.
Table 16: No-padding run on September 26, 2026. N is the eligible stored-Pass-1 item count; the next three columns are parseable outputs. Invalid and length-stop counts cover all three conditions. A length stop does not imply an invalid leading answer; parseable leading Yes/No answers are retained. No automatic retries were made.
Model
Expert answer
Expert + rationale
Difference ( p )
Δ accuracy
Llama-3.3-70B
7.9
43.4
+35.6 ( 3×10−46 )
−28.4
GPT-4o-mini
7.4
33.5
+26.1 ( 9×10−35 )
−20.3
Claude-Sonnet-4
7.8
32.1
+24.3 ( 7×10−30 )
−20.5
Llama-3.1-8B
27.8
45.4
+17.9 ( 6×10−10 )
−10.0
Gemini-2.5-Flash
4.6
17.3
+12.7 ( 1×10−14 )
−9.6
GPT-5.6-terra
13.6
10.4
−3.2 (0.077)
−7.2
Appendix
Table 17: Padded rationale comparison. Filler length changes between conditions. Rates are reversal percentages; differences use paired intersections and can differ from the rounded marginal-rate subtraction. Accuracy change is relative to Pass 1 under expert-plus-rationale.
Model
Target
N
No gate
Gate
Δ (pp)
p
Claude-Sonnet-4
Matching
26
57.7
61.5
+3.8
1.00
Claude-Sonnet-4
Conflicting
223
24.7
20.2
−4.5
0.013
Gemini-2.5-Flash
Matching
33
33.3
27.3
−6.1
0.62
Gemini-2.5-Flash
Conflicting
216
15.7
16.2
+0.5
1.00
GPT-4o-mini
Matching
21
66.7
66.7
+0.0
1.00
GPT-4o-mini
Conflicting
228
28.1
24.6
−3.5
0.077
Appendix
Table 18: No-padding gate control. Target matching the benchmark means Pass 1 was wrong; a conflicting target means it was correct. Rates are reversal percentages; p is paired exact McNemar.
Model
Contrast
N
Effect [95% CI] (pp)
p
GPT-4o-mini
Source label
249
+4.82 [ +1.61 , +8.43 ]
0.012
GPT-4o-mini
Gate
249
−3.21 [ −6.43 , +0.00 ]
0.077
Claude-Sonnet-4
Source label
249
+5.62 [ +2.01 , +9.24 ]
0.004
Claude-Sonnet-4
Gate
249
−3.61 [ −6.83 , −0.40 ]
0.049
Gemini-2.5-Flash
Source label
249
+4.42 [ +0.80 , +8.03 ]
0.027
Gemini-2.5-Flash
Gate
249
−0.40 [ −4.02 , +3.21 ]
1.00
Appendix
Table 19: No-padding paired contrasts. Source-label effects compare expert versus anonymous sources with identical rationales; gate effects compare gated versus ungated expert sources. CIs use 50,000 paired item resamples. Negative gate effects mean fewer reversals, not necessarily greater accuracy.
Figure 6: No-padding rationale comparisons across seven models. Points show paired reversal differences and 95% item-bootstrap intervals (50,000 resamples); sample sizes and exact McNemar tests are in Table 19 . Anonymous and expert sources supply the same rationale. The separator marks the two collection periods.
Model
Context
N
No gate
Gate
Δ (pp)
p
GPT-5.6-terra
Answer only
250
13.6
10.8
−2.80
0.14
GPT-5.6-terra
Rationale
249
10.4
10.8
+0.40
1.00
Grok-4.5
Answer only
250
8.0
4.0
−4.00
0.006
Grok-4.5
Rationale
249
12.0
10.4
−1.61
0.39
Gemini-3.6-Flash
Answer only
213
5.2
2.4
−2.82
0.070
Gemini-3.6-Flash
Rationale
190
5.8
8.4
+2.63
0.18
Appendix
Table 20: Evidence-gate comparisons for the four additional models. Rates are paired reversal percentages; differences are percentage points and p is exact McNemar. Padding changes with the gate, so these are prompt-package comparisons, not isolated gate effects.
Model
Expert, may be wrong
Anonymous
Expert
Expert + track record
Claude-Sonnet-4
1.4
7.4
7.8
8.8
GPT-4o-mini
2.2
3.0
7.4
8.2
Gemini-2.5-Flash
4.4
4.8
4.6
9.0
GPT-5.6-terra
9.2
6.8
13.6
15.2
Grok-4.5
4.0
5.2
8.0
12.0
Appendix
Table 21: Reversal rates (%) across asserted source-reliability levels. Evidence and the opposing answer are fixed.
Model
N
Disagreement
Agreement
Interaction [95% CI]
Claude-Sonnet-4
500
+0.40
−0.60
+1.00 [ −1.00 , +3.00 ]
Gemini-2.5-Flash
500
−0.20
+0.20
−0.40 [ −3.00 , +2.20 ]
GPT-4o-mini
500
+4.40
+0.00
+4.40 [ +2.20 , +6.80 ]
Llama-3.1-8B
498
+8.03
−10.24
+18.27 [ +11.04 , +25.30 ]
Llama-3.3-70B
495
+2.42
−0.20
+2.63 [ +0.81 , +4.44 ]
Claude-Opus-4.8
249
−0.80
−2.41
+1.61 [ −1.20 , +4.42 ]
Appendix
Table 22: Expert-label effects under disagreement and agreement and their interaction (pp). All columns in a row use the same four-condition intersection; CIs use 50,000 paired item resamples. Padded suite.
Model
N
A4
A3
A3−A4
p
Claude-Sonnet-4
500
6.2
7.8
+1.6
0.096
Gemini-2.5-Flash
500
4.0
4.6
+0.6
0.66
GPT-4o-mini
500
4.4
7.4
+3.0
7.3×10−4
Claude-Opus-4.8
249
2.8
2.8
+0.0
1.00
Gemini-3.6-Flash
218
2.8
5.1
+2.3
0.23
GPT-5.6-terra
250
10.0
13.6
+3.6
0.049
Appendix
Table 23: Expert versus neutral source metadata on each pairwise intersection. Rates are percentages; effects are percentage points. Padded suite.
Figure 7: Expert-label and accuracy effects on the 250-item pool. (a) Paired expert-minus-anonymous reversal. (b) Pass-2 minus Pass-1 accuracy under expert disagreement. Points are percentage-point changes; whiskers are 95% paired item-bootstrap intervals (50,000 resamples). Each panel reports its valid n . These contrasts use the padded protocol.
Model identifier
A0
A1
A3
P ( p )
Δ accuracy ( p )
openai/gpt-5.6-terra
5.6
6.8
13.6
+6.80 ( 2×10−5 )
−8.8 ( 2×10−4 )
x-ai/grok-4.5
1.6
5.2
8.0
+2.80 (0.092)
−4.8 (0.012)
google/gemini-3.6-flash
3.1
2.7
5.3
+2.35 (0.18)
−2.7 (0.15)
anthropic/claude-opus-4.8
2.8
3.6
2.8
−0.80 (0.63)
+0.4 (1.00)
Appendix
Table 24: Four-model results on the 250-item pool. A0/A1/A3 are marginal reversal percentages; P uses the paired A1/A3 intersection. Accuracy change is Pass 2 minus Pass 1 under A3 . Exact retention and paired rates appear in Table 25 .
Model
N0/N1/N3
N13
A1
A3
P
Δ acc.
Claude-Opus-4.8
249/249/249
249
3.6
2.8
−0.80
+0.4
Gemini-3.6-Flash
224/221/226
213
2.8
5.2
+2.35
−2.7
GPT-5.6-terra
250/250/250
250
6.8
13.6
+6.80
−8.8
Grok-4.5
250/250/250
250
5.2
8.0
+2.80
−4.8
Appendix
Table 25: Retention for the four additional models out of 250 attempted items each. P and reversal rates use the A1/A3 intersection; Δ accuracy uses all valid A3 outputs.
Model
A0
A1
A3
Dups.
GPT-4o-mini
1,988
1,986
1,986
666
Claude-Sonnet-4
1,985
1,986
1,984
0
Gemini-2.0-Flash
1,986
1,984
1,987
0
Llama-3.1-8B
1,988
1,988
1,989
93
Mistral-7B
1,981
1,980
1,980
0
GPT-5.6-terra
250
250
250
0
Appendix
Table 26: Valid unique items per model-condition. The first five models were evaluated on 1,989 items each; the remaining four on 250 items each with a different prompt protocol. Dups. counts removed valid duplicate records.
Asset
Use in this work
License / terms noted by source
PubMedQA
Medicine items after dropping Maybe labels
MIT license in the public dataset distribution; original dataset paper cited.
SciFact
Scientific claim-verification items after dropping neutral/insufficient-evidence labels
CC BY-NC 2.0 according to the dataset card and license metadata; original paper cited.
ContractNLI
Contract-reasoning items after dropping Neutral labels
Creative Commons Attribution 4.0 International according to the dataset site and included license file; original paper cited.
Llama-3.1-8B- and Llama-3.3-70B-Instruct
Open-weight behavioral evaluation; preference scoring for 3.1-8B
Meta Llama Community License and Acceptable Use Policy.
Mistral-7B-Instruct-v0.3
Open-weight behavioral evaluation and preference scoring
Apache 2.0 according to the Mistral model card.
GPT, Claude, Gemini, and Grok APIs
Proprietary-model behavioral evaluation
Accessed through provider/API terms; no weights or provider-owned model artifacts are redistributed.
Appendix
Table 27: Existing assets used in the audit and the terms under which they are used.
Model
Mismatch
Mean Δℓ
Llama-3.1-8B
63.2%
−1.10
Mistral-7B
26.8%
−1.16
Appendix
Table 28: Open-weight auxiliary diagnostics under expert-labeled disagreement ( A3 ). “Mismatch” is the share of reversals without a corresponding sign change in the teacher-forced preference score.
Model
Acc A1/A2/A3
Rev. A1/A2/A3
Harm. A1/A2/A3
GPT-4o-mini
0.792 / 0.826 / 0.718
10.4 / 7.0 / 18.6
78.8 / 68.6 / 86.0
Gemini-2.0-Flash
0.842 / 0.836 / 0.822
8.6 / 8.2 / 10.0
69.8 / 73.2 / 76.0
Claude-Sonnet-4
0.728 / 0.824 / 0.716
19.3 / 8.1 / 21.9
79.7 / 60.0 / 79.1
Appendix
Table 29: Paired low-prestige control subset. A2 uses an assistant with no special expertise; A1 is anonymous disagreement; A3 is expert-labeled disagreement on the same paired item pool.
Model
Acc A1/A2
Rev. A1/A2
Harm. A1/A2
GPT-4o-mini
0.784 / 0.830
10.5 / 5.4
85.4 / 76.2
Gemini-2.0-Flash
0.838 / 0.838
6.9 / 6.9
80.8 / 80.8
Claude-Sonnet-4
0.719 / 0.821
17.0 / 6.0
85.0 / 64.3
Appendix
Table 30: Temperature-0 robustness on the paired low-prestige control subset.
Cohort
Pool
Filler
Pass 1 / Pass 2
Main (Jan., reported)
1,989
Active
Same-run (reported)
Proprietary/open controls (July, reported)
500
Active
Stored initial answers
Four-model / T=0 (July, reported)
250
Active
Stored initial answers
Clean three-model (July, reported)
250
None
Exact times unavailable
Clean four-model (Sept.)
250
None
July / September
Temporal GPT-4o-mini (July, reported)
224
Original
Same-run (reported)
Appendix
Table 31: Protocol registry. Sample sizes are designed pools, not valid-pair counts. Reported dates lack complete per-request timestamps. No row represents a new evaluation on evidence-complete v2 inputs.
Model
N
PR [95% CI]
PAcc [95% CI]
GPT-4o-mini
1986
+6.90 [ +5.64 , +8.16 ]
−5.19 [ −6.50 , −3.93 ]
Claude-Sonnet-4
1984
+2.62 [ +1.31 , +3.88 ]
+0.30 [ −1.01 , +1.61 ]
Gemini-2.0-Flash
1984
−1.86 [ −3.07 , −0.66 ]
+2.37 [ +1.21 , +3.58 ]
Llama-3.1-8B
1987
+5.18 [ +3.27 , +7.15 ]
−3.77 [ −5.69 , −1.81 ]
Mistral-7B
1978
+5.01 [ +3.89 , +6.17 ]
−0.86 [ −2.02 , +0.30 ]
GPT-5.6-terra
250
+6.80 [ +4.00 , +10.00 ]
−5.20 [ −8.40 , −2.00 ]
Appendix
Table 32: Paired effects of the expert label on reversal and accuracy (pp), with 95% item-bootstrap intervals (20,000 resamples), on items whose initial answers match across branches. The first five models use the main benchmark and the last four the 250-item pool. Both protocols are padded, so filler differences also enter these contrasts.
Model
Sample
N
PR [item CI]
PAcc [document CI]
GPT-4o-mini
All
1986
+6.90 [ +5.64 , +8.16 ]
−5.19 [ −6.48 , −3.93 ]
Coverage exclusion
1938
+7.07 [ +5.83 , +8.36 ]
−5.31 [ −6.62 , −4.03 ]
Medicine/Science
1188
+7.91 [ +6.23 , +9.68 ]
−5.56 [ −7.32 , −3.87 ]
Claude-Sonnet-4
All
1984
+2.62 [ +1.31 , +3.88 ]
+0.30 [ −0.89 , +1.51 ]
Coverage exclusion
1936
+2.53 [ +1.24 , +3.82 ]
+0.36 [ −0.82 , +1.59 ]
Medicine/Science
1186
−2.53 [ −4.05 , −1.01 ]
+3.54 [ +2.10 , +5.05 ]
Appendix
Table 33: Main-audit sensitivity. PR intervals resample items; PAcc intervals resample source documents within domains, with domain weights held fixed. Coverage exclusion drops the 48 Law items with incomplete excerpts before estimation; it does not certify the remaining labels.
Model
Sample
N
PR [item CI]
PAcc [document CI]
GPT-4o-mini
All
249
+4.82 [ +1.61 , +8.43 ]
−4.82 [ −8.40 , −1.56 ]
Coverage exclusion
244
+4.92 [ +1.64 , +8.61 ]
−4.92 [ −8.56 , −1.61 ]
Claude-Sonnet-4
All
249
+5.62 [ +2.01 , +9.24 ]
−3.21 [ −6.85 , +0.38 ]
Coverage exclusion
244
+5.33 [ +1.64 , +9.02 ]
−2.87 [ −6.56 , +0.80 ]
Gemini-2.5-Flash
All
249
+4.42 [ +0.80 , +8.03 ]
−2.81 [ −6.52 , +0.80 ]
Coverage exclusion
244
+4.51 [ +0.82 , +8.20 ]
−2.87 [ −6.59 , +0.80 ]
Appendix
Table 34: No-padding rationale sensitivity. Coverage exclusion drops the control-pool items with incomplete excerpts from each model's paired intersection. Item intervals and document-cluster intervals address different sampling questions.
Model
Cohort
Medicine
Science
Law
Claude-Sonnet-4
main
887/887
299/203
798/76
Gemini-2.0-Flash
main
887/887
298/203
799/76
Llama-3.1-8B
main
889/889
298/202
800/76
Mistral-7B
main
879/879
299/203
800/76
GPT-4o-mini
main
889/889
299/203
798/76
Claude-Sonnet-4
clean_api
109/109
40/39
100/89
Appendix
Table 35: Paired questions / distinct source documents by domain. PubMed IDs, SciFact document IDs, and ContractNLI document filenames define the clusters. Cohorts are not pooled.
Model
N
IR [95% CI]
IF [95% CI]
Claude-Sonnet-4
500
+1.00 [ −1.00 , +3.00 ]
−0.20 [ −2.40 , +2.00 ]
Gemini-2.5-Flash
500
−0.40 [ −3.00 , +2.20 ]
+0.00 [ −3.00 , +3.00 ]
GPT-4o-mini
500
+4.40 [ +2.20 , +6.80 ]
+4.40 [ +2.00 , +7.00 ]
Llama-3.1-8B
498
+18.27 [ +11.04 , +25.30 ]
−2.21 [ −8.23 , +3.82 ]
Llama-3.3-70B
495
+2.63 [ +1.01 , +4.44 ]
+2.22 [ +0.40 , +4.24 ]
Claude-Opus-4.8
249
+1.61 [ −1.20 , +4.42 ]
−3.21 [ −5.62 , −0.80 ]
Appendix
Table 36: Four-condition paired interactions (pp). IR=Pdisagree−Pagree is defined on reversal and IF=Pdisagree+Pagree on source following; only IF compares the same outcome under disagreement and agreement. Item-bootstrap percentile intervals; padded suite.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feedback to address them. We examined three ways experts can provide feedback for LLM codebook revision: (i) editing LLM-generated revisions driven by cross-LLM disagreement (Codebook Verifying), (ii) answering questions about LLM disagreements (Question Answering), and (iii) labeling disagreement cases with rationales (Rationale Labeling). Experiments on thousands of tutoring-session transcripts show that Rationale Labeling yielded the highest LLM-labeling accuracy (64.9%) against expert labels, outperforming the expert-revised codebook (57.8%). The best Question Answering setting also outperformed it (60.5%). Our work shows that LLMs can be used to strategically target expert attention, shortening months of codebook revision to days without sacrificing labeling performance.
Zeyu He, Zhuqian Zhou, Kirk Vanacore +2
The Pennsylvania State University, University Park, PA, USA · Cornell University, Ithaca, NY, USA
Large language models are increasingly used in multi-agent systems, where they see and respond to other agents' answers. A key risk is conformity: a model may abandon its own answer simply because others agree on a different one. Prior studies show that LLMs often revise toward a majority answer, but it remains unclear whether these revisions help correct mistakes as often as they introduce new errors. In this paper, we conduct a controlled study in which an LLM first answers a question, then sees simulated peer responses before making a final decision. We manipulate two social cues: consensus structure and authority labels assigned to peers, and measure how they influence beneficial and harmful revisions. Across four open-weight LLMs and seven QA datasets, we find that peer agreement makes it much easier to mislead initially correct models than to correct initially wrong ones. Authority labels make models more likely to choose the endorsed answer, regardless of whether it is correct. More concerningly, generic reasoning interventions such as chain-of-thought and reflection do not reliably reduce harmful revision while preserving beneficial revision. These findings suggest that multi-agent LLM systems should verify peer answers rather than simply aggregate them.
Jiaming Qu, Lucheng Fu, Yibo Hu
Amazon · Georgia Institute of Technology · Illinois Institute of Technology
An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there. We demonstrate we can estimate where a judge is ignorant rather than where experts genuinely disagree, and propose using this to direct expert labelling. Code and data are available at https://github.com/composo-ai/judge-uncertainty-decomposition.