Unanimity Without Persuasion: A Single Round of Debate Erases the Disagreement That Verification Needs

Yang Shu

cs.AIAug 10, 2026

Unanimity Without Persuasion: A Single Round of Debate Erases the Disagreement That Verification Needs

Authors: Yang Shu

Organizations: Zhejiang University, Hangzhou, China

Abstract

A debate panel can become unanimous without becoming more correct. This is dangerous for downstream safeguards: a substituted verification ballot can change only narrow-margin votes, while richer arbiters lose disagreement as a natural targeting signal. We show that one debate round can erase that resource without requiring persuasion. Tracking a heterogeneous 7-judge panel through a blind round and three debate rounds on 600 code-correctness candidates, unanimity on a fixed cohort jumps from 39.5% to 95.2% in round 1 (93.1% of the total collapse), while accuracy moves by less than one point and 96.3% of verdict flips follow the displayed peer majority. An execution-based verification ballot corrects 8 of 2,037 pre-debate candidate-substitution instances but changes zero in every later round; by round 3 every wrong decision is unanimous, erasing dissent that had flagged two-thirds of the panel's errors. Identical-cohort controls explain why: no-peer reconsideration reproduces 79.6% of the collapse, real labels without reasoning reproduce 91.5%, and random labels steer flips toward whatever they display; the full-debate condition adds 4.3 percentage points over labels only (clustered 95% CI 0.7--8.1). The one-round collapse reproduces in two additional real runs and two fake-label seeds, remains under panel sizes 3--7, and appears in MATH-500. Parse failures concentrate on contested candidates ( $p<0.001$ ), making attrition non-ignorable. The design implication is operational: verify before any second-pass evaluation or peer exposure, and never treat post-debate unanimity as independent evidence of reliability.

Figures & tables

Figure 1: Mechanism overview. Round metrics use the fixed $n{=}291$ cohort: one debate round takes unanimity from 39.5% to 95.2%, pivotal cases from 10 to 2, and execution-ballot substitution to 0/2,037 realized flips. The control callout uses the separate matched $n{=}280$ cohort (Section 5).

Attrition point	ret.	drop.	ret.	drop.	$\chi^{2}(3)$
	$n$		Pivotal share (R0)
Round 1	323	277	3.7%	31.1%	151.9
Round 2	300	300	3.7%	29.0%	135.9
Round 3	291	309	3.4%	28.5%	134.7
Round 1 (fake)	416	184	9.6%	31.5%	78.9

Table 1: Parse-failure attrition is strongly non-random: candidates dropped for unparseable judge output are far more often pivotal at round 0 than retained ones (

\chi^{2}

over the four round-0 margin bins

\{1,3,5,7\}

, df

=3

; all

p<10^{-3}

). Retained/dropped round-0 panel accuracy differs correspondingly (e.g. 92.0% vs. 70.8% at round 1).

Figure 2: Debate-induced consensus collapse on the fixed cohort ( $n{=}291$ candidates usable in all rounds). The pivotal share (orange-red) falls from 3.4% to 0.7% after one round and to zero thereafter; unanimity (green) rises from 39.5% to 95.2% in round 1, then only to 98.6% and 99.3%. Essentially the entire effect of debate on disagreement is exhausted by the first exposure to peers’ opinions (per-judge flip rates: Table 2 ).

Round	Pivotal	Unanimity	Flips	Changes	Net
				Verifier subst.
0 (blind)	10	39.5%	—	8 (8✓)	$+0.39$ pp
1	2	95.2%	11.78%	0	$0.00$
2	0	98.6%	0.74%	0	$0.00$
3	0	99.3%	0.10%	0	$0.00$

Table 2: Blind round plus three debate rounds on the fixed cohort (

n{=}291

). “Flips” is per-judge verdict changes vs. the previous round (of

291\times 7{=}2{,}037

). “Verifier subst.”: decision-changing candidate–substitution instances from splicing the execution ballot in place of each judge in turn (2,037 instances per round; the 8 round-0 changes are 2 unique candidates

\times

4 majority-side replacements, all corrections) and the net accuracy gain averaged over replacement positions.

Judge	Real flips	Fake flips
GLM-5-Turbo	0.7%	1.7%
Kimi-K2.6	2.3%	3.6%
Claude Haiku	7.3%	7.9%
DeepSeek-V4-Flash	9.2%	10.6%
Qwen3-max	13.9%	14.5%
GPT-4o-mini	24.1%	26.1%

Table 3: Per-judge round-0

\to

1 flip rates under real and fabricated peer opinions, both computed on the identical

n{=}303

candidates usable in both conditions. Rates span an order of magnitude (0.7%–26.1%) and the cross-condition ranking is almost perfectly preserved (Spearman

\rho=0.99

p<0.001

, bootstrap CI

[0.93,1.0]

). Every judge flips at least as often in the fake condition, where the i.i.d. random labels confront it with an opposing majority more often – so absolute rates are not comparable across conditions, only rankings.

Figure 3: Five conditions on the identical cohort of $n{=}280$ candidates. A no-peer second look reaches 79.6% of full debate’s unanimity jump; random labels steer flips but slow net convergence; real labels without reasoning reach 91.5%; the full-debate condition adds 4.3pp over labels only. Flip volume is nearly identical in all four intervention conditions (10.4–11.8%).

Explore similar work

Aug 7, 2026cs.AI

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

LLM judge panels are a standard evaluation tool, but prior work reports highly correlated panel errors: nine judges provide roughly the effective information of two independent ones, and aggregation closes only a small fraction of the gap. A natural remedy--a signal from a different evidence source, e.g., executing a test suite--produced no distinguishable change in the panel's effective-vote count at scale (-0.04, 95% CI [-0.10, +0.02]). Aggregate dependence and conditional decision utility are different questions. Elementary majority arithmetic fixes the affected set for single-ballot substitution: only decisions with a one-vote margin can change. The empirical question is whether panel error rates rise and useful substitutions concentrate there. They do: the entire accuracy gain concentrates on these pivotal queries, where it is large (+10.4 to +23.3 percentage points across three headline configurations), and is exactly zero elsewhere. We confirm the pattern across three code benchmarks and four panel sizes (a 9-judge extension and 56 dependent subsampling checks, gain +6.5 to +16.1 percentage points). On HumanEval+/MBPP+, a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries; signal-only remains stronger at 87.60%. Thus population-level dependence diagnostics and margin-stratified utility are complementary, and the affected-set characterization yields a call-reduction rule for any specified single-ballot substitution policy.

Yang Shu

Zhejiang University, Hangzhou, China

LLM Evaluation LLM-as-a-Judge Code Generation Evaluation Majority Voting

Ask PDF

May 26, 2026cs.CL

Debate Helps Weak Judges Reward Stronger Models

Despite theoretical promise, debate as a scalable oversight protocol has produced mixed empirical results: gains in some settings, and null effects in others, especially when the judge does not have information hidden from it. We study proposer-critic debate in a stronger-debater/weaker-judge setting on programmatically verifiable code and logic tasks. Debate helps the judge over a consultancy baseline when the critic provides a usable advantage: the critic's classification ability must exceed the judge's, and the judge must treat critic speeches as claims to verify rather than testimony to summarize. On the three of five pairings where the condition holds, proposer-critic debate's gains are statistically significant over consultancy, and these pairings are the most capable model pairings. On the two non-responder pairings in our set, debate produces null effects, and judge verification rates drop by tens of percentage points once a critic enters the transcript. In these cases the critic's binary-classification ability and the judge's are within noise of each other, and the critic's disagreement is parsed as testimony rather than a claim to check. Ablating rebuttal rounds from debate produces no measurable change in judge performance: a single independent critique recovers the bulk of debate's benefit at lower inference cost. These findings suggest a cheaper primitive for training-free scalable oversight in verifiable domains (answer, critique, judge) and a pre-deployment audit (does the critic beat the judge, and will the judge verify it?) that predicts when debate will help.

Ethan Elasky, Frank Nakasako, Naman Goyal

Palaestra Research · Independent

Scalable Oversight LLM-as-a-Judge Multi-Agent Debate Human Oversight of AI Reasoning Verification

Ask PDF

Oct 3, 2026cs.AI

When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning

Multi-agent debate can improve reasoning, yet often fails to beat simple majority voting. We argue that successful debate requires two distinct mechanisms: proposal supply must surface a correct answer, and readout must identify that answer when voting misses it. We formalize the first requirement through recoverable headroom, which measures cases where a correct proposal is available but the majority answer is wrong. For the second, we develop Latent Verification Debate (LVD), an accounting model in which candidate proposals receive answer-specific verification evidence before final generation. Controlled fixed-proposal interventions estimate this latent effect in equivalent peer-support units and show that correct evidence changes answer probabilities and generated decisions while proposal supply remains fixed. To improve proposal supply, we construct societies from neural-thicket agents using labeled and label-free coverage objectives. Across two backbones and matched-budget reasoning benchmarks, coverage-selected societies increase complementary proposal supply and improve aggregate accuracy in repeated stochastic evaluations. Round-level controls further show that interaction provides gains beyond applying the same finalizer directly to the initial proposals. These results identify proposal coverage and truth-sensitive evidence use as complementary conditions for debate to outperform voting. Code is available at https://github.com/Wang-ML-Lab/when-debate-helps.

Zihao Zhao, Tunyu Zhang, Haizhou Shi +3

University of Illinois Urbana-Champaign · Rutgers University

Multi-Agent Debate Multi-Agent Reasoning LLM Agent Verification

Ask PDF