Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines

Jingjie Ning; Xueqi Li; Chengyu Yu

cs.SEApr 1, 2026

Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines

Authors: Jingjie Ning, Xueqi Li, Chengyu Yu

Organizations: School of Computer Science, Carnegie Mellon University

Abstract

Multi-LLM revision pipelines, in which a second model reviews and improves a draft produced by a first, are widely assumed to derive their gains from genuine error correction. We question this assumption with a controlled decomposition experiment that uses four matched conditions to separate second-pass gains into three additive components: re-solving, scaffold, and content. We evaluate this design across two model pairs on three benchmarks spanning knowledge-intensive MCQ and competitive programming. Our results show that the gains of multi-LLM revision are not monolithic, but depend on task structure, draft quality, and the type of draft information. On MCQ tasks, where the answer space is constrained and drafts provide little structural guidance, most gains are consistent with stronger-model re-solving, and directly routing queries to the stronger model can be more effective than revising a weak draft. On code generation tasks, however, two-stage prompting remains useful because even semantically null drafts can provide substantial structural scaffolding, while weak draft content can be harmful. Finally, role-reversed experiments show that strong drafts clearly benefit weak reviewers. Ultimately, our findings demonstrate that the utility of multi-LLM revision is dynamically bottlenecked by task structure and draft quality, necessitating more targeted pipeline designs rather than blanket revision strategies.

Figures & tables

Figure 1: Illustration of the decomposition.

Figure 2: Signed decomposition of second-pass gains in the (weak $\rightarrow$ strong) setting into re-solving, scaffold, and content. MCQ gains are mostly re-solving, whereas LiveCodeBench is scaffold-dominated with negative content. Error bars show paired 95% CI.

		Primary							Supplementary
		Accuracy (%)				Effects (pp)			Effects (pp)
Dataset	Pair	$x_{1}$	$x_{2}$	$x_{3}$	$x_{4}$	Re-solv.	Scaff.	Content	Re-solv.	Scaff.	Content
GPQA	P1	61.6	79.3	76.3	74.7	+14.6 ∗∗∗	$-$ 1.5 ns	+4.5 ns	$-$ 14.6 ∗∗∗	$-$ 5.1 ns	+14.6 ∗∗∗
GPQA	P2	33.3	87.4	89.9	90.4	+56.6 ∗∗∗	+0.5 ns	$-$ 3.0 ns	$-$ 55.6 ∗∗∗	$-$ 3.5 ns	+26.8 ∗∗∗
HLE	P1	12.6	20.2	17.5	18.4	+4.9 ∗	+0.9 ns	+1.8 ns	$-$ 3.8 ns	$-$ 0.7 ns	+2.9 ns
HLE	P2	11.3	37.5	41.7	39.9	+30.4 ∗∗∗	$-$ 1.8 ns	$-$ 2.4 ns	$-$ 30.6 ∗∗∗	$-$ 1.1 ns	+12.0 ∗∗∗

Table 1: Accuracy and effect decomposition on MCQ tasks. The primary setting (weak

\to

strong) reports accuracy and all three effects; the supplementary setting (strong

\to

weak) reports effects only (full accuracy in Appendix Table 3 ).

{}^{***}\!p<.001

;

{}^{*}\!p<.05

; ns: not significant (two-tailed McNemar’s test with Yates correction).

Figure 3: Benefit–harm view of the real-draft content effect ( $x_{2}-x_{4}$ ). Points below the diagonal indicate net benefit and points above it net harm. MCQ lies near cancellation, whereas LiveCodeBench shifts from net harm to net benefit across settings.

	Primary								Supplementary
	Accuracy (%)				Effects (pp)				Effects (pp)
Pair	$x_{1}$	$x_{2}$	$x_{3}$	$x_{4}$	Total	Re-solv.	Scaffold	Content	Re-solv.	Scaffold	Content
Pair 1	50.6	83.9	61.1	87.0	+33.3	+10.5 ∗∗∗	+25.9 ∗∗∗	$-$ 3.1 ∗∗	$-$ 7.3 ∗∗	+9.2 ∗∗∗	+3.2 ∗
Pair 2	28.5	78.1	43.1	86.0	+49.6	+14.6 ∗∗∗	+42.9 ∗∗∗	$-$ 7.9 ∗∗∗	$-$ 15.6 ∗∗∗	$-$ 1.7 ns	+25.1 ∗∗∗

Table 2: Accuracy and effect decomposition on LiveCodeBench. The primary setting (weak

\to

strong) reports accuracy and all three decomposed effects; the supplementary setting (strong

\to

weak) reports effects only (full accuracy in Appendix Table 3 ).

{}^{***}\!p<.001

;

{}^{**}\!p<.01

;

{}^{*}\!p<.05

; ns: not significant.

Figure 4: LiveCodeBench difficulty split for Pair 1. The content effect becomes increasingly negative from easy to hard.

Figure 5: Mechanism-level decomposition of second-pass outcomes in the primary setting. On MCQ, diagnostic cases are dominated by the re-solving family; on LiveCodeBench, they are dominated by scaffold-positive cases.

Figure 6: Representative MCQ cases. Left: positive re-solving, where the reviewer abandons the weak premise and reaches the correct answer independently. Right: negative re-solving, where the second pass overrides a correct weak answer and drifts to an incorrect one.

Figure 7: Representative code cases. Left: negative content, where a draft anchors revision to a brittle artifact. Right: positive scaffold, where scaffold supports a robust solution.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	Pair	$x_{1}$	$x_{2}$	$x_{3}$	$x_{4}$
GPQA (198)	Primary	Pair 1	122 (61.6%)	157 (79.3%)	151 (76.3%)	148 (74.7%)
	Primary	Pair 2	66 (33.3%)	173 (87.4%)	178 (89.9%)	179 (90.4%)
	Supplem.	Pair 1	153 (77.3%)	143 (72.2%)	124 (62.6%)	114 (57.6%)
	Supplem.	Pair 2	178 (89.9%)	114 (57.6%)	68 (34.3%)	61 (30.8%)
HLE (451)	Primary	Pair 1	57 (12.6%)	91 (20.2%)	79 (17.5%)	83 (18.4%)
	Primary	Pair 2	51 (11.3%)	169 (37.5%)	188 (41.7%)	180 (39.9%)

Appendix

Table 3: Accuracy for all conditions. Format: correct count (percentage). Pair 1 uses Gemini Flash Lite as generator and GPT-5-mini as reviewer. Pair 2 uses GPT-4o-mini as generator and Gemini Flash as reviewer.

Dataset	Setting	Pair	Total	Re-solv.	Scaffold	Content
GPQA	Primary	Pair 1	$+17.7$	$+14.6$	$-1.5$	$+4.5$
	Primary	Pair 2	$+54.0$	$+56.6$	$+0.5$	$-3.0$
	Supplem.	Pair 1	$-5.1$	$-14.6$	$-5.1$	$+14.6$
	Supplem.	Pair 2	$-32.3$	$-55.6$	$-3.5$	$+26.8$
HLE	Primary	Pair 1	$+7.5$	$+4.9$	$+0.9$	$+1.8$
	Primary	Pair 2	$+26.2$	$+30.4$	$-1.8$	$-2.4$

Appendix

Table 4: Effect decomposition for all datasets, settings, and model pairs. All values are in percentage points. Re-solving

=x_{3}-x_{1}

; Scaffold

=x_{4}-x_{3}

; Content

=x_{2}-x_{4}

; Total

=x_{2}-x_{1}

Setting	Pair	Comparison	Acc A	Acc B	$\Delta$	$n_{\text{✓}\text{✗}}$	$n_{\text{✗}\text{✓}}$	$\chi^{2}$	$p$
Primary	Pair 1	Content ( $x_{2}-x_{4}$ )	79.3	74.7	$+4.5$	16	7	2.78	.095 ns
		Scaffold ( $x_{4}-x_{3}$ )	74.7	76.3	$-1.5$	9	12	0.19	.663 ns
		Re-solving ( $x_{3}-x_{1}$ )	76.3	61.6	$+14.6$	45	16	12.85	.0003***
	Pair 2	Content ( $x_{2}-x_{4}$ )	87.4	90.4	$-3.0$	4	10	1.79	.181 ns
		Scaffold ( $x_{4}-x_{3}$ )	90.4	89.9	$+0.5$	6	5	0.00	1.00 ns
		Re-solving ( $x_{3}-x_{1}$ )	89.9	33.3	$+56.6$	114	2	106.2	$<$ .001***

Appendix

Table 5: McNemar’s tests on GPQA (two-tailed, Yates correction).

Setting	Pair	Comparison	Acc A	Acc B	$\Delta$	$n_{\text{✓}\text{✗}}$	$n_{\text{✗}\text{✓}}$	$\chi^{2}$	$p$
Primary	Pair 1	Content ( $x_{2}-x_{4}$ )	20.2	18.4	$+1.8$	44	36	0.61	.434 ns
		Scaffold ( $x_{4}-x_{3}$ )	18.4	17.5	$+0.9$	32	28	0.15	.699 ns
		Re-solving ( $x_{3}-x_{1}$ )	17.5	12.6	$+4.9$	64	42	4.16	.041*
	Pair 2	Content ( $x_{2}-x_{4}$ )	37.5	39.9	$-2.4$	33	44	1.30	.254 ns
		Scaffold ( $x_{4}-x_{3}$ )	39.9	41.7	$-1.8$	33	41	0.66	.416 ns
		Re-solving ( $x_{3}-x_{1}$ )	41.7	11.3	$+30.4$	166	29	94.85	$<$ .001***

Appendix

Table 6: McNemar’s tests on HLE (two-tailed, Yates correction).

Setting	Pair	Comparison	Acc A	Acc B	$\Delta$	$n_{\text{✓}\text{✗}}$	$n_{\text{✗}\text{✓}}$	$\chi^{2}$	$p$
Primary	Pair 1	Content ( $x_{2}-x_{4}$ )	83.9	87.0	$-3.1$	43	76	8.61	.003**
		Scaffold ( $x_{4}-x_{3}$ )	87.0	61.1	$+25.9$	306	33	218.2	$<$ .001***
		Re-solving ( $x_{3}-x_{1}$ )	61.1	50.6	$+10.5$	325	214	22.45	$<$ .001***
	Pair 2	Content ( $x_{2}-x_{4}$ )	78.1	86.0	$-7.9$	44	127	39.32	$<$ .001***
		Scaffold ( $x_{4}-x_{3}$ )	86.0	43.1	$+42.9$	468	16	420.3	$<$ .001***
		Re-solving ( $x_{3}-x_{1}$ )	43.1	28.5	$+14.6$	258	104	64.67	$<$ .001***

Appendix

Table 7: McNemar’s tests on LiveCodeBench (two-tailed, Yates correction).

Tier	$n$	$x_{1}$	$x_{2}$	$x_{3}$	$x_{4}$	Content
Easy	322	69.3%	94.1%	74.8%	94.7%	$-$ 0.6 pp
Medium	382	47.1%	86.1%	66.8%	89.5%	$-$ 3.4 pp
Hard	350	37.1%	72.0%	42.3%	77.1%	$-$ 5.1 pp

Appendix

Table 8: Accuracy by difficulty tier on LiveCodeBench, Pair 1 primary setting. Content effect =

x_{2}-x_{4}

Pair	Null Scaffold	True-Null Scaffold	$p$
Pair 1	87.0	86.4	0.617
Pair 2	86.0	85.1	0.481

Appendix

Table 9: Scaffold ablation on LiveCodeBench. Left: comparison between the Null Scaffold and the True-Null Scaffold . Right: forward-setting accuracies and significance tests for their difference.

Dataset	Pair	$n$	✗ ✗ ✗ ✗	✗ ✗ ✗ ✓	✗ ✗ ✓ ✗	✗ ✗ ✓ ✓	✗ ✓ ✗ ✗	✗ ✓ ✗ ✓	✗ ✓ ✓ ✗	✗ ✓ ✓ ✓	✓ ✗ ✗ ✗	✓ ✗ ✗ ✓	✓ ✗ ✓ ✗	✓ ✗ ✓ ✓	✓ ✓ ✗ ✗	✓ ✓ ✗ ✓	✓ ✓ ✓ ✗	✓ ✓ ✓ ✓
GPQA	P1	198	17	1	3	4	8	5	3	35	11	1	3	1	2	2	3	99
GPQA	P2	198	10	3	3	6	2	3	1	104	2	0	0	1	0	0	1	62
HLE	P1	451	281	22	8	12	19	8	14	30	34	1	1	1	6	1	5	8
HLE	P2	451	195	18	24	23	13	8	12	107	17	0	2	3	5	7	3	14
LCB	P1	1054	77	21	10	45	15	83	16	254	5	5	2	5	7	197	5	307
LCB	P2	1054	97	104	5	21	34	261	10	222	1	1	1	1	0	102	0	194

Appendix

Table 10: Pattern counts across all

2^{4}

per-question outcome patterns for the two primary model pairs. Patterns are written in literal

x_{1}x_{2}x_{3}x_{4}

order, where ✓ denotes correct and ✗ denotes incorrect.

Multi-LLM Collaboration

Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.

See all 85 papers

Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds
Jul 31, 2026Yefan Tao, Gerald Friedland, Madhusudhanan Chandrasekaran +1LLM Self-Correction LLM Self-Refinement
Amazon
Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
Jun 1, 2026Jiaming Qu, Lucheng Fu, Yibo HuMulti-Agent LLM Systems Normative Conformity in Language Models
Amazon · Georgia Institute of Technology · Illinois Institute of Technology
REVES: REvision and VErification--Augmented Training for Test-Time Scaling
Jun 17, 2026Yuanxin Liu, Ruida Zhou, Xinyan Zhao +6LLM Self-Correction LLM Answer Verification
Northwestern University · Amazon AGI · Qualcomm AI Research +1

Jul 31, 2026J/K move · Enter open · S save

Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds

Yefan Tao, Gerald Friedland, Madhusudhanan Chandrasekaran, Luyang Kong

Amazon

LLM Self-Correction LLM Self-Refinement Language Model Self-Assessment LLM Reliability Metacognition in Language Models Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

Reflection, the ability to revisit and revise prior reasoning, is central to how humans improve their answers. Large language models (LLMs) are increasingly prompted to "reflect," yet whether this resembles human revision remains unclear. We introduce the Human-LLM Reflection Framework (HRF), a controlled two-pass protocol comparing human and LLM revision under identical conditions across self-, peer-, and cross-agent settings. Using an information-theoretic analysis based on per-iteration cross-entropy reduction, we find two failure modes of LLM reflection. On objective tasks with finite answer spaces, reflection yields near-zero information gain (Delta I approx 0), behaving as neutral re-generation indistinguishable from re-sampling. On subjective tasks, it yields significant negative gain (Delta I < 0), moving predictions away from the target. Human revision, by contrast, yields positive gain in both settings. Cross-agent experiments localize the failure to the revision step, not input quality: LLMs degrade even high-quality human responses. Diagnostic analyses (revision conditioned on first-pass correctness, and oracle-guided revision against a random-reshuffle baseline) show that which sub-step dominates varies by task and by model rather than reducing to a single mechanism: self-error detection is present on objective multiple-choice tasks but weak on subjective ones, and recovery under an oracle error signal exceeds the baseline for some models and falls below it for others. The unifying account is structural: without external information, self-conditioned revision cannot reduce uncertainty about the target, so LLM reflection is better understood as conditioned re-generation than as genuine error-driven revision.

Abstract

Multi-LLM Collaboration

Explore similar work

Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds

Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity

REVES: REvision and VErification--Augmented Training for Test-Time Scaling