Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and Answer-Letter Preference

Izhar Ali

cs.AIMay 18, 2026

Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and Answer-Letter Preference

Authors: Izhar Ali

Organizations: Rowan University.

Abstract

Disagreement between language models is often read as missing knowledge. Questions that need the same missing knowledge should then be right or wrong together. We test for this structure with the eigenvalues of the questions' correlation matrix. Within one model, independently drawn answers show no detectable structure; it appears when all questions in one pass over the benchmark share a random seed or few-shot prompt. Across 24 models, the largest structure beyond overall skill is a preference among answer letters: when the options are rotated, it follows the letter, not the question, and it depends on one model, Llama-3.2-1B; without it only a smaller excess remains. Counting a question correct only if right in every rotation removes the part that acts on guesses. What remains is a marginal excess at the 5% line, tied post hoc to two near-duplicate model pairs and to Llama-3.2-1B.

Figures & tables

Figure 1: Llama-3.2-1B is accurate where the answer is placed at “A”, in each of the four rotations; Gemma-2-9B is about equally accurate at every letter. Filled: all 500 questions, with the options rotated so that the correct answer is at that letter; bars: Wilson 95% CI. Open: each of the four rotations alone (about 125 questions).

Figure 2: (a) Within one model:

\lambda_{\max}/\lambda_{+}

(log axis) of each run; grey bar: 5–95% of its simulated distance from the edge (

z

, App. B ), on this axis; circles: independent draws; filled: an eigenvalue above

\lambda_{+}

. (b) Across models, rotations 1–3: correlation over questions of each question’s eigenvector-2 loading with the mean rotation-0 loading of the questions that had its new gold letter (same letter), or with its own rotation-0 loading (same question). (c)

\lambda_{2}

–

\lambda_{4}

of the 24-model matrix in rotation 0 and all-4-correct, divided by the 99th percentile (1000 draws) of two adequate one-skill nulls (rotation 0: 3PL left and KG-u, uniform know-or-guess, right; all-4-correct: 2PL left, Rasch right); filled: above it. Other nulls: App. D (soft-preference: not rejected).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Run	$N_{b}$	$\lambda_{+}$	$\lambda_{\max}$	MC pct. $z$	MP count
Independent, MMLU ( $K=100$ , $T=1$ )
Qwen2.5-7B	43	2.74	2.56	60.1	0
Mistral-7B-v0.3	83	3.65	3.45	51.3	0
Llama-3-8B	173	5.36	4.97	10.2	0
SmolLM2-1.7B	406	9.09	8.74	28.6	0
Phi-3-mini	271	7.00	6.91	83.5	0

Appendix

Table A1: The test on every within-model run (

\varepsilon=0.05

\lambda_{+}

: MP edge. MC pct.

z

: Monte Carlo percentile of

z

(App. B ) among 1000 simulated matrices of independent answers with the same pass rates. MP count: number of eigenvalues above

\lambda_{+}

. GSM8K runs use

T=1

Figure A1: Detection rate of the MP count against the latent correlation

\rho

of a factor planted on all borderline questions or on 30 or 9 of them, at the pass rates of two MMLU runs. Whiskers: Wilson 95% intervals.

Figure A2: A shared seed or prompt makes a pass’s letter shares vary beyond chance. Of the shared runs, only the seed run and Llama-3-8B’s prompt run also group the top eigenvector’s loadings by gold letter. (a) Across-pass variance of a pass’s letter shares, summed over letters, divided by the mean of that sum after each question’s answers are shuffled across passes. (b)

\eta^{2}

, the share of the variance of the top eigenvector’s loadings that the gold letter explains. Grey tick: null 99th (a) or 95th (b) percentile; filled: beyond it.

Figure A3: Each model’s rotation-0 score on eigenvector 2 of the 24-model matrix against its share of D answers.

	2PL	Rasch
Pct. of $\lambda_{2},\lambda_{3},\lambda_{4}$	71.8, 98.5, 99.1	94.6, 97.0, 98.6
$p_{M}$ , 24 models	0.026	0.041
Sibling subsets	0/24 (0.09–0.65)	0/24 (0.12–0.69)
Random subsets	24/24 ( $\leq$ 0.047)	24/24 ( $\leq$ 0.046)
Pair removals	0/4 (0.22–0.69)	0/4 (0.32–0.38)
Two other models removed	0.90 ( $\leq$ 0.099)	0.74 ( $\leq$ 0.133)

Appendix

Table A2: Max-statistic test on all-4-correct (reject at

p_{M}<0.05

), against the 2PL and Rasch nulls. Pct.: percentile of each eigenvalue in the null (2000 draws; 2PL

\lambda_{4}

is just below its 99th percentile in the 1000 draws of Fig. 2 c). Subsets: number that reject, and the range or maximum of

p_{M}

; sibling: one model kept from each sibling group; random: five of the other 15 models removed. Pair removals: one Phi and one Llama model. Two other models removed: the 190 removals of two models outside the pairs (fraction that reject, maximum

p_{M}

). Unparseable: replies imputed, or questions dropped (1000 draws), either the 95 on which some model has no wrong parsed rotation and at least one unparseable one, or all 217 that contain one. Size: rejection rate in 300 datasets simulated from each fitted one-skill model. Calibrated

p_{M}

: under 2PL and Rasch truth. Power: max-statistic test / rank-2 rule, 200 datasets per strength

s

(2PL fit).

Model	A	B	C	D
Llama-3.2-1B	0.81	0.04	0.09	0.05
Qwen2.5-0.5B	0.23	0.06	0.12	0.58
SmolLM2-1.7B	0.07	0.20	0.17	0.56
Model pair			$r$	Rank
Phi-3-mini, Phi-3.5-mini			0.40	1
Llama-3-8B, Llama-3.1-8B			0.27	2

Appendix

Table A3: Simple baselines. Top: rotation-0 letter shares of the three models whose largest share exceeds 0.5; no other model’s largest share exceeds 0.35. Bottom: correlation of model pairs’ residuals after the all-4-correct 2PL fit, with rank among 276 pairs (mean

-0.04

, standard deviation 0.08).

Correlated Errors in Language Models

See all 9 papers

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer
Oct 26, 2026Xinyi NiLLM Prompting Knowledge Conflicts in Language Models
Brandeis University
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA
LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning
Jul 11, 2026Ning LiuLLM Answer Verification Test-Time Scaling
Independent Researcher

Oct 26, 2026J/K move · Enter open · S save

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer

Xinyi Ni

Brandeis University

LLM Prompting Knowledge Conflicts in Language Models LLM Reliability

Open paper AskPDF arXiv

Language models now answer questions alongside other claims: a note in the system prompt, a message from another agent, their own earlier answer. We ask what it takes to talk a model out of an answer it gets right on its own, and separate three factors that earlier studies vary together: where the advice sits in the prompt, who is said to have written it, and what the model has already written. On Qwen3.5-27B and four-option MMLU-Pro items, with the rest of the advice held fixed in each comparison, we find three things. First, position matters far more than the API channel, in opposite directions for a bare claim and an argument: a bare "the answer is X" does most harm just before the model answers, while the same claim with a false rationale does most harm before the question, where it raises error on items the model otherwise almost never misses from 1% to 67%; moving the claim between the system message and the user message at the same position changes error by a few points. Second, attributing the note to an authority rather than to another participant, and phrasing it assertively, each raise error by 10-27 points, while the two authorities we tested differ by under two. Third, replaying a correct worked answer before the advice protects it whether the text is shown as the model's own or as someone else's, and the same replay keeps wrong answers from being corrected by a bare correct note. The position result replicates under an independently written prompt. Code, items and all result tables are released at https://github.com/coolkillercat/safety.

Abstract

Correlated Errors in Language Models

Explore similar work

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning