Recent studies have shown that position bias remains a critical vulnerability in large language models (LLMs), undermining both the integrity of multiple-choice evaluations and the reliability of retrieval-augmented generation. However, existing understanding of this phenomenon is largely derived from studies with limited model coverage and English-centric benchmarks, leaving the behavior of frontier models and diverse languages underexplored. In this work, we conduct a comprehensive investigation of position bias across model scales ranging from billion-level to trillion-level parameters, covering both multiple-choice and multi-document tasks in five languages. Our experiments reveal that while position bias remains noticeable in small-scale models and Non-English languages, increasing model scale and employing standard Chain-of-Thought prompting significantly mitigate this issue. Moreover, we demonstrate that emerging thinking models with extended reasoning capabilities are nearly immune to position bias, suggesting a costly but general approach toward resolving this vulnerability.
Figures & tables
Figure 1: Position bias on English Multi-Choice tasks. Results are reported on MMLU benchmark Hendrycks et al. (2021) using non-thinking models. The label of golden answer is shifted from “(A)” to “(D)” for all data, and the options are positioned in alphabetical order of the labels, i.e., “(A)” appearing first and “(D)” last.
0shot
5shot
0shot CoT
5shot CoT
AVG.
Q3-1.7B
15.3
56.9
5.1
5.5
20.7
Q3-8B
6.6
6.2
5.0
3.7
5.4
Q3-30B
8.6
19.8
3.4
1.0
8.2
Q3-32B
7.8
3.4
2.9
1.1
3.8
Q3-235B
12.1
10.1
2.7
1.3
6.6
DS-V3.2
1.6
10.1
3.4
1.0
4.0
Table 1: The performance difference Δθ of LLMs on English Multi-Choice tasks. Results are reported on MMLU using non-thinking models.
Figure 2: Position bias on Multilingual Multi-Choice tasks. Results are reported on MMMLU benchmark using non-thinking models and 0-shot CoT template. The label of golden answer is shifted from “(A)” to “(D)” for all data, and the options are positioned in alphabetical order of the labels, i.e., “(A)” appearing first and “(D)” last.
EN
ZH
AR
JA
ES
AVG.
Q3-1.7B
5.1
19.5
12.8
19.9
20.1
18.1
Q3-8B
5.0
12.8
10.7
6.6
6.6
9.2
Q3-30B
3.4
7.0
18.5
27.5
11.5
16.1
Q3-32B
2.9
2.6
29.5
3.7
3.5
9.8
Q3-235B
2.7
3.1
1.3
6.9
7.0
4.6
DS-V3.2
3.4
7.6
9.0
12.4
6.7
8.9
Table 2: The performance difference Δθ of LLMs on Multilingual Multi-Choice tasks. Results are reported on MMMLU benchmark using non- thinking models and 0-shot CoT prompt.
Figure 3: Position bias on Multilingual Multi-Choice tasks. We use the Deepseek-V3.2, Deepseek-V4-Pro and the 0-shot CoT tempalte for evaluation.
Figure 4: Position bias on Multi-Document QA.
EN
ZH
AR
ES
AVG.
Q3-1.7B
6.8
8.5
6.7
9.8
8.0
Q3-8B
3.5
4.5
2.3
6.2
4.1
Q3-30B
2.3
3.0
4.3
5.8
3.9
Q3-32B
2.8
2.0
4.5
3.8
3.3
Q3-235B
2.3
3.8
5.5
1.2
3.2
DS-V3.2
2.1
3.8
6.3
3.2
3.9
Table 3: The performance difference Δθ of LLMs on multi-document tasks. Results are reported on non-thinking models.
Figure 5: Position bias on Multi-Document tasks. We use the Deepseek-V3.2 and Deepseek-V4-Pro for evaluation.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Error Type
Percentage
Type 1: Conflicting Docs
38.4%
Type 2: Relevance Identification
27.7%
Type 3: Unknown
33.9%
Appendix
Table 4: Error type distribution among cases where thinking models outperform non-thinking models.
Large language models (LLMs) are increasingly used to generate multiple-choice questions (MCQs), where correct answers should ideally be uniformly distributed across options. However, we observe that LLMs exhibit systematic position biases during generation. Through extensive experiments with 10 LLMs and 5 vision-language models (VLMs) on three MCQ generation tasks, we show that these biases are structured, with similar patterns emerging within model families. To investigate the underlying mechanisms, we conduct probing experiments and find that hidden representations in the question stem encode predictive signals of the correct answer position, suggesting that answer position may be implicitly planned during generation. Building on this insight, we apply activation steering to manipulate internal representations and influence answer position. Our results show that steering can partially control positional preferences and substantially shift answer position distributions. Our findings provide a practical framework for studying implicit positional planning in LLMs and highlight the importance of controllable generation for reliable MCQ construction and evaluation.
Xuemei Tang, Xufeng Duan, Zhenguang G. Cai
The Hong Kong Polytechnic University · The Chinese University of Hong Kong
Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options. An open question is whether their predictions are influenced by the order of answer choices, which would indicate a form of position bias and undermine their reliability. In this paper, we identify and analyze this problem in LALMs. We demonstrate that no model is immune to this bias through extensive experiments on six LALMs across three widely used benchmarks and their spoken counterparts. Shuffling the order of answer options can cause performance fluctuations of up to 24% and even change model rankings, raising concerns about the reliability of current evaluation practices. We also study permutation-based strategies and show that they can mitigate bias in most cases. Our work represents the first systematic investigation of this issue in LALMs, and we hope it raises awareness and motivates further research in this direction.
Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity. I introduce inspect_permute, an open-source extension to the inspect_ai evaluation framework that runs exhaustive answer-order permutations per question and reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals. I apply the tool across four vendors (gpt-4o-mini, claude-haiku-4-5, gemini-2.5-flash, grok-3) on five MMLU subjects, 24,000 API calls under temperature-0 generation, with falsifier predictions pre-registered via a public SHA-256 hash before half the data was observed. Position bias turns out to be statistically detectable only within a roughly 60-95% base-accuracy Goldilocks zone. Below it, processing-load dominance swamps subject-specific signal; above it, ceiling effects compress the variance below the chi-squared test resolution. Detectable cells separate into two mechanism types: monotone A-to-D decrease (processing_load, in low-tier models) and non-monotone D-drop (content_ambiguity, in a narrow capability band). Standard MMLU places every frontier-tier model above the detection band, so absence of signal there should be read as not measurable, not unbiased. Together with the ceiling-effect characterisation in arXiv:2606.26185, this work brackets the detectable region of position-bias measurement and makes the field central question askable in a verifiable form. Package, data, preregistration under MIT.