On the Position Bias of Large Language Models: A Comprehensive Study

Yining Li; Huayang Li; Lemao Liu; Taro Watanabe

cs.CLOct 16, 2026

On the Position Bias of Large Language Models: A Comprehensive Study

Authors: Yining Li, Huayang Li, Lemao Liu, Taro Watanabe

Organizations: Jilin University · Fudan University · Nara Institute of Science and Technology

Abstract

Recent studies have shown that position bias remains a critical vulnerability in large language models (LLMs), undermining both the integrity of multiple-choice evaluations and the reliability of retrieval-augmented generation. However, existing understanding of this phenomenon is largely derived from studies with limited model coverage and English-centric benchmarks, leaving the behavior of frontier models and diverse languages underexplored. In this work, we conduct a comprehensive investigation of position bias across model scales ranging from billion-level to trillion-level parameters, covering both multiple-choice and multi-document tasks in five languages. Our experiments reveal that while position bias remains noticeable in small-scale models and Non-English languages, increasing model scale and employing standard Chain-of-Thought prompting significantly mitigate this issue. Moreover, we demonstrate that emerging thinking models with extended reasoning capabilities are nearly immune to position bias, suggesting a costly but general approach toward resolving this vulnerability.

Figures & tables

Panel of Figure 1 — Figure 1: Position bias on English Multi-Choice tasks. Results are reported on MMLU benchmark Hendrycks et al. (2021) using non-thinking models. The label of golden answer is shifted from “(A)” to “(D)” for all data, and the options are positioned in alphabetical order of the labels, i.e., “(A)” appearing first and “(D)” last.

	0shot	5shot	0shot CoT	5shot CoT	AVG.
Q3-1.7B	15.3	56.9	5.1	5.5	20.7
Q3-8B	6.6	6.2	5.0	3.7	5.4
Q3-30B	8.6	19.8	3.4	1.0	8.2
Q3-32B	7.8	3.4	2.9	1.1	3.8
Q3-235B	12.1	10.1	2.7	1.3	6.6
DS-V3.2	1.6	10.1	3.4	1.0	4.0

Table 1: The performance difference

\Delta_{\theta}

of LLMs on English Multi-Choice tasks. Results are reported on MMLU using non-thinking models.

Panel of Figure 2 — Figure 2: Position bias on Multilingual Multi-Choice tasks. Results are reported on MMMLU benchmark using non-thinking models and 0-shot CoT template. The label of golden answer is shifted from “(A)” to “(D)” for all data, and the options are positioned in alphabetical order of the labels, i.e., “(A)” appearing first and “(D)” last.

	EN	ZH	AR	JA	ES	AVG.
Q3-1.7B	5.1	19.5	12.8	19.9	20.1	18.1
Q3-8B	5.0	12.8	10.7	6.6	6.6	9.2
Q3-30B	3.4	7.0	18.5	27.5	11.5	16.1
Q3-32B	2.9	2.6	29.5	3.7	3.5	9.8
Q3-235B	2.7	3.1	1.3	6.9	7.0	4.6
DS-V3.2	3.4	7.6	9.0	12.4	6.7	8.9

Table 2: The performance difference

\Delta_{\theta}

of LLMs on Multilingual Multi-Choice tasks. Results are reported on MMMLU benchmark using non- thinking models and 0-shot CoT prompt.

Panel of Figure 3 — Figure 3: Position bias on Multilingual Multi-Choice tasks. We use the Deepseek-V3.2, Deepseek-V4-Pro and the 0-shot CoT tempalte for evaluation.

Panel of Figure 4 — Figure 4: Position bias on Multi-Document QA.

	EN	ZH	AR	ES	AVG.
Q3-1.7B	6.8	8.5	6.7	9.8	8.0
Q3-8B	3.5	4.5	2.3	6.2	4.1
Q3-30B	2.3	3.0	4.3	5.8	3.9
Q3-32B	2.8	2.0	4.5	3.8	3.3
Q3-235B	2.3	3.8	5.5	1.2	3.2
DS-V3.2	2.1	3.8	6.3	3.2	3.9

Table 3: The performance difference

\Delta_{\theta}

of LLMs on multi-document tasks. Results are reported on non-thinking models.

Panel of Figure 5 — Figure 5: Position bias on Multi-Document tasks. We use the Deepseek-V3.2 and Deepseek-V4-Pro for evaluation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Error Type	Percentage
Type 1: Conflicting Docs	38.4%
Type 2: Relevance Identification	27.7%
Type 3: Unknown	33.9%

Appendix

Table 4: Error type distribution among cases where thinking models outperform non-thinking models.

Position Bias

Position bias is the tendency of models or people to favor information because of where it appears, independent of its content.

See all 33 papers

Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation
May 3, 2026Xuemei Tang, Xufeng Duan, Zhenguang G. CaiMultiple-Choice Question Generation Language Model Bias Evaluation
The Hong Kong Polytechnic University · The Chinese University of Hong Kong
Hearing the Order: Investigating Position Bias in Large Audio-Language Models
Oct 1, 2025Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei +3Audio-Language Model Evaluation Audio-Language Models
National Taiwan University
Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
Jul 23, 2026Hiroki TambaLLM Evaluation Language Model Bias Evaluation
Independent researcher.

May 3, 2026J/K move · Enter open · S save

Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation

Xuemei Tang, Xufeng Duan, Zhenguang G. Cai

The Hong Kong Polytechnic University · The Chinese University of Hong Kong

Multiple-Choice Question Generation Language Model Bias Evaluation Position Bias

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to generate multiple-choice questions (MCQs), where correct answers should ideally be uniformly distributed across options. However, we observe that LLMs exhibit systematic position biases during generation. Through extensive experiments with 10 LLMs and 5 vision-language models (VLMs) on three MCQ generation tasks, we show that these biases are structured, with similar patterns emerging within model families. To investigate the underlying mechanisms, we conduct probing experiments and find that hidden representations in the question stem encode predictive signals of the correct answer position, suggesting that answer position may be implicitly planned during generation. Building on this insight, we apply activation steering to manipulate internal representations and influence answer position. Our results show that steering can partially control positional preferences and substantially shift answer position distributions. Our findings provide a practical framework for studying implicit positional planning in LLMs and highlight the importance of controllable generation for reliable MCQ construction and evaluation.

Abstract

Position Bias

Explore similar work

Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation

Hearing the Order: Investigating Position Bias in Large Audio-Language Models

Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks