Position Bias

Follow this topic Ask about this topic

Position bias studies how answer order, passage placement, list rank, or token position can change a decision even when the underlying content is unchanged. Researchers measure it with controlled permutations and counterbalanced tests in multiple-choice benchmarks, LLM judging, retrieval and recommendation, and use attention or residual-stream analyses to investigate its causes. Mitigations include randomizing or averaging over orderings, calibrating scores, training for permutation consistency, and changing positional representations. A key tension is that aggregate accuracy can hide bias, while reducing order sensitivity may cost latency or ranking quality and may not transfer across tasks or models.

Latest papers

33 papers1 paper in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
Oct 26, 2026Mingyang Song, Mao Zheng, Zhiyuan Ma +1Language Model Bias Evaluation Position Bias
Foundation Model Department, Tencent, China
On the Position Bias of Large Language Models: A Comprehensive Study
Oct 16, 2026Yining Li, Huayang Li, Lemao Liu +1Position Bias Language Model Bias Evaluation
Jilin University · Fudan University · Nara Institute of Science and Technology
PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions
Sep 26, 2026Yida LinLanguage Model Post-Training Language Model Calibration
Victoria University of Wellington Wellington, New Zealand
Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4
Sep 25, 2026Amanda FitchFraming Effects in Language Models Language Model Bias Evaluation
Google
The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits
Sep 8, 2026Siddharth Vohra, Manikandan RavikiranLanguage Model Bias Evaluation Demographic Bias in Language Models
Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA · Indian Institute of Technology Mandi, India
Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers
Aug 27, 2026Markus Frohmann, Mahdiyar Ali Akbar Alavi, Elizabeth Lingg +1LLM Reranking Learning to Rank
Thomson Reuters Labs · University of Toronto, Vector Institute
Does Rank Still Matter? Position Bias When AI Agents Shop on Our Behalf
Aug 24, 2026Davood Wadi, Yu MaAgentic Search Agentic Commerce
Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
Aug 15, 2026Volodymyr OvcharovLLM Evaluation Multiple-Choice Question Answering
LEX AI
Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders
Aug 14, 2026Keito Kozaki, Keigo Sakurai, Ren Togo +2Sequential Recommendation Self-Attention
Hokkaido University Sapporo, Hokkaido, Japan
Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders
Jul 26, 2026Ge Zhang, Jingru Cheng, Huiyuan ChenLearning to Rank Adversarial Attacks
Stanford University · Independent Researcher
Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
Jul 23, 2026Hiroki TambaLLM Evaluation Language Model Bias Evaluation
Independent researcher.
OrDA: Orthogonal Disentanglement of Access Habits Framework for Homepage Marketing Block Recommendations
Jul 15, 2026Lingxiao Zhang, Xiaobo Li, Tao XuRepresentation Disentanglement Recommender Systems
Ant Group · Hangzhou, Zhejiang, China
Position Bias Correction is Insufficient for One-Pass Attention Sorting
Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Long-Context Language Modeling Language Model Bias Evaluation
Analemma
Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling
Jun 26, 2026Changze Lv, Zhenghua Wang, Yiran Ding +9Rotary Positional Embeddings Long-Context Language Modeling
Fudan University · Shanghai Key Laboratory of Intelligent Information Processing · Westlake University
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
VCG: A Multimodal Retrieval Framework for E-Commerce Video Feeds under Extreme Cold-Start Conditions
Jun 17, 2026Katya Mirylenka, Egor Malykh, Mahdyar Ravanbakhsh +10Multimodal IR Cold-Start Recommendation
TU Wien, Vienna, Austria · Zalando Switzerland AG, Zurich, Switzerland · Zalando SE, Berlin, Germany
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
Jun 15, 2026Jieyuan Liu, Jianyang Gu, Shijie Chen +2VLM Evaluation Multimodal RAG
University of California, San Diego · The Ohio State University
Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection
Jun 15, 2026Mirza Samad Ahmed Baig, Syeda Anshrah Gillani, Asher AliLLM Auditing Recommender Systems
Fandaqah, Al Khobar, Saudi Arabia. · Hamdard University, Karachi, Pakistan.
Where does Absolute Position come from in decoder-only Transformers?
Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriRotary Positional Embeddings Transformer Interpretability
Intuition Machines · Sapienza University of Rome
RAMPART: Registry-based Agentic Memory with Priority-Aware Runtime Transformation
Jun 3, 2026Nikodem TomczakMulti-Agent LLM Systems LLM Agent Memory
Thulge Labs, Singapore
When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering
May 13, 2026Yikun Han, Mengfei Lan, Halil KilicogluSelective Prediction Healthcare
University of Illinois Urbana-Champaign
Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
May 10, 2026Jiwei Tang, Zhijing Huang, Xinyu Zhang +5LLM Compression Position Bias
Sun Yat-sen University · Hong Kong Polytechnic University · Beijing Normal–Hong Kong Baptist University
Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation
May 3, 2026Xuemei Tang, Xufeng Duan, Zhenguang G. CaiMultiple-Choice Question Generation Language Model Bias Evaluation
The Hong Kong Polytechnic University · The Chinese University of Hong Kong
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
Apr 29, 2026Jon-Paul CacioliLanguage Model Bias Evaluation Position Bias
Independent Researcher, Melbourne, Australia
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models
Apr 21, 2026Xiao WangMultiple-Choice Question Answering CoT Reasoning
Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoring
Apr 20, 2026Sinan G. Aksoy, Alexandra A. Sabrio, Erik VonKaenel +1Semantic Textual Similarity LLM-as-a-Judge
Pacific Northwest National Laboratory · Washington University in St. Louis · Humana Inc.
Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration
Apr 11, 2026Hang Lv, Hongchao Gu, Ruiqing Yang +5Learning to Rank LLM Reranking
University of Science and Technology of China · Alibaba Group
Towards Position-Robust Talent Recommendation via Large Language Models
Apr 2, 2026Silin Du, Hongyan LiuRecommender Systems Large Language Model-Based Recommendation
Department of Management Science and Engineering, Tsinghua University, Beijing, China.
A Structural Theory of Position Bias in Transformers
Feb 18, 2026Hanna Herasimchyk, Robin Labryga, Tomislav Prusina +1Lost-in-the-Middle Effect Position Bias
University of Hamburg Hamburg, Germany

Oct 26, 2026J/K move · Enter open · S save

Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?

Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou

Foundation Model Department, Tencent, China

Language Model Bias Evaluation Position Bias Multiple-Choice Question Answering

Open paper AskPDF arXiv

Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27--29% relative to the original order, with first failures continuing into the final presentation. This loss of initial successes is substantially larger than under fixed-order repetition, even for highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.