Position Bias

Position bias in large language models (LLMs) refers to the tendency of these models to disproportionately weight information based on its location within an input sequence, often favoring information at the beginning or end. Current research focuses on mitigating this bias through techniques like parameter-efficient fine-tuning, data augmentation, and unsupervised debiasing methods that leverage the models' own responses. Addressing position bias is crucial for improving the reliability and accuracy of LLMs in various applications, particularly those involving long-context processing such as question answering and document retrieval, ultimately leading to more robust and trustworthy AI systems.

Papers

October 18, 2024

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs
Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, Xiaojiang Liu
Large Language Model Absolute Stance Bias Long Context Distance Matter Long Input Relevant Information Position Bias

April 5, 2024

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
João Coelho, Bruno Martins, João Magalhães, Jamie Callan, Chenyan Xiong
Representation Learning Transformer Based Model Dense Retrieval Text Representation Contrastive Pre Training Position Bias

April 1, 2024

Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs
Zheng Zhang, Fan Yang, Ziyan Jiang, Zheng Chen, Zhengyang Zhao, Chengyuan Ma, Liang Zhao, Yang Liu
Large Language Model Context Sequence Long Input Context Position Bias

January 2, 2024

Zero-Shot Position Debiasing for Large Language Models
Zhongkun Liu, Zheng Chen, Mengqi Zhang, Zhaochun Ren, Pengjie Ren, Zhumin Chen
Large Language Model Sampling Bias Unsupervised Sentence Position Bias

September 29, 2023

Split and Merge: Aligning Position Biases in Large Language Model based Evaluators
Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, Yang Liu
Large Language Model Human Evaluation Split and Fit LLM Evaluator Position Bias

Position Bias

Papers

Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs

Zero-Shot Position Debiasing for Large Language Models

Split and Merge: Aligning Position Biases in Large Language Model based Evaluators