cs.CLJun 26, 2026

Position Bias Correction is Insufficient for One-Pass Attention Sorting

Authors: Qiong TangXiangkun HuXiangyang LiuYiran ChenYunfan Shao

Organizations: Analemma

Abstract

Long-context language models suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents based on attention patterns, but its multiple sort-and-generate cycles increase deployment cost. We hypothesize that position bias is the primary bottleneck and propose Debiased One-Pass Attention Sorting, which estimates a per-prompt position-bias curve from the low-attention majority of documents and uses it to correct raw attention scores (via subtraction or division) to enable single-pass sorting. Our experiments on two models refute this hypothesis in the tested setting: on LLaMA-2-7B-32K-Instruct, debiasing produces identical results to uncalibrated single-pass sorting (94.83% containment accuracy), while on YaRN-Llama-2-7b-64k, debiasing improves accuracy by 8.67 percentage points but remains 14.84pp behind iterative sorting, closing only 37% of the gap. These results suggest that position-bias correction is insufficient to match iterative sorting, and that repeated reordering provides additional benefits beyond bias correction.

Explore similar work

CardsList
  1. When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

    Aug 4, 2026Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2Positional Encoding