cs.IRJun 1, 2026

Attention Calibration for Position-Fair Dense Retrieval

Authors: Andrianos Michail, Elias Schuhmacher, Juri Opitz, Simon Clematide, Rico Sennrich

Organizations: University of Zurich · Zurich, Switzerland

Abstract

Dense retrieval compresses a passage into a single vector, but this compression is positionally skewed: early content dominates the embedding, and retrieval degrades when the relevant span appears later. Prior work proposed an inference-time method that counteracts this skew by equalizing the pooling token's attention across passage segments. However, (i) it redistributes attention at a fixed strength, (ii) it forces the pooling token's attention to itself to a fixed basket-level mass despite substantial variation across layers and architectures, and (iii) its effect on retrieval has not been evaluated. We introduce a strength coefficient that interpolates between uncalibrated and fully equalized attention, together with an efficient implementation that reduces peak calibration memory overhead from 5-7 GiB to under 1 MiB. Across three embedding models and two pooling schemes, moderate calibration provides a better retrieval trade-off than full equalization. We introduce a variant that preserves the pooling token's self-attention mass and redistributes only the remaining mass. On a position-aware retrieval benchmark spanning 10 languages and 31 domains, a configuration selected on English FineWeb-PosQ and transferred without tuning reduces position sensitivity in all 16 evaluated length-quartile, model, and retrieval-setting combinations, by up to 43% relative, while improving nDCG@10 by up to 4.8% relative and leaving general retrieval effectiveness on NanoBEIR essentially unchanged. Calibration runs at indexing time, adding no query-time latency. We release our code at github.com/impresso/fair-sentence-transformers

Explore similar work

CardsList
  1. When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

    Aug 4, 2026Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2Positional Encoding

  2. Parameter-Efficient Retrievers for Polish and European Languages

    Sep 11, 2026Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec +1RetrieversSmall Language Models