The strong capabilities of recent Large Language Models (LLMs) have made them highly effective for zero-shot re-ranking task. Attention-based re-ranking methods, which derive relevance scores directly from attention weights, offer an efficient and interpretable alternative to generation-based re-ranking methods. However, they still face two major limitations. First, attention signals are highly concentrated a small subset of tokens within a few documents, making others indistinguishable. Second, attention often overemphasizes phrases lexically similar to the query, yielding biased rankings that irrelevant documents with mere lexical resemblance are regarded as relevant. In this paper, we propose ReAttn, a post-hoc re-weighting strategy for attention-based re-ranking methods. It first compute the cross-document IDF weighting to down-weight attention on query-overlapping tokens that frequently appear across the candidate documents, reducing lexical bias and emphasizing distinctive terms. It then employs entropy-based regularization to mitigate over-concentrated attention, encouraging a more balanced distribution across informative tokens. Both adjustments operate directly on existing attention weights without additional training or supervision. Extensive experiments demonstrate the effectiveness of our method.
Figures & tables
Figure 1: Illustration of two issues in attention-based re-ranking. (a) Signal concentration: the total attention mass is heavily concentrated on a few tokens of a few documents, leaving most candidates with negligible attention scores. (b) Lexical bias: attention disproportionately highlights tokens that are lexically similar to the query (e.g., producer, Jiang Wen), causing irrelevant documents with lexical overlap to the query to receive inflated relevance scores and misleading the ranking.
NQ
COVID
NFCorpus
FiQA
Scifact
Scidocs
FEVER
Climate
DBPedia
Robust04
News
Avg
BM25
30.5
59.5
32.2
23.6
67.9
14.9
65.1
16.5
31.8
40.7
39.5
38.4
Base LLM: Llama-3.2-3B-Instruct
RankGPT w/o
30.0
59.5
32.2
23.6
67.9
14.9
65.9
17.1
31.8
40.7
39.5
38.5
RankGPT Bubble
33.2
61.8
32.0
22.4
66.1
14.8
65.8
17.1
34.8
40.5
40.2
39.0
ICR
49.2
72.3
33.8
31.8
73.3
17.4
82.6
24.2
34.7
47.2
44.7
46.5
ICR+ReAttn
49.8
72.5
34.4
32.5
74.1
17.5
83.0
24.5
36.8
47.7
45.4
47.3
Table 1: Performance comparison across different base LLMs and re-ranking methods on BEIR benchmarks evaluated by nDCG@10. Bold indicates the best results.
LongMemEval
Clipper
Retriever
Retrieval
End-to-End
Retrieval
End-to-End
Recall@k
Performance
Recall@k
Performance
k = 5
k = 10
Top-5
Top-10
k = 3
k = 5
Top-3
Top-5
Base LLM: Llama-3.2-3B-Instruct
Full context
-
-
28.1
-
-
25.2
BM25
57.5
67.5
46.1
44.9
74.6
83.7
20.0
22.8
Table 2: Performance comparison across different base LLMs and re-ranking methods on LongMemEval and Clipper. The base LLM denotes the LLM used for both the retriever and end-to-end generation.
NQ
COVID
NFCorpus
FiQA
Scifact
Scidocs
FEVER
Climate
DBPedia
Robust04
News
Avg
Base LLM: Llama-3.2-3B-Instruct
ICR
49.2
72.3
33.8
31.8
73.3
17.4
82.6
24.2
34.7
47.2
44.7
46.5
ICR+IDF only
49.6
72.3
34.2
32.3
73.8
17.3
82.7
24.2
36.0
47.3
44.9
46.8
ICR+Entropy only
49.4
72.4
33.9
32.0
73.6
17.4
82.9
24.4
35.5
47.6
45.3
46.8
ICR+ReAttn
49.8
72.5
34.4
32.5
74.1
17.5
83.0
24.5
36.8
47.7
45.4
47.3
QRhead
54.9
77.4
35.1
35.1
74.7
18.3
83.7
24.5
36.0
49.7
45.1
48.6
Table 3: Performance comparison across different base LLMs and re-ranking methods on BEIR benchmarks evaluated by nDCG@10. Bold indicates the best results.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Prompt used for ICR, QRhead and our method.
Figure 3: Prompt used for rankGPT.
Method
Passage
Rank
ReAttn w/o IDF
The Scar ( Blizna ) is a 1976 Polish film written and directed by Krzysztof Kie ś lowski and starring Franciszek Pieczka . Filmed on location in Olech ó w , Poland , the film is about a man put in charge of the construction …
3
ReAttn
The Scar ( Blizna ) is a 1976 Polish film written and directed by Krzysztof Kie ś lowski and starring Franciszek Pieczka . Filmed on location in Olech ó w , Poland , the film is about a man put in charge of the construction …
6
Appendix
Table 4: Attention scores for all tokens in a FEVER example with the query “Night of the Living Dead was originally directed by Krzysztof Kieslowski.”
Method
Passage
Rank
ReAttn w/o IDF
Emperor Motion Pictures ( known as EMP ) is a film producer and distributor , part of the Emperor Group . Following the 2003 box - office hits The Twins Effect and The Medallion , EMP has produced … The Sun Also Rises and Forever Enthralled , two works by renowned Chinese auteurs Jiang Wen and Chen Kaige . …
2
ReAttn
Emperor Motion Pictures ( known as EMP ) is a film producer and distributor , part of the Emperor Group . Following the 2003 box - office hits The Twins Effect and The Medallion , EMP has produced … The Sun Also Rises and Forever Enthralled , two works by renowned Chinese auteurs Jiang Wen and Chen Kaige . …
10
Appendix
Table 5: Attention scores for all tokens in a FEVER example with the query “Jiang Wen is exclusively a producer.”
Method
Passage
Rank
ReAttn w/o IDF
Cape Can av eral Air Force Station Launch Complex 19 Ċ Launch Complex 19 ( LC - 19 ) is a deactivated launch site on Cape Can av eral Air Force Station , Florida used by NASA to launch all of the Gemini manned space fl ights . It was also used by unmanned Titan I and Titan II missiles .L C - 19 was in use from 195 9 to 196 6 , during which time it saw 27 launches , 10 of which were manned . The first use of LC - 19 was on August 14 , 195 9 . This was a Titan I and the mission was declared a failure after the rocket exploded while still on the pad .
12
ReAttn
Cape Can av eral Air Force Station Launch Complex 19 Ċ Launch Complex 19 ( LC - 19 ) is a deactivated launch site on Cape Can av eral Air Force Station , Florida used by NASA to launch all of the Gemini manned space fl ights . It was also used by unmanned Titan I and Titan II missiles .L C - 19 was in use from 195 9 to 196 6 , during which time it saw 27 launches , 10 of which were manned . The first use of LC - 19 was on August 14 , 195 9 . This was a Titan I and the mission was declared a failure after the rocket exploded while still on the pad .
18
Appendix
Table 6: Attention scores for all tokens in a DBPedia-Entity example with the query “Give me all launch pads operated by NASA.”