The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning

Matthew Perlman; Atharva Nijasure; James Allan

cs.CLAug 24, 2026

The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning

Authors: Matthew Perlman, Atharva Nijasure, James Allan

Organizations: University of Massachusetts Amherst

Abstract

LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction. We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.

Figures & tables

	NDCG	Mean Score Margin
Base Model	0.199	$-$ 0.204
Fine-Tuned Model	0.911	8.768
Performance Gain	0.712	8.972

Table 1: Performance of the base model and fine-tuned model evaluated over 50 queries

\times

100 documents.

Figure 1: The performance difference between the fine-tuned model and ablated model, measured per- omitted-head in terms of mean score margin. Scores are normalized by the maximum absolute value across all heads, such that the scale reflects relative importance in the network. Red indicates omitting the given head hurts performance and green indicates it improves performance. Omitting fine-tuning on many of the heads in layers 7 through 19 and 26 through 29 hurts performance most. Several heads in layers 19 through 32 perform marginally better using the base model parameters.

Figure 2: The performance difference between the fine-tuned model and ablated model, measured per- omitted-layer in terms of mean score margin. Omitting layers 14 and 29 hurts performance most.

Figure 3: The performance difference between the fine-tuned model and ablated model, measured per- omitted-window of size 6 in terms of NDCG. Window ablations show that earlier layers, roughly 1 through 20, are more performance-critical.

Figure 4: The performance difference between the base model and ablated model, measured per- kept-head in terms of mean score margin. Scores are normalized by the maximum absolute value across all heads, such that the scale reflects relative importance in the network. Green indicates keeping the given head improves performance and red indicates it hurts performance. Keeping fine-tuning on many of the heads in layers 10 through 18 and 25 through 32 benefits performance most.

Figure 5: The performance difference between the base model and ablated model, measured per- kept-layer in terms of mean score margin. Keeping fine-tuning produces roughly uniform performance gains throughout other than layers 14 and 29, which yielded notably higher improvement than others.

Figure 6: The performance difference between the base model and ablated model, measured per- kept-window of size 6 in terms of NDCG. These ablations show that fine-tuning layers in the range 10 to 18 is generally most important for improving NDCG over the base model.

Figure 7: The difference between normalized lexical matching attention, measured per- head , before and after fine-tuning. Attention to lexical matching generally decreases in the fine-tuned model, especially for layers 11 on. Lexical matching notably increases in the last layer relative to the base model.

Figure 8: The difference between normalized rarity-sensitive attention, measured per- head , before and after fine-tuning. Rarity-sensitive attention generally increased in the fine-tuned model for layers 8 through 19 and decreased for layers 20 through 32. Despite general trends at the window level, rarity sensitivity fluctuates dramatically per-head.

Figure 9: The difference between normalized document-query attention, measured per- head , before and after fine-tuning. The fine-tuned model exhibits document-query attention more than the base model in layers 8–22.

Feature	Keep $\rho$	Omit $\rho$
Lexical Matching	0.37	$-$ 0.63
Rarity Sensitivity	0.92	$-$ 0.89
Document-Query Interaction	0.71	$-$ 0.68

Table 2: Spearman correlation between the effect of per-window ablations on NDCG performance and per-window change in normalized feature attention after fine-tuning, computed over sliding windows of size 6. Keep correlations are positive because windows that learn stronger feature attention also produce larger performance gains when kept in isolation; omit correlations are negative because those same windows produce larger performance drops when removed. For correlations over other sliding window sizes, see § G .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	NDCG	Mean Score Margin
Base Model	0.437	-0.106
Fine-Tuned Model	0.970	8.866
Performance Gain	0.533	8.972

Appendix

Table 3: Performance of the base model and fine-tuned model evaluated over 50 queries

\times

10 documents.

Panel of Figure 10 — Figure 10: Lexical match attention in the base model (Left) vs. fine-tuned model (Right).

Panel of Figure 11 — Figure 11: Rarity-sensitive attention in the base model (Left) vs. fine-tuned model (Right).

Panel of Figure 12 — Figure 12: Document-query attention in the base model (Left) vs. fine-tuned model (Right).

Figure 13: Normalized attention to lexical matching, measured per-head, in the Base Model.

Figure 14: Normalized attention to rare terms, measured per-head, in the Base Model.

Figure 15: Normalized attention across document-query segments, measured per-head, in the Base Model.

Figure 16: Normalized attention to lexical matching, measured per-head, in the fine-tuned model.

Figure 17: Normalized attention to rare terms, measured per-head, in the fine-tuned model.

Figure 18: Normalized attention across document-query segments, measured per-head, in the fine-tuned model.

Figure 19: Change in normalized feature attention for rare document tokens attending rare lexical match query tokens, averaged over sliding windows of size 6, between the base and fine-tuned models.

Figure 20: Change in normalized feature attention for document tokens attending lexical match query tokens, averaged over sliding windows of size 6, between the base and fine-tuned models.

Figure 21: Change in normalized feature attention for document tokens attending rare tokens, averaged over sliding windows of size 6, between the base and fine-tuned models.

Feature	Keep $\rho$	Omit $\rho$
Rare Document Tokens Attending
Rare Lexical Match Query Tokens	0.88	$-$ 0.94
Document Tokens Attending
Lexical Match Query Tokens	0.92	$-$ 0.92
Document Tokens
Attending Rare Tokens	0.92	$-$ 0.88

Appendix

Table 4: Spearman correlation between the effect of per-window ablations on NDCG performance and per-window change in normalized feature attention after fine-tuning. Keep correlations are positive because windows that learn stronger feature attention also produce larger performance gains when kept in isolation; omit correlations are negative because those same windows produce larger performance drops when removed.

Feature	Keep $\rho$	Omit $\rho$
Window Size 3
Lexical Matching	0.06	$-$ 0.39
Rarity Sensitivity	0.79	$-$ 0.80
Document-Query Interaction	0.61	$-$ 0.64
Window Size 4
Lexical Matching	0.19	$-$ 0.55

Appendix

Table 5: Spearman correlation between the effect of per-window ablations on NDCG performance and per-window change in normalized feature attention after fine-tuning. Keep correlations are positive because windows that learn stronger feature attention also produce larger performance gains when kept in isolation; omit correlations are negative because those same windows produce larger performance drops when removed.

Figure 22: Omit-window results using window size 3.

Figure 23: Omit-window results using window size 4.

Figure 24: Keep-window results using window size 3.

Figure 25: Keep-window results using window size 4.

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 133 papers

Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models
Apr 27, 2026Yuxing Tian, Fengran Mo, Zhiqi Huang +2Attention Head Analysis Learning to Rank
Université de Montréal · Capital One · McGill University & MILA
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Jul 24, 2026Wei Zhang, Xinwu Liu, Yihang ChengLLM Fine-Tuning Low-Rank Adaptation
Faculty of Artificial Intelligence in Education Laboratory for Artificial Intelligence and New Forms of Education Central China Normal University Wuhan, Hubei, China
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Feb 23, 2026Yuxing Tian, Fengran Mo, Weixu Zhang +2LLM Reranking Learning to Rank
University of Montreal · McGill University & MILA · International Digital Economy Academy

Apr 27, 2026J/K move · Enter open · S save

Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models

Yuxing Tian, Fengran Mo, Zhiqi Huang, Weixu Zhang+1

Université de Montréal · Capital One · McGill University & MILA

Attention Head Analysis Learning to Rank LLM Reranking Attention Mechanisms LLM Routing

Open paper AskPDF arXiv

Large Language Models (LLMs) have recently been explored as fine-grained zero-shot re-rankers by leveraging attention signals to estimate document relevance. However, existing methods either aggregate attention signals across all heads or rely on a statically selected subset identified by heuristic rules. This solution can be suboptimal because the informative heads can vary across queries or domains. Moreover, naively combining multiple heads can degrade performance due to redundancy or conflicting ranking signals. In this paper, we propose a query-dependent head selection method, RouteHead, for attention-based re-ranking with LLMs. Specifically, we learn a lightweight router that can map each query to an optimal head set, and relevance scores are computed by aggregating attention signals only from these heads. Since query-to-head optimal labels are unavailable, we first construct pseudo labels via an offline search. The router represents each head with a learnable embedding and represents each query using an embedding extracted from the hidden states of the frozen LLM. Then it is trained on the pseudo labels with a sparsity regularizer. Experiments on diverse benchmarks and multiple LLM backbones show that the proposed method consistently outperforms strong baselines.

Abstract

LLM Reranking

Explore similar work

Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting