Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs

Peiyu Zhang; Heng Ping; Shinibali Bhattacharyya; Anwesan Pal; Nikos Kanakaris; Kevin Cheang; Shixuan Li; Wei Yang

cs.CLNov 6, 2026

Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs

Authors: Peiyu Zhang, Heng Ping, Shinibali Bhattacharyya, Anwesan Pal, Nikos Kanakaris, Kevin Cheang, Shixuan Li, Wei Yang,

Organizations: University of Southern California · Amazon Web Services

Abstract

Long-context language models often neglect information in the middle of the input, a phenomenon known as "lost-in-the-middle". Part of this gap may stem from the long-range decay of rotary position embedding (RoPE), which weakens the positional signal between distant tokens. Existing input compression deletes low-importance tokens, and attention reweighting acts only after RoPE has weakened this signal. RoPE-level methods modify positions directly but apply the same remapping to every query. To address these limitations, we propose QGPM (Query-Guided Positional Modulation), a training-free method that reshapes RoPE positions using query relevance from the model's own attention heads. QGPM builds a broad relevance map for lower layers and a sharp one for upper layers. Low-relevance tokens advance by less than one position on fast-rotating planes, pulling relevant evidence toward the query. Slow-rotating planes keep the original layout, with every token in place. Experiments on NoLiMa and RULER with multiple context lengths show that QGPM improves over base models and strong baselines, with the largest gains on retrieval tasks, and generalizes across multiple backbones. Our code is available at https://github.com/iZHpy/QGPM.

Figures & tables

Figure 1: Overview of QGPM. (a) A plain prefill reads the attention from the query to each context sentence in a few selected heads. The fused sentence ranking is shaped into a broad relevance map and a sharp score map. Context before the first relevant sentence is protected and keeps a score of 1. (b) Each map sets a positional stride for every token on every frequency plane. Low-relevance tokens advance by less than one position on fast planes. Slow planes, relevant sentences, and the query keep unit stride. Cumulative sums give the remapped positions. (c) In the answering pass, the lower layers use the broad map and the upper layers use the sharp map. Decoding continues in unit steps from the last position of each map.

	Qwen2.5-7B						Qwen3-4B
Method	NoLiMa	QA-1	QA-2	MK	MV	Avg.	NoLiMa	QA-1	QA-2	MK	MV	Avg.
Vanilla	28.5	79.6	57.0	94.3	83.5	68.6	16.1	77.5	56.6	99.6	96.0	69.2
LLMLingua-2	19.7	51.0	40.0	0 2.2	65.3	35.6	10.4	49.1	36.6	0 2.0	75.9	34.8
Ms-PoE	32.9	77.2	52.0	36.8	68.5	53.5	16.6	75.4	55.3	88.2	84.7	64.0
Self-Extend	29.5	79.3	57.5	78.8	82.6	65.5	18.1	76.5	57.5	99.8	92.2	68.8
STRING	33.1	79.4	57.8	92.3	76.6	67.8	22.9	78.8	60.0	99.3	93.0	70.8

Table 1: Accuracy (%) averaged over context lengths, from 4K to 32K for NoLiMa and from 4K to 64K for RULER. MK and MV denote NIAH-MK and NIAH-MV, and Avg. is the mean over the five tasks. Bold marks the best result for each backbone. † Includes 64K, which exceeds the configured window of Ministral-8B. Appendix E gives the results at each length.

Figure 2: Accuracy (%) of all five tasks at each context length. Gray lines show LLMLingua-2, Ms-PoE, and Self-Extend. The shaded region marks 64K for Ministral-8B, which exceeds the configured window of this backbone.

QGPM vs.	NoLiMa	NIAH	QA	All
Vanilla	16/0/0	31/6/3	24/2/14	71/8/17
LLMLingua-2	16/0/0	40/0/0	40/0/0	96/0/0
Ms-PoE	16/0/0	40/0/0	38/1/1	94/1/1
Self-Extend	16/0/0	34/3/3	26/2/12	76/5/15
STRING	16/0/0	38/2/0	22/2/16	76/4/16

Table 2: Wins, ties, and losses (W/T/L) of QGPM against each method over settings of backbone, task, and context length. The four columns cover 16, 40, 40, and 96 settings.

	Qwen2.5-7B			NoLiMa
Method	NoLiMa	QA-1	MV	Ministral	Granite
Vanilla	28.5	79.6	83.5	11.5	18.6
STRING	33.1	79.4	76.6	17.1	25.9
Top-K	41.1	80.6	91.1	n/a	n/a
Reorder	39.9	80.1	92.2	n/a	n/a
QGPM	46.1	80.4	93.4	24.1	35.6

Table 3: Average accuracy (%) of QGPM, input editing, and Oracle. MV denotes NIAH-MV, and n/a marks settings that are not run. Top-K and Reorder rely on Qwen3-Reranker-4B, whose tokenizer is not fully compatible with those of Ministral-8B and Granite-3.3-8B. Oracle (gray) uses evidence positions marked by hand, which are costly to obtain for every backbone. Bold marks the best result other than Oracle.

Figure 3: Accuracy (%) of QGPM, input editing, and Oracle at each context length on Qwen2.5-7B. Top-K and Reorder share one color as the two input editing variants. Oracle (dashed) uses annotated evidence and serves as a reference.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Layers	Heads	$\theta$	Window
Qwen2.5-7B	28	28 / 4	$10^{7}$	1,010,000
Qwen3-4B	36	32 / 8	$5\times 10^{6}$	262,144
Ministral-8B	36	32 / 8	$10^{8}$	32,768
Granite-3.3-8B	40	32 / 8	$10^{7}$	131,072

Appendix

Table 4: Backbone configurations from the HuggingFace releases. Heads lists query heads and key-value heads. Window is the configured maximum number of positions.

		Qwen2.5-7B						Qwen3-4B
Task	Method	4K	8K	16K	32K	64K	Avg.	4K	8K	16K	32K	64K	Avg.
NoLiMa	Vanilla	0 36.7	0 32.9	0 27.2	0 17.2	–	0 28.5	0 30.6	0 19.4	0 10.3	00 4.0	–	0 16.1
	LLMLingua-2	0 29.8	0 18.5	0 17.4	0 13.2	–	0 19.7	0 23.2	0 10.7	00 5.0	00 2.8	–	0 10.4
	Ms-PoE	0 41.2	0 36.6	0 32.2	0 21.7	–	0 32.9	0 32.2	0 19.9	00 9.5	00 4.6	–	0 16.6
	Self-Extend	0 36.2	0 31.7	0 27.1	0 22.8	–	0 29.5	0 30.7	0 19.9	0 13.9	00 8.0	–	0 18.1
	STRING	0 38.0	0 34.4	0 33.1	0 26.7	–	0 33.1	0 35.6	0 29.5	0 18.7	00 7.9	–	0 22.9

Appendix

Table 5: Accuracy (%) at each context length on Qwen2.5-7B and Qwen3-4B. NoLiMa is evaluated up to 32K, and Avg. is the mean over the evaluated lengths. Bold marks the best result for each backbone, task, and length.

		Ministral-8B						Granite-3.3-8B
Task	Method	4K	8K	16K	32K	64K †	Avg.	4K	8K	16K	32K	64K	Avg.
NoLiMa	Vanilla	0 25.8	0 13.0	00 4.5	00 2.6	–	0 11.5	0 31.8	0 21.7	0 12.9	00 7.9	–	0 18.6
	LLMLingua-2	0 15.9	00 6.1	00 1.9	00 0.8	–	00 6.2	0 22.8	0 16.9	0 10.3	00 6.5	–	0 14.1
	Ms-PoE	0 19.8	00 9.6	00 3.6	00 2.9	–	00 9.0	0 31.6	0 19.8	0 14.4	00 8.3	–	0 18.5
	Self-Extend	0 25.8	0 11.5	00 4.4	00 2.5	–	0 11.1	0 29.0	0 18.7	0 12.0	00 6.5	–	0 16.6
	STRING	0 32.9	0 19.8	0 10.6	00 5.1	–	0 17.1	0 40.3	0 30.0	0 20.8	0 12.4	–	0 25.9

Appendix

Table 6: Accuracy (%) at each context length on Ministral-8B and Granite-3.3-8B. NoLiMa is evaluated up to 32K, and Avg. is the mean over the evaluated lengths. Bold marks the best result for each backbone, task, and length. † Exceeds the configured window of Ministral-8B.

Task	Method	4K	8K	16K	32K	64K	Avg.
NoLiMa	Vanilla	36.7	32.9	27.2	17.2	–	28.5
	STRING	38.0	34.4	33.1	26.7	–	33.1
	Top-K	47.9	50.7	37.2	28.7	–	41.1
	Reorder	50.4	44.7	38.2	26.4	–	39.9
	QGPM	56.5	54.9	43.8	29.1	–	46.1
	Oracle	57.9	50.8	45.8	34.8	–	47.3

Appendix

Table 7: Accuracy (%) of QGPM, input editing, and Oracle at each context length on Qwen2.5-7B. Oracle (gray) is shown for reference. Bold marks the best result other than Oracle.

Lost-in-the-Middle Effect

See all 11 papers

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Apr 15, 2026Zichong Li, Chen Liang, Liliang Ren +3Long-Context Language Modeling Language Model Robustness
Work is done during internship at Microsoft. · Georgia Institute of Technology · Microsoft.

May 15, 2026J/K move · Enter open · S save

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Yufeng Du, Phillip Harris, Minyang Tian, Eliu A Huerta+4

University of Illinois at Urbana-Champaign · USA · University of Bonn+3

Rotary Positional Embeddings Self-Attention Long-Context Language Modeling Transformer Attention Positional Encoding

Open paper AskPDF arXiv

We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We prove that as context length increases, RoPE-based attention becomes unpredictable and loses two properties that are central to its effectiveness. First, it loses its locality bias: RoPE is no more likely to favor nearer positions than substantially farther ones. Second, it loses consistency in token relevance: a key vector that receives a higher attention score than an alternative at one position may receive a lower score at another. In both cases, the probability of failure approaches 0.5, no better than random guessing. We further prove that the attention score can remain unchanged when a key token is moved to a different position, or even replaced by a different token, indicating a failure to distinguish positions or tokens. Adjusting the RoPE base trades off distinguishing positions against distinguishing tokens but cannot preserve both at the same time. Increasing the RoPE base hyperparameter, a common practice in today's long-context models, helps distinguish different tokens, but inevitably sacrifices the ability to distinguish positions. Our empirical analysis shows that multi-head, multi-layer architectures are insufficient to overcome these limitations. Our findings suggest that fundamentally new mechanisms for encoding position and token order may be needed in future Transformer long-context language models.

Abstract

Lost-in-the-Middle Effect

Explore similar work

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation