Frayed RoPE and Long Inputs: A Geometric Perspective

Davis Wertheimer; Aozhong Zhang; Derrick Liu; Penghang Yin; Naigang Wang

cs.LGFeb 24, 2026

Frayed RoPE and Long Inputs: A Geometric Perspective

Authors: Davis Wertheimer, Aozhong Zhang, Derrick Liu, Penghang Yin, Naigang Wang

Organizations: IBM Research · University at Albany, SUNY

Abstract

Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate ``out of distribution,'' but it is not clear how extra rotation relates to or causes pathological behavior. Through empirical and theoretical analysis we advance a unified geometric understanding of attention behavior with RoPE. We find that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens: placeholders that allow attention heads to avoid token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological behavior by inhibiting sink token functionality. From this geometric perspective, we propose RoPE-ID (In Distribution), a straightforward modification that allows attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to a subset of channels. We demonstrate the effectiveness of RoPE-ID for extended inputs using 1B and 3B parameter Transformers on the LongBench and RULER information retrieval benchmarks.

Figures & tables

Figure 1: A 2D diagram of our observed latent geometry. Left: Keys/queries cluster tightly into opposing point clouds with negative dot products. The sink token has low norm and thus the greatest dot product. Matched key/query pair $q,k$ align orthogonally, letting their product approach and exceed the sink’s. Right: RoPE on long inputs makes keys/queries disperse and overlap, causing spurious alignment. Sink token stops functioning.

Figure 2: Effect of RoPE across context length on pairwise angular distances within heads for Llama3-8B, Gemma-7B and OLMo-7B.

Figure 3: 2D PCA projections of Llama3 representations under different context lengths and RoPE settings (3rd key head of layer 21 and its queries). RoPE at long contexts destroys cluster separation, and increases stable rank (in parentheses) with sequence length, consistent with Theorem 1 .

Figure 4: Left : Histogram across layers and heads showing the percentage of variance (relative to origin) explained by the first principal component of latent key/query clusters. Middle : Ratio of first singular value (FSV) after and before RoPE. Blue lines plot individual key/query heads, red plots the average trend. RoPE shrinks the FSV, but accelerates beyond the training length, producing cluster dispersal. Right : Mean stable rank of key clusters by input length, showing monotonic increase with RoPE, indicating cluster dispersal. Error bars denote standard deviation.

Figure 5: Left: Key $\ell_{2}$ norm as a function of position in sequence. Sink token is consistently small. Right: Keys have low dot product against subsequent queries in expectation, except for the first and most recent tokens. Scores are normalized by the highest value, in this case always the sink.

Figure 6: Left: Sink token attention weight vs. input length. With RoPE, sink-token attention decays to zero beyond the training length; without RoPE it remains stable. Right: Maximum Query–Key dot product vs. input length. The max QK product increases with length only when RoPE is active.

Figure 7: Expected RoPE behavior for our proposed method and three baselines. Dotted line indicates hypothetical training length of 4k and head dimension is 128. Left: Repeats the Fig. 4 singular value ratio before/after RoPE, for a synthetic point cloud of ones vectors. Right: Long-term RoPE decay for the same techniques, showing similar behaviors.

	Llama-1B			Llama-3B
Method	4k	8k	16k	4k	8k	16k
RoPE	39.72	0.01	0.03	46.19	0.14	0.01
High frequency	16.04	7.60	2.37	28.02	14.31	5.14
HalfRoPE	43.07	0.14	0	51.28	0.4	0.03
YaRN	40.24	35.55	30.25	43.90	45.09	40.14
RoPE-ID	39.15	29.71	14.29	44.86	37.51	21.95

Table 1: Average RULER benchmark accuracy scores by sequence length. Highest average score for each sequence length and model size is in bold ; runner-up is underlined .

	Llama-1B			Llama-3B
Method	4k	8k	16k	4k	8k	16k
RoPE	14.61	8.23	8.73	18.62	11.36	10.42
High frequency	11.8	11.44	11.04	14.19	13.82	13.78
HalfRoPE	15.38	8.73	8.86	19.42	10.7	10.62
YaRN	14.84	14.54	14.09	15.87	19.29	19.63
RoPE-ID (scaling)	15.83	15.83	15.80	15.92	17.13	17.94

Table 2: LongBench accuracy scores, averaged over 14 English tasks, by sequence length. Highest average score for each sequence length and model size is in bold ; runner-up is underlined .

	Llama-1B				Llama-3B
Method	ARC-C	HellaSwag	PIQA	Avg.	ARC-C	HellaSwag	PIQA	Avg.
RoPE	25.77	44.00	69.26	46.34	29.18	53.95	72.74	51.96
High frequency	25.17	42.99	69.26	45.81	29.61	53.32	71.87	51.60
HalfRoPE	26.45	44.00	68.77	46.41	32.17	53.87	72.03	52.69
YaRN	25.60	41.89	68.61	45.37	29.10	52.46	72.20	51.25
RoPE-ID	25.60	43.58	68.88	46.02	30.03	53.95	72.25	52.08

Table 3: Standard evaluation (accuracy) on common sense reasoning tasks

Figure 8: Average RULER accuracy scores for fine-tuned 1b models. RoPE-ID and YaRN trade off performance over length, but demonstrate surprising synergy when applied jointly.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Attention patterns of three heads in LLaMA3-8B. Sink token behavior is clearly observed, even when performing non-trivial token mixing.

Figure 10: Mean of the pairwise dot product of query and key values across all heads for Llama3-8B, Gemma-7B and OLMo-7B which showcases the effect of RoPE across various context lengths showing similar trend as Fig. 2

Figure 11: Silhouette Score (left) and Davies-Bouldin Index (right) showcasing the effect of RoPE on Internal Representation Clustering for Llama3-8B-Instruct. Lower Silhouette Score and higher Davies-Bouldin Index represents more overlap.

Parameters	1B	3B
Vocab	128256	128256
Width	1280	2048
Layers	32	48
Heads	16	16
KV heads	4	4
Head dim	80	128

Appendix

Table 4: Model architectures used for pretraining and evaluation

Figure 12: Llama 1B ablation studies with average RULER scores on the y-axis. The x-axis of the 4 plots cover fraction of channels, wavelength (high frequency), wavelength (low frequency) and temperature scaling, respectively. The color coded legend represents context lengths 4k, 8k and 16k.

Llama-1B Models
Fraction of
Channels	4K	8K	16K
1/8	46.30	35.12	2.18
1/4	41.59	38.75	36.41
3/8	39.11	32.83	27.12
1/2	39.15	35.64	30.83

Appendix

Table 5: RULER scores for LlaMA 1B ablations as function of fraction of channels, wavelength, and temperature scaling

1B Models
Method	Seq.	N-S1	N-S2	N-S3	N-MK1	N-MK2	N-MK3	N-MV	N-MQ	VT	CWE	FWE	QA-1	QA-2	Avg.
RoPE	4k	100	100	96.6	73.6	2.4	3.4	23.45	24.65	10.24	21.76	27.4	22.67	10.24	39.72
	8k	0	0	0	0	0	0	0	0	0	0	0	0.13	0	0.01
	16k	0	0	0	0	0	0	0	0	0	0	0	0.4	0	0.03
High Frequency	4k	42	44.8	35	37.4	0.2	3.6	7.05	4.85	0	0.1	1	16.95	15.6	16.04
	8k	18.8	15.6	15	19.4	0	0.4	6.35	1.2	0	0.04	2.87	6.13	13	7.60

Appendix

Table 6: Performance of different methods on the RULER benchmark. Highest average score for each sequence length and model size is in bold ; runner-up is underlined .

Methods	Seq.	Single-Doc QA	Multi-Doc QA	Summarization	Few-Shot Learning	Code	Avg.
1B Models
RoPE	4k	6.65	4.32	14.22	27.27	23.55	14.61
	8k	4.53	1.69	11.48	7.55	19.75	8.23
	16k	4.84	2.08	12.74	8.19	19.33	8.73
High Frequency	4k	5.26	4.22	12.97	19.3	19.96	11.8
	8k	5.31	4.22	12.61	17.61	20.47	11.44

Appendix

Table 7: Performance of different methods on LongBench, averaged by task type. Highest total average score for each sequence length and model size is in bold ; runner-up is underlined .

Llama-1B Models
Fine-Tuning	YaRN	RoPE-ID	RoPE-ID
			+ YaRN
4K	43.29	39.49	47.09
8K	39.27	35.81	41.94
16K	34.55	30.63	38.59
32K	32.74	30.66	36.16

Appendix

Table 8: RULER scores for LlaMA 1B models up to 128k context length, where experiments cover tuning with YaRN, RoPE-ID, and RoPE-ID + YaRN.

Figure 13: Pairwise angular distances within- and between-clusters for our trained models. Base model matches prior LLM observations, including an inflection point for key-query product with RoPE at the training length (4k). RoPE-ID maintains stable behavior over time.

Figure 14: Clustering behavior for our trained models. Vanilla transformer (top) matches prior LLM observations; RoPE-ID (bottom) maintains stable cluster separation.

Rotary Positional Embeddings

Rotary positional embeddings (RoPE) encode token positions by rotating attention features, supporting relative-position reasoning in language and multimodal models.

See all 98 papers

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
WavePrune: One period is often enough for RoPE
Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional Embeddings Positional Encoding
Tsinghua University

May 15, 2026J/K move · Enter open · S save

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Yufeng Du, Phillip Harris, Minyang Tian, Eliu A Huerta+4

University of Illinois at Urbana-Champaign · USA · University of Bonn+3

Rotary Positional Embeddings Self-Attention Long-Context Language Modeling Transformer Attention Positional Encoding

Open paper AskPDF arXiv

We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We prove that as context length increases, RoPE-based attention becomes unpredictable and loses two properties that are central to its effectiveness. First, it loses its locality bias: RoPE is no more likely to favor nearer positions than substantially farther ones. Second, it loses consistency in token relevance: a key vector that receives a higher attention score than an alternative at one position may receive a lower score at another. In both cases, the probability of failure approaches 0.5, no better than random guessing. We further prove that the attention score can remain unchanged when a key token is moved to a different position, or even replaced by a different token, indicating a failure to distinguish positions or tokens. Adjusting the RoPE base trades off distinguishing positions against distinguishing tokens but cannot preserve both at the same time. Increasing the RoPE base hyperparameter, a common practice in today's long-context models, helps distinguish different tokens, but inevitably sacrifices the ability to distinguish positions. Our empirical analysis shows that multi-head, multi-layer architectures are insufficient to overcome these limitations. Our findings suggest that fundamentally new mechanisms for encoding position and token order may be needed in future Transformer long-context language models.

Abstract

Rotary Positional Embeddings

Explore similar work

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Remember to Forget: Gated Adaptive Positional Encoding

WavePrune: One period is often enough for RoPE