Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

Mohammad Aflah Khan; Krishna P. Gummadi; Manish Gupta; Abhilasha Ravichander

cs.LGMar 12, 2026

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

Authors: Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander

Organizations: Max Planck Institute for Software Systems · Microsoft, Hyderabad

Abstract

Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10 $\times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability.

Figures & tables

Figure 1: Estimated RoPE cache memory usage as a function of sequence length. Applying RoPE to only a fraction of hidden dimensions substantially reduces cache size, yielding significant savings at long context lengths and in resource-constrained settings such as edge devices. More details are described in Appendix C .

Panel of Figure 2 — Figure 2: Training loss trajectories on the FineWeb dataset for sequential models with varying RoPE configurations.

Panel of Figure 3 — Figure 3: Training loss trajectories on the FineWeb-Edu dataset comparing sequential and parallel attention architectures under varying Partial RoPE configurations.

Figure 4: Training loss trajectories for the 8B model with varying Partial RoPE configurations.

Figure 5: Benchmark performance across tasks for the Sequential 1B model trained on FineWeb (Seq. Len. 2048)

Panel of Figure 6 — Figure 6: Training loss trajectories for parallel attention models trained on FineWeb-Edu under different experimental settings. The left panel varies random seeds, while the right panel varies learning rates.

Panel of Figure 7 — Figure 7: Comparison of training loss trajectories with and without QK-Norm across different 1B parameter models.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model (Head Dim)	1%	4%	10% / 10.2%	25%	50%	75%	100%
Pythia-1B (256)	2	–	26	64	128	192	256
Llama-1B (64)	–	2	6	16	32	48	64
Llama-8B (128)	–	–	12	32	64	96	128

Appendix

Table 1: Number of rotated dimensions used for each RoPE percentage configuration. Values are adjusted to ensure an even number of dimensions since RoPE rotates hidden states pairwise.

Model	URL
EleutherAI/gpt-j-6b	hf.co/EleutherAI/gpt-j-6b/blob/main/config.json#L21
EleutherAI/gpt-neox-20b	hf.co/EleutherAI/gpt-neox-20b/blob/main/config.json#L19
EleutherAI/pythia-12b	hf.co/EleutherAI/pythia-12b/blob/main/config.json#L17
microsoft/phi-2	hf.co/microsoft/phi-2/blob/main/config.json#L20
nvidia/Nemotron-4-340B-Base	hf.co/nvidia/Nemotron-4-340B-Base/blob/main/model_config.yaml#L34
Qwen/Qwen3-Next-80B-A3B-Instruct	hf.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob/main/config.json#L31

Appendix

Table 2: Examples of widely used open-weight/open-source models employing Partial RoPE , with references indicating this choice. Parameter names vary across models due to differences in training frameworks. Note: For Pythia, we show one family member, though all members use the same 25% partial RoPE.

Llama-3.2-1B x FW-Edu x Seq. Len. 2048
Config	ARC Challenge	ARC Easy	LogiQA	LAMBADA (OpenAI)	LAMBADA (Standard)	PIQA	SciQ	PubMedQA	Wino -Grande	WSC
0% RoPE (NoPE)	0.31 (0.01)	0.57 (0.01)	0.28 (0.02)	0.36 (0.01)	0.27 (0.01)	0.70 (0.01)	0.75 (0.01)	0.56 (0.02)	0.51 (0.01)	0.55 (0.05)
4% RoPE	0.31 (0.01)	0.58 (0.01)	0.28 (0.02)	0.36 (0.01)	0.27 (0.01)	0.69 (0.01)	0.75 (0.01)	0.51 (0.02)	0.52 (0.01)	0.38 (0.05)
10% RoPE	0.32 (0.01)	0.60 (0.01)	0.28 (0.02)	0.38 (0.01)	0.28 (0.01)	0.71 (0.01)	0.75 (0.01)	0.57 (0.02)	0.54 (0.01)	0.44 (0.05)
25% RoPE	0.31 (0.01)	0.58 (0.01)	0.26 (0.02)	0.38 (0.01)	0.29 (0.01)	0.70 (0.01)	0.75 (0.01)	0.52 (0.02)	0.53 (0.01)	0.54 (0.05)
50% RoPE	0.32 (0.01)	0.59 (0.01)	0.26 (0.02)	0.37 (0.01)	0.29 (0.01)	0.71 (0.01)	0.76 (0.01)	0.55 (0.02)	0.53 (0.01)	0.63 (0.05)

Appendix

Table 3: MCQ Benchmark Results. We run evaluations through EleutherAI’s Language Model Evaluation Harness ( Gao et al., 2023 ) under Zero-Shot setting.

Config	LAMBADA (OpenAI)	LAMBADA (Standard)	LAMBADA
Llama-3.2-1B x FW-Edu x 2048
0% RoPE (NoPE)	26.72 (0.96)	69.87 (2.90)	48.30 (11.00)
4% RoPE	27.71 (1.00)	68.78 (2.78)	48.25 (10.48)
10% RoPE	23.25 (0.82)	64.50 (2.63)	43.87 (10.50)
25% RoPE	23.51 (0.83)	64.78 (2.75)	44.14 (10.52)
50% RoPE	23.89 (0.85)	64.87 (2.75)	44.38 (10.45)

Appendix

Table 4: Perplexity results for LAMBADA. We run evaluations through EleutherAI’s Language Model Evaluation Harness ( Gao et al., 2023 ) under Zero-Shot setting.

Rotary Positional Embeddings

Rotary positional embeddings (RoPE) encode token positions by rotating attention features, supporting relative-position reasoning in language and multimodal models.

See all 98 papers

WavePrune: One period is often enough for RoPE
Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional Embeddings Positional Encoding
Tsinghua University
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Jun 5, 2026Shaowen Wang, Yuke Zheng, Tansheng Zhu +4Rotary Positional Embeddings Self-Attention
Tsinghua University · Hunyuan Team, Tencent · Xiongan AI Institute +1
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Sep 28, 2026Jarod Lévy, Mathurin Videau, Jad Yehya +3Rotary Positional Embeddings Long-Context Language Modeling
Meta AI, Paris · Inria, Université Paris-Saclay, Palaiseau, France

Oct 3, 2026J/K move · Enter open · S save

WavePrune: One period is often enough for RoPE

Guancheng Du, Luotian Huang, Shaowen Wang, Si Li+1

Tsinghua University

Rotary Positional Embeddings Positional Encoding Relative Positional Encoding Long-Context Language Modeling LLM Inference Acceleration

Open paper AskPDF arXiv

Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.

Abstract

Rotary Positional Embeddings

Explore similar work

WavePrune: One period is often enough for RoPE

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings