We propose Parabolic Positional Encoding (PaPE), a parabola-based positional encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as from videos, event camera streams, images, or point clouds-our objective is to encode their positions while accounting for the characteristics of vision modalities. Prior works have largely extended positional encodings from 1D-sequences in language to nD-structures in vision, but only with partial account of vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. Extrapolation experiments on ImageNet-1K show how PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5 over the next-best encoding. Generality experiments on 8 datasets across 4 modalities show that PaPE is a general vision positional encoding, as PaPE matches the best baseline on 5 datasets and exceeds all on 2 datasets. The code is available at https://github.com/DTU-PAS/parabolic-position-encoding.
Figures & tables
Figure 2: Overview of Parabolic Positional Encoding (PaPE). PaPE decomposes attention (a) into distance (b), direction (c), and semantics (d). Using the dog’s eye as the query, PaPE learns to look in a bottom-right direction, while decaying attention with distance. The attention (a) is compatible with efficient attention kernels through separate query-key transformations. Colormap:
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
79.9
76.8
72.6
65.9
57.5
41.3
27.2
16.8
9.3
RoPE [ 41 ]
80.3
80.9
80.6
79.7
78.2
74.0
67.0
56.2
42.3
RoPE-Mixed [ 26 ]
80.0
80.4
79.2
77.1
73.6
61.5
43.4
27.1
17.1
nD-ALiBi [ 19 ]
80.0
78.4
77.8
76.6
75.3
72.2
68.3
64.1
59.1
LookHere [ 20 ]
80.5
81.4
81.1
80.4
79.7
77.5
74.2
69.6
64.4
PaPE (ours)
80.6
81.6
81.7
81.6
81.3
80.5
79.2
77.3
74.9
Table 1: PaPE extrapolates well beyond the training resolution. Training resolution is 2242 and evaluations are at resolutions up to 10242 . No additional training or modifications are applied. Best and next-best . PaPE extrapolates better than the next-best encoding by up to 10.5 .
Spatio-temporal
Image
Point cloud
Multimodal
Average
Positional Encoding
UCF101
DvsGesture
GEN1
ImageNet-1K
COCO
ScanNet
ModelNet40
nuScenes
Acc.
Acc.
mAP
Acc.
mAP
mIoU
Acc.
mAP
nD-sincos [ 48 ]
38.7
93.4
27.3
79.9
34.7
72.6
92.2
67.4
63.3
RoPE [ 41 ]
43.9
93.1
33.6
80.3
38.8
71.7
93.0
68.3
65.3
RoPE-Mixed [ 26 ]
41.2
83.3
31.7
80.0
38.0
71.1
93.0
66.3
63.1
nD-ALiBi [ 19 ]
41.6
89.2
28.8
80.0
33.9
70.6
93.2
68.5
63.2
Table 2: PaPE: a general vision positional encoding. Comparison of PaPE and baselines across 8 diverse datasets. Best and next-best . mAP is measured at 0.5:0.95:0.05 IoU intervals. PaPE matches the best baseline on 5 of 8 datasets and exceeds all on 2 datasets (UCF101 and GEN1), demonstrating the broad usefulness of PaPE across multiple vision modalities and tasks.
Dataset
nD-sincos
RoPE
RoPE-Mixed
nD-ALiBi
LookHere
PaPE (ours)
PaPE-RI (ours)
UCF101
39.5
45.2
41.3
41.9
-
49.5
41.8
DvsGesture
93.4
93.1
87.8
92.4
-
93.4
93.1
ImageNet-1K
79.6
80.2
80.1
80.0
80.4
80.6
79.8
Table 3: Parameter-matched. Increasing the parameters of the baselines to match the additional parameter counts introduced by PaPE does not change the ranking reported in Table 2 .
Dataset
nD-sincos
RoPE
RoPE-Mixed
nD-ALiBi
LookHere
PaPE (ours)
PaPE-RI (ours)
ImageNetV2 [ 35 ]
68.3
69.2
68.2
68.1
68.9
69.2
68.8
ImageNet-Renditions [ 25 ]
27.0
29.2
28.2
27.2
28.6
29.4
26.8
ObjectNet [ 5 ]
28.9
31.0
28.5
29.2
29.8
31.1
29.4
Table 4: Out-of-distribution generalization. PaPE does not degrade more than the baselines when evaluated on out-of-distribution versions of ImageNet.
Ablation
2242
3202
3842
4482
5122
6402
7682
8962
10242
PaPE
79.2
80.3
80.1
79.5
78.9
77.5
75.4
72.9
69.8
- ⟨ai,Δrij⊙2⟩
77.2
77.4
75.5
72.2
67.9
57.7
47.0
37.7
30.1
- ⟨bi,Δrij⟩
78.7
79.3
78.5
77.4
76.1
73.1
69.5
64.9
59.5
- Wa,Wb
78.4
78.9
78.4
77.6
76.6
74.2
71.4
68.1
64.0
- ai activation
78.6
79.7
79.4
78.4
76.9
71.9
63.5
50.7
36.0
- Wp
78.9
79.9
79.8
79.4
78.7
77.2
75.4
72.7
69.5
Table 5: Ablations on ImageNet-1K. All components of PaPE contribute to its accuracy as removing any component results in lower accuracy at the training resolution of 2242 . Wp does not influence the extrapolation performance, but we observe that all other components affect it, with distance decay (see ⟨ai,Δrij⊙2⟩ and ai activation) particularly driving much of the high extrapolation.
Training
Inference
Ablation
Params. (M)
VRAM (MB)
GFLOPS
Time (ms)
VRAM (MB)
GFLOPS
Time (ms)
PaPE
97.5
1901
119.7
14.6
433
37.6
1.6
- ⟨ai,Δrij⊙2⟩
92.0
1791
112.8
13.7
388
35.3
1.5
- ⟨bi,Δrij⟩
92.0
1791
112.8
13.8
388
37.4
1.5
- Wa,Wb
86.4
1673
106.6
12.9
363
35.3
1.4
- ai activation
97.5
1901
119.7
14.6
433
37.6
1.6
Table 6: Ablation efficiency. Measured on an RTX 4090 with a batch size of 1. See Appendix D for detailed setup. Context awareness ( Wa , Wb ) and Wp are the main contributors of PaPE’s moderate overhead. Together with the results in Table 5 , this shows that we can remove Wp to achieve high extrapolation without the moderate overhead.
m
2
4
8
16
32
50
64
Accuracy
78.7
78.4
79.0
79.1
78.9
79.2
79.0
Table 7: Ablation of m on ImageNet-1K. PaPE obtains the highest accuracy at m=50 . The differences are small and a low m may be preferable for increased efficiency.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
CPVT
CvT
ViT-PE
PaPE (ours)
UCF101
-
-
38.3 ± 3.1
49.5 ± 1.6
ImageNet-1K
79.2
78.9
79.2
80.6
COCO
32.1
34.1
14.7
38.9
Appendix
Table 8: Additional baselines. PaPE confirms its high generality when compared to context-aware baselines and learned positional embeddings from ViT.
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
CPVT
79.2
79.6
79.1
78.0
76.6
73.4
68.5
62.9
56.6
CvT
78.9
79.5
79.0
78.2
77.1
73.7
68.4
61.4
53.9
ViT-PE
79.2
79.1
78.1
76.6
75.1
71.7
67.4
62.6
57.3
PaPE (ours)
80.6
81.6
81.7
81.6
81.3
80.5
79.2
77.3
74.9
Appendix
Table 9: Extrapolation with additional baselines. PaPE remains best at extrapolation by large margins ranging from 17.6 to 21.0 at resolution 10242 .
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
79.9
79.9 (+3.1)
78.8 (+6.2)
77.4 (+11.5)
75.8 (+18.3)
71.8 (+30.5)
66.9 (+39.7)
61.3 (+44.5)
56.0 (+46.7)
RoPE [ 41 ]
80.3
80.6 (-0.3)
80.0 (-0.6)
78.9 (-0.8)
77.6 (-0.6)
74.5 (+0.5)
69.9 (+2.9)
64.8 (+8.6)
59.8 (+17.5)
RoPE-Mixed [ 26 ]
80.0
72.9 (-7.5)
72.1 (-7.1)
73.0 (-4.1)
67.8 (-5.8)
63.5 (+2.0)
56.1 (+12.7)
49.5 (+22.4)
42.9 (+25.8)
nD-ALiBi [ 19 ]
80.0
77.9 (-0.5)
77.2 (-0.6)
76.0 (-0.6)
74.4 (-0.9)
70.6 (-1.6)
65.4 (-2.9)
59.8 (-4.3)
53.9 (-5.2)
LookHere [ 20 ]
80.5
80.9 (-0.5)
80.2 (-0.9)
78.9 (-1.5)
77.4 (-2.3)
73.6 (-3.9)
68.8 (-5.4)
63.7 (-5.9)
59.0 (-5.4)
PaPE (ours)
80.6
80.7 (-0.9)
79.6 (-2.1)
78.1 (-3.5)
76.5 (-4.8)
72.0 (-8.5)
66.4 (-12.8)
60.4 (-16.9)
54.8 (-20.1)
Appendix
Table 10: Extrapolation results with position interpolation. The brackets indicate the increase ( green ) or decrease ( red ) compared to the corresponding configuration without interpolation in Table 1 . Best and next-best . nD-sincos benefits greatly from interpolation whereas RoPE and RoPE-Mixed benefit from resolution 6402 and above. Position interpolation is detrimental to all of the attention bias methods (nD-ALiBi, LookHere, PaPE, and PaPE-RI).
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
79.9
79.9
78.8
77.4
75.8
71.8
66.9
61.3
56.0
RoPE [ 41 ]
80.3
80.6
80.0
78.9
77.6
74.5
69.9
64.8
59.8
RoPE-Mixed [ 26 ]
80.0
72.9
72.1
73.0
67.8
63.5
56.1
49.5
42.9
nD-ALiBi [ 19 ]
80.0
78.4
77.8
76.6
75.3
72.2
68.3
64.1
59.1
LookHere [ 20 ]
80.5
81.4
81.1
80.4
79.7
77.5
74.2
69.6
64.4
PaPE (ours)
80.6
81.6
81.7
81.6
81.3
80.5
79.2
77.3
74.9
Appendix
Table 11: Extrapolation results for best-performing configuration per positional encoding. With position interpolation: nD-sincos, RoPE, and RoPE-Mixed. Without: nD-ALiBi, LookHere, PaPE, and PaPE-RI. Best and next-best . Although position interpolation is detrimental to PaPE, it remains the best encoding for classification extrapolation—even when applying position interpolation to the encodings that benefit from it.
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
68.3
64.2
58.1
50.7
42.3
27.1
16.6
10.1
5.6
RoPE [ 41 ]
69.2
70.3
69.7
67.9
65.9
60.1
51.8
41.2
29.1
RoPE-Mixed [ 26 ]
68.2
69.2
67.7
64.3
59.5
44.2
28.3
17.0
10.4
nD-ALiBi [ 19 ]
68.1
69.4
67.7
65.7
63.9
59.4
54.6
49.1
43.4
LookHere [ 20 ]
68.9
70.8
70.2
69.5
68.3
65.5
61.0
56.5
51.6
PaPE (ours)
69.2
70.9
71.2
70.6
70.1
68.9
66.7
64.5
61.5
Appendix
Table 12: Extrapolation on ImageNetV2. PaPE maintains its rank as the best positional encoding at extrapolating, increasing the accuracy at 10242 by 9.9 over the next-best (LookHere).
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
27.0
18.4
13.3
8.7
5.4
2.5
1.3
0.9
0.4
RoPE [ 41 ]
29.2
26.8
24.1
21.2
18.3
12.8
8.3
4.9
2.7
RoPE-Mixed [ 26 ]
28.2
24.3
20.2
16.4
12.4
6.3
2.6
1.3
0.8
nD-ALiBi [ 19 ]
27.2
24.5
21.2
18.3
16.2
12.8
10.2
8.1
6.2
LookHere [ 20 ]
28.6
26.4
23.4
21.1
18.8
15.3
12.1
9.5
7.5
PaPE (ours)
29.4
28.0
26.7
25.4
24.3
22.0
19.7
17.8
15.7
Appendix
Table 13: Extrapolation on ImageNet-Renditions. PaPE maintains its rank as the best positional encoding at extrapolating, increasing the accuracy at 10242 by 8.2 over the next-best (LookHere).
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
28.9
24.6
20.9
15.1
10.7
5.9
3.5
2.0
1.0
RoPE [ 41 ]
31.0
33.2
33.2
32.2
30.0
24.3
18.7
13.4
9.2
RoPE-Mixed [ 26 ]
28.5
29.8
28.2
25.6
22.0
13.8
7.6
4.5
2.9
nD-ALiBi [ 19 ]
29.2
31.6
30.2
28.9
26.9
23.0
20.0
16.9
14.9
LookHere [ 20 ]
29.8
32.3
32.0
31.2
29.9
27.1
24.4
21.6
19.1
PaPE (ours)
31.1
34.3
34.8
34.6
34.1
32.6
30.9
28.7
26.9
Appendix
Table 14: Extrapolation on ObjectNet. PaPE maintains its rank as the best positional encoding at extrapolating, increasing the accuracy at 10242 by 7.8 over the next-best (LookHere).
Positional Encoding
Fold 1
Fold 2
Fold 3
Mean ± std
nD-sincos [ 48 ]
39.3
41.1
38.1
39.5 ± 1.5
RoPE [ 41 ]
45.4
46.2
44.2
45.2 ± 1.0
RoPE-Mixed [ 26 ]
40.4
41.6
41.9
41.3 ± 0.8
nD-ALiBi [ 19 ]
41.4
43.9
40.3
41.9 ± 1.8
PaPE (ours)
51.4
48.2
49.1
49.5 ± 1.6
PaPE-RI (ours)
40.2
41.9
43.4
41.8 ± 1.6
Appendix
Table 15: Parameter-matched per-fold accuracy on UCF101. PaPE ( best ) confirms its position as highest scoring over RoPE ( next-best ). The difference is statistically significant at p=0.026.
Positional Encoding
2242
3202
3842
4482
5122
6402
7682
8962
10242
nD-sincos [ 48 ]
79.6
76.8
72.7
65.8
58.0
39.5
22.3
11.8
6.9
RoPE [ 41 ]
80.2
80.8
80.1
79.0
77.3
72.4
64.3
52.9
39.8
RoPE-Mixed [ 26 ]
80.1
80.2
78.3
74.8
69.9
55.2
36.9
22.9
14.7
nD-ALiBi [ 19 ]
80.0
80.4
79.9
79.0
77.9
75.2
71.2
65.6
59.1
LookHere [ 20 ]
80.4
81.5
81.3
80.8
80.1
78.2
75.2
70.6
65.3
PaPE (ours)
80.6
81.6
81.7
81.6
81.3
80.5
79.2
77.3
74.9
Appendix
Table 16: Parameter-matched extrapolation on ImageNet-1K. PaPE preserves its strong classification extrapolation as the best positional encoding with a 9.6 improvement in accuracy at 10242 over the next-best (LookHere).
Size
UCF101
DvsGesture
GEN1
ImageNet-1K
COCO
Reduced ( d=680)
50.8
93.8
34.9
80.7
38.8
Full ( d=768 )
49.5
93.4
34.8
80.6
38.9
Appendix
Table 17: Parameter-matched by reducing hidden size. Reducing the hidden size from d=768 to d=680 does not diminish the performance of PaPE, supporting that PaPE’s advantage stems from an improved positional encoding and not from increasing the number of parameters.
Positional Encoding
Fold 1
Fold 2
Fold 3
Mean ± std
nD-sincos [ 48 ]
37.8
39.1
39.3
38.7 ± 0.8
RoPE [ 41 ]
45.3
43.8
42.8
43.9 ± 1.2
RoPE-Mixed [ 26 ]
40.6
41.1
41.9
41.2 ± 0.7
nD-ALiBi [ 19 ]
41.2
41.5
42.1
41.6 ± 0.5
PaPE (ours)
51.4
48.2
49.1
49.5 ± 1.6
PaPE-RI (ours)
43.0
40.2
43.3
42.2 ± 1.7
Appendix
Table 18: UCF101 per-fold accuracy. PaPE ( best ) achieves noticeable gains over RoPE ( next-best ) on all folds. The difference is statistically significant at p=0.011.
m
2242
3202
3842
4482
5122
6402
7682
8962
10242
2
78.7
79.9
79.6
79.1
78.5
77.1
75.2
72.4
69.0
4
78.4
79.5
79.3
78.7
78.0
76.5
74.4
71.6
68.2
8
79.0
79.7
79.4
78.9
78.1
76.4
74.3
71.6
68.2
16
79.1
80.1
80.0
79.5
78.8
77.4
75.3
72.8
69.8
32
78.9
80.1
79.8
79.4
78.7
77.4
75.5
72.8
69.7
50
79.2
80.3
80.1
79.5
78.9
77.5
75.4
72.9
69.8
Appendix
Table 19: Ablation of m with extrapolation. We follow the ablation setup in Section 5.6 . m does not have a large effect on extrapolation.
Training
Inference
Method
Total Params.
Pos. Params.
VRAM (MB)
GFLOPS
Time (ms)
VRAM (MB)
GFLOPS
Time (ms)
nD-sincos
86.4M
0
1669
105.9
12.5
362
35.1
1.4
nD-ALiBi
86.4M
0
1673
105.2
12.5
365
35.1
1.4
RoPE
86.4M
0
1669
105.9
12.6
363
35.1
1.4
RoPE-Mixed
86.4M
18.4K
1671
105.9
12.7
363
35.1
1.4
LookHere
86.4M
0
1694
105.2
12.6
387
35.1
1.4
Appendix
Table 20: Efficiency. Resource usage is averaged over 1000 samples with a batch size of 1, using 2242 resolution images from ImageNet-1K. PaPE with m=50 produces a moderate resource increase from 7% to 20%.
Figure 3: Model analysis on ImageNet-1K. Red ( z>0 ) highlights heads that lean heavily on positional information, while blue ( z<0 ) marks heads that prioritize semantic content in deciding what to attend to. Positions are used most strongly in early layers.
Parameter
UCF101
DvsGesture
GEN1
ImageNet-1K
COCO
ScanNet
ModelNet40
nuScenes
Batch size
32
32
64
1024
64
12
32
3
Learning rate
0.00003
0.00005
0.0001
0.0006
0.0006
0.006
0.001
0.003
Weight decay
0.05
0.05
0.05
0.05
0.05
0.05
0.01
0.03
Momentum ( B1)
0.9
0.9
0.9
0.9
0.9
0.9
0.9
0.9
Momentum ( B2)
0.999
0.999
0.999
0.999
0.999
0.999
0.999
0.99
Warmup epochs
20
15
7.5
15
15
40
5
1
Appendix
Table 21: Hyperparameters. The hyperparameters are shared for all positional encodings for a given dataset.
Positional embeddings (PEs) in Vision Transformers (ViTs) are known to impact performance and robustness, but their role in shaping internal spatial representations is not well understood. In this work, we study how different forms of PEs influence the representational geometry of ViTs and how these changes relate to robustness under content-disrupting distribution shifts. We introduce a metric, the Spatial Similarity Distance Correlation (SSDC), to quantify spatial structure in token representations. Using this metric, we show that ViTs trained without PEs still develop non-trivial spatial structure, but this structure is driven by visual content and collapses under token permutation. In contrast, we find that all PEs considered (learned absolute, sinusoidal, and rotary) are associated with a consistent shift toward an index-anchored spatial organization. Representations in these models remain stable under perturbations that disrupt content, and exhibit substantially improved robustness to such distributional shifts. We further show that while different PEs produce distinct depth-wise trajectories of spatial structure, their robustness properties are largely similar (with secondary variation across encoding schemes), suggesting that robustness appears to depend on the presence of a stable positional reference frame more than it depends on the specific encoding mechanism. These results offer a geometric account of how positional encodings shape internal representations, with implications for the principled design of future encoding schemes.
Vision Transformers have achieved remarkable success in computer vision, but their common use of learnable one-dimensional positional encodings weakens the inherent two-dimensional spatial structure of images after patch flattening. Existing positional encodings often lack geometric constraints and do not preserve a monotonic relationship between Euclidean spatial distances and sequential index distances, limiting ViTs' ability to exploit spatial proximity priors. Motivated by the usefulness of periodicity in positional encoding, we propose Weierstrass elliptic Positional Encoding (WePE), a mathematically grounded method for encoding two-dimensional coordinates in the complex domain. WePE maps normalized 2D patch coordinates onto the complex plane and constructs compact four-dimensional positional features using the Weierstrass elliptic function and its derivative. The double periodicity provides a principled representation of 2D positions, and its intrinsic lattice structure naturally matches the regular geometry of image patch grids. Its nonlinear geometric properties help model spatial distance relationships more faithfully, while the algebraic addition formula enables relative positional information between arbitrary patch pairs to be derived directly from their absolute encodings. WePE is plug-and-play and resolution-agnostic, allowing seamless integration into existing ViTs. Extensive experiments show that WePE brings consistent performance gains in most settings. With precomputed lookup tables, these improvements introduce no noticeable computational or memory overhead. Additional analyses and ablation studies further validate the effectiveness of the proposed method.
Zhihang Xin, Rui Wang, Xitong Hu +1
School of Mathematics and Data Science, Jiangnan University, Wuxi 214122, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China
We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).