Parabolic Positional Encoding: A Vision-Centric, Principled, Extrapolatable, and General Approach
Organizations: Technical University of Denmark · KTH Royal Institute of Technology
Abstract
We propose Parabolic Positional Encoding (PaPE), a parabola-based positional encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as from videos, event camera streams, images, or point clouds-our objective is to encode their positions while accounting for the characteristics of vision modalities. Prior works have largely extended positional encodings from 1D-sequences in language to nD-structures in vision, but only with partial account of vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. Extrapolation experiments on ImageNet-1K show how PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5 over the next-best encoding. Generality experiments on 8 datasets across 4 modalities show that PaPE is a general vision positional encoding, as PaPE matches the best baseline on 5 datasets and exceeds all on 2 datasets. The code is available at https://github.com/DTU-PAS/parabolic-position-encoding.
Figures & tables
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 79.9 | 76.8 | 72.6 | 65.9 | 57.5 | 41.3 | 27.2 | 16.8 | 9.3 |
| RoPE [ 41 ] | 80.3 | 80.9 | 80.6 | 79.7 | 78.2 | 74.0 | 67.0 | 56.2 | 42.3 |
| RoPE-Mixed [ 26 ] | 80.0 | 80.4 | 79.2 | 77.1 | 73.6 | 61.5 | 43.4 | 27.1 | 17.1 |
| nD-ALiBi [ 19 ] | 80.0 | 78.4 | 77.8 | 76.6 | 75.3 | 72.2 | 68.3 | 64.1 | 59.1 |
| LookHere [ 20 ] | 80.5 | 81.4 | 81.1 | 80.4 | 79.7 | 77.5 | 74.2 | 69.6 | 64.4 |
| PaPE (ours) | 80.6 | 81.6 | 81.7 | 81.6 | 81.3 | 80.5 | 79.2 | 77.3 | 74.9 |
| Spatio-temporal | Image | Point cloud | Multimodal | Average | |||||
| Positional Encoding | UCF101 | DvsGesture | GEN1 | ImageNet-1K | COCO | ScanNet | ModelNet40 | nuScenes | |
| Acc. | Acc. | mAP | Acc. | mAP | mIoU | Acc. | mAP | ||
| nD-sincos [ 48 ] | 38.7 | 93.4 | 27.3 | 79.9 | 34.7 | 72.6 | 92.2 | 67.4 | 63.3 |
| RoPE [ 41 ] | 43.9 | 93.1 | 33.6 | 80.3 | 38.8 | 71.7 | 93.0 | 68.3 | 65.3 |
| RoPE-Mixed [ 26 ] | 41.2 | 83.3 | 31.7 | 80.0 | 38.0 | 71.1 | 93.0 | 66.3 | 63.1 |
| nD-ALiBi [ 19 ] | 41.6 | 89.2 | 28.8 | 80.0 | 33.9 | 70.6 | 93.2 | 68.5 | 63.2 |
| Dataset | nD-sincos | RoPE | RoPE-Mixed | nD-ALiBi | LookHere | PaPE (ours) | PaPE-RI (ours) |
| UCF101 | 39.5 | 45.2 | 41.3 | 41.9 | - | 49.5 | 41.8 |
| DvsGesture | 93.4 | 93.1 | 87.8 | 92.4 | - | 93.4 | 93.1 |
| ImageNet-1K | 79.6 | 80.2 | 80.1 | 80.0 | 80.4 | 80.6 | 79.8 |
| Dataset | nD-sincos | RoPE | RoPE-Mixed | nD-ALiBi | LookHere | PaPE (ours) | PaPE-RI (ours) |
| ImageNetV2 [ 35 ] | 68.3 | 69.2 | 68.2 | 68.1 | 68.9 | 69.2 | 68.8 |
| ImageNet-Renditions [ 25 ] | 27.0 | 29.2 | 28.2 | 27.2 | 28.6 | 29.4 | 26.8 |
| ObjectNet [ 5 ] | 28.9 | 31.0 | 28.5 | 29.2 | 29.8 | 31.1 | 29.4 |
| Ablation | |||||||||
| PaPE | 79.2 | 80.3 | 80.1 | 79.5 | 78.9 | 77.5 | 75.4 | 72.9 | 69.8 |
| - | 77.2 | 77.4 | 75.5 | 72.2 | 67.9 | 57.7 | 47.0 | 37.7 | 30.1 |
| - | 78.7 | 79.3 | 78.5 | 77.4 | 76.1 | 73.1 | 69.5 | 64.9 | 59.5 |
| - | 78.4 | 78.9 | 78.4 | 77.6 | 76.6 | 74.2 | 71.4 | 68.1 | 64.0 |
| - activation | 78.6 | 79.7 | 79.4 | 78.4 | 76.9 | 71.9 | 63.5 | 50.7 | 36.0 |
| - | 78.9 | 79.9 | 79.8 | 79.4 | 78.7 | 77.2 | 75.4 | 72.7 | 69.5 |
| Training | Inference | ||||||
| Ablation | Params. (M) | VRAM (MB) | GFLOPS | Time (ms) | VRAM (MB) | GFLOPS | Time (ms) |
| PaPE | 97.5 | 1901 | 119.7 | 14.6 | 433 | 37.6 | 1.6 |
| - | 92.0 | 1791 | 112.8 | 13.7 | 388 | 35.3 | 1.5 |
| - | 92.0 | 1791 | 112.8 | 13.8 | 388 | 37.4 | 1.5 |
| - | 86.4 | 1673 | 106.6 | 12.9 | 363 | 35.3 | 1.4 |
| - activation | 97.5 | 1901 | 119.7 | 14.6 | 433 | 37.6 | 1.6 |
| m | 2 | 4 | 8 | 16 | 32 | 50 | 64 |
| Accuracy | 78.7 | 78.4 | 79.0 | 79.1 | 78.9 | 79.2 | 79.0 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | CPVT | CvT | ViT-PE | PaPE (ours) |
| UCF101 | - | - | 38.3 3.1 | 49.5 1.6 |
| ImageNet-1K | 79.2 | 78.9 | 79.2 | 80.6 |
| COCO | 32.1 | 34.1 | 14.7 | 38.9 |
| Positional Encoding | |||||||||
| CPVT | 79.2 | 79.6 | 79.1 | 78.0 | 76.6 | 73.4 | 68.5 | 62.9 | 56.6 |
| CvT | 78.9 | 79.5 | 79.0 | 78.2 | 77.1 | 73.7 | 68.4 | 61.4 | 53.9 |
| ViT-PE | 79.2 | 79.1 | 78.1 | 76.6 | 75.1 | 71.7 | 67.4 | 62.6 | 57.3 |
| PaPE (ours) | 80.6 | 81.6 | 81.7 | 81.6 | 81.3 | 80.5 | 79.2 | 77.3 | 74.9 |
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 79.9 | 79.9 (+3.1) | 78.8 (+6.2) | 77.4 (+11.5) | 75.8 (+18.3) | 71.8 (+30.5) | 66.9 (+39.7) | 61.3 (+44.5) | 56.0 (+46.7) |
| RoPE [ 41 ] | 80.3 | 80.6 (-0.3) | 80.0 (-0.6) | 78.9 (-0.8) | 77.6 (-0.6) | 74.5 (+0.5) | 69.9 (+2.9) | 64.8 (+8.6) | 59.8 (+17.5) |
| RoPE-Mixed [ 26 ] | 80.0 | 72.9 (-7.5) | 72.1 (-7.1) | 73.0 (-4.1) | 67.8 (-5.8) | 63.5 (+2.0) | 56.1 (+12.7) | 49.5 (+22.4) | 42.9 (+25.8) |
| nD-ALiBi [ 19 ] | 80.0 | 77.9 (-0.5) | 77.2 (-0.6) | 76.0 (-0.6) | 74.4 (-0.9) | 70.6 (-1.6) | 65.4 (-2.9) | 59.8 (-4.3) | 53.9 (-5.2) |
| LookHere [ 20 ] | 80.5 | 80.9 (-0.5) | 80.2 (-0.9) | 78.9 (-1.5) | 77.4 (-2.3) | 73.6 (-3.9) | 68.8 (-5.4) | 63.7 (-5.9) | 59.0 (-5.4) |
| PaPE (ours) | 80.6 | 80.7 (-0.9) | 79.6 (-2.1) | 78.1 (-3.5) | 76.5 (-4.8) | 72.0 (-8.5) | 66.4 (-12.8) | 60.4 (-16.9) | 54.8 (-20.1) |
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 79.9 | 79.9 | 78.8 | 77.4 | 75.8 | 71.8 | 66.9 | 61.3 | 56.0 |
| RoPE [ 41 ] | 80.3 | 80.6 | 80.0 | 78.9 | 77.6 | 74.5 | 69.9 | 64.8 | 59.8 |
| RoPE-Mixed [ 26 ] | 80.0 | 72.9 | 72.1 | 73.0 | 67.8 | 63.5 | 56.1 | 49.5 | 42.9 |
| nD-ALiBi [ 19 ] | 80.0 | 78.4 | 77.8 | 76.6 | 75.3 | 72.2 | 68.3 | 64.1 | 59.1 |
| LookHere [ 20 ] | 80.5 | 81.4 | 81.1 | 80.4 | 79.7 | 77.5 | 74.2 | 69.6 | 64.4 |
| PaPE (ours) | 80.6 | 81.6 | 81.7 | 81.6 | 81.3 | 80.5 | 79.2 | 77.3 | 74.9 |
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 68.3 | 64.2 | 58.1 | 50.7 | 42.3 | 27.1 | 16.6 | 10.1 | 5.6 |
| RoPE [ 41 ] | 69.2 | 70.3 | 69.7 | 67.9 | 65.9 | 60.1 | 51.8 | 41.2 | 29.1 |
| RoPE-Mixed [ 26 ] | 68.2 | 69.2 | 67.7 | 64.3 | 59.5 | 44.2 | 28.3 | 17.0 | 10.4 |
| nD-ALiBi [ 19 ] | 68.1 | 69.4 | 67.7 | 65.7 | 63.9 | 59.4 | 54.6 | 49.1 | 43.4 |
| LookHere [ 20 ] | 68.9 | 70.8 | 70.2 | 69.5 | 68.3 | 65.5 | 61.0 | 56.5 | 51.6 |
| PaPE (ours) | 69.2 | 70.9 | 71.2 | 70.6 | 70.1 | 68.9 | 66.7 | 64.5 | 61.5 |
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 27.0 | 18.4 | 13.3 | 8.7 | 5.4 | 2.5 | 1.3 | 0.9 | 0.4 |
| RoPE [ 41 ] | 29.2 | 26.8 | 24.1 | 21.2 | 18.3 | 12.8 | 8.3 | 4.9 | 2.7 |
| RoPE-Mixed [ 26 ] | 28.2 | 24.3 | 20.2 | 16.4 | 12.4 | 6.3 | 2.6 | 1.3 | 0.8 |
| nD-ALiBi [ 19 ] | 27.2 | 24.5 | 21.2 | 18.3 | 16.2 | 12.8 | 10.2 | 8.1 | 6.2 |
| LookHere [ 20 ] | 28.6 | 26.4 | 23.4 | 21.1 | 18.8 | 15.3 | 12.1 | 9.5 | 7.5 |
| PaPE (ours) | 29.4 | 28.0 | 26.7 | 25.4 | 24.3 | 22.0 | 19.7 | 17.8 | 15.7 |
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 28.9 | 24.6 | 20.9 | 15.1 | 10.7 | 5.9 | 3.5 | 2.0 | 1.0 |
| RoPE [ 41 ] | 31.0 | 33.2 | 33.2 | 32.2 | 30.0 | 24.3 | 18.7 | 13.4 | 9.2 |
| RoPE-Mixed [ 26 ] | 28.5 | 29.8 | 28.2 | 25.6 | 22.0 | 13.8 | 7.6 | 4.5 | 2.9 |
| nD-ALiBi [ 19 ] | 29.2 | 31.6 | 30.2 | 28.9 | 26.9 | 23.0 | 20.0 | 16.9 | 14.9 |
| LookHere [ 20 ] | 29.8 | 32.3 | 32.0 | 31.2 | 29.9 | 27.1 | 24.4 | 21.6 | 19.1 |
| PaPE (ours) | 31.1 | 34.3 | 34.8 | 34.6 | 34.1 | 32.6 | 30.9 | 28.7 | 26.9 |
| Positional Encoding | Fold 1 | Fold 2 | Fold 3 | Mean ± std |
| nD-sincos [ 48 ] | 39.3 | 41.1 | 38.1 | 39.5 ± 1.5 |
| RoPE [ 41 ] | 45.4 | 46.2 | 44.2 | 45.2 ± 1.0 |
| RoPE-Mixed [ 26 ] | 40.4 | 41.6 | 41.9 | 41.3 ± 0.8 |
| nD-ALiBi [ 19 ] | 41.4 | 43.9 | 40.3 | 41.9 ± 1.8 |
| PaPE (ours) | 51.4 | 48.2 | 49.1 | 49.5 ± 1.6 |
| PaPE-RI (ours) | 40.2 | 41.9 | 43.4 | 41.8 ± 1.6 |
| Positional Encoding | |||||||||
| nD-sincos [ 48 ] | 79.6 | 76.8 | 72.7 | 65.8 | 58.0 | 39.5 | 22.3 | 11.8 | 6.9 |
| RoPE [ 41 ] | 80.2 | 80.8 | 80.1 | 79.0 | 77.3 | 72.4 | 64.3 | 52.9 | 39.8 |
| RoPE-Mixed [ 26 ] | 80.1 | 80.2 | 78.3 | 74.8 | 69.9 | 55.2 | 36.9 | 22.9 | 14.7 |
| nD-ALiBi [ 19 ] | 80.0 | 80.4 | 79.9 | 79.0 | 77.9 | 75.2 | 71.2 | 65.6 | 59.1 |
| LookHere [ 20 ] | 80.4 | 81.5 | 81.3 | 80.8 | 80.1 | 78.2 | 75.2 | 70.6 | 65.3 |
| PaPE (ours) | 80.6 | 81.6 | 81.7 | 81.6 | 81.3 | 80.5 | 79.2 | 77.3 | 74.9 |
| Size | UCF101 | DvsGesture | GEN1 | ImageNet-1K | COCO |
| Reduced ( | 50.8 | 93.8 | 34.9 | 80.7 | 38.8 |
| Full ( ) | 49.5 | 93.4 | 34.8 | 80.6 | 38.9 |
| Positional Encoding | Fold 1 | Fold 2 | Fold 3 | Mean ± std |
| nD-sincos [ 48 ] | 37.8 | 39.1 | 39.3 | 38.7 ± 0.8 |
| RoPE [ 41 ] | 45.3 | 43.8 | 42.8 | 43.9 ± 1.2 |
| RoPE-Mixed [ 26 ] | 40.6 | 41.1 | 41.9 | 41.2 ± 0.7 |
| nD-ALiBi [ 19 ] | 41.2 | 41.5 | 42.1 | 41.6 ± 0.5 |
| PaPE (ours) | 51.4 | 48.2 | 49.1 | 49.5 ± 1.6 |
| PaPE-RI (ours) | 43.0 | 40.2 | 43.3 | 42.2 ± 1.7 |
| m | |||||||||
| 2 | 78.7 | 79.9 | 79.6 | 79.1 | 78.5 | 77.1 | 75.2 | 72.4 | 69.0 |
| 4 | 78.4 | 79.5 | 79.3 | 78.7 | 78.0 | 76.5 | 74.4 | 71.6 | 68.2 |
| 8 | 79.0 | 79.7 | 79.4 | 78.9 | 78.1 | 76.4 | 74.3 | 71.6 | 68.2 |
| 16 | 79.1 | 80.1 | 80.0 | 79.5 | 78.8 | 77.4 | 75.3 | 72.8 | 69.8 |
| 32 | 78.9 | 80.1 | 79.8 | 79.4 | 78.7 | 77.4 | 75.5 | 72.8 | 69.7 |
| 50 | 79.2 | 80.3 | 80.1 | 79.5 | 78.9 | 77.5 | 75.4 | 72.9 | 69.8 |
| Training | Inference | |||||||
| Method | Total Params. | Pos. Params. | VRAM (MB) | GFLOPS | Time (ms) | VRAM (MB) | GFLOPS | Time (ms) |
| nD-sincos | 86.4M | 0 | 1669 | 105.9 | 12.5 | 362 | 35.1 | 1.4 |
| nD-ALiBi | 86.4M | 0 | 1673 | 105.2 | 12.5 | 365 | 35.1 | 1.4 |
| RoPE | 86.4M | 0 | 1669 | 105.9 | 12.6 | 363 | 35.1 | 1.4 |
| RoPE-Mixed | 86.4M | 18.4K | 1671 | 105.9 | 12.7 | 363 | 35.1 | 1.4 |
| LookHere | 86.4M | 0 | 1694 | 105.2 | 12.6 | 387 | 35.1 | 1.4 |
| Parameter | UCF101 | DvsGesture | GEN1 | ImageNet-1K | COCO | ScanNet | ModelNet40 | nuScenes |
| Batch size | 32 | 32 | 64 | 1024 | 64 | 12 | 32 | 3 |
| Learning rate | 0.00003 | 0.00005 | 0.0001 | 0.0006 | 0.0006 | 0.006 | 0.001 | 0.003 |
| Weight decay | 0.05 | 0.05 | 0.05 | 0.05 | 0.05 | 0.05 | 0.01 | 0.03 |
| Momentum ( | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 |
| Momentum ( | 0.999 | 0.999 | 0.999 | 0.999 | 0.999 | 0.999 | 0.999 | 0.99 |
| Warmup epochs | 20 | 15 | 7.5 | 15 | 15 | 40 | 5 | 1 |
Positional Encoding
Positional encoding gives Transformers signals about token order, distance, and spatial or temporal relationships.
See all 129 papers