Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Sharpa · National University of Singapore
Embodied perception requires positional representations that capture both physical geometry and the spatial and temporal extent of observations. We introduce POD-RoPE (Physical Observation Distribution RoPE), which represents token observation footprints as distributions over camera centers, viewing directions, and sampling times. Under independent sampling, expected rotations encode relative relationships between distributions; when each distribution collapses to a single address, the construction recovers pointwise RoPE. The same rotation statistics support Q/K matching and coherence-dependent V/O transformations, allowing observation geometry to guide both information selection and content aggregation. Per-token pre-aggregation avoids increasing sequence length or processing pairs of observation nodes. We evaluate POD-RoPE across four robot policy architectures, simulation and real-world manipulation, and complementary reconstruction, depth-estimation, and novel-view-synthesis tasks. Integrating POD-RoPE into Fast-WAM improves mean success from 91.83% to 92.88% across all 50 RoboTwin 2.0 tasks, and from 32.5% to 52.5% on two real-robot tasks on Sharpa North. Controlled reconstruction ablations support the contributions of distribution encoding and V/O transformations. In H20 model-block benchmarks, POD-RoPE achieves median forward-plus-backward speedups of 1.91× over P-RoPE and 5.34× over RayRoPE, with a median latency overhead of 17.4% over Standard RoPE. These results support physical observation distributions as an effective and computationally efficient basis for geometry-aware attention.
Figures & tables
Baseline
Ours
Task
Clean
Rand.
Clean
Rand.
adjust_bottle
100
100
100
100
beat_block_hammer
99
97
100
98
blocks_ranking_rgb
100
100
100
100
blocks_ranking_size
94
98
95
94
click_alarmclock
100
100
100
100
Table 1: Fast-WAM success rates (%) on 50 RoboTwin 2.0 tasks. Baseline ( Yuan et al., 2026 ) is Fast-WAM; Ours integrates POD-RoPE into Fast-WAM. Both use identical training and evaluation settings. Light blue shading identifies our method; the best average results are bold.
Task
π0
RDT
ACT
Baseline
Ours
Baseline
Ours
Baseline
Ours
Clean
Rand.
Clean
Rand.
Clean
Rand.
Clean
Rand.
Clean
Rand.
Clean
Rand.
adjust bottle
90
56
100
100
81
75
97
88
97
23
99
26
beat block hammer
43
21
51
33
77
37
89
33
56
3
77
0
blocks ranking rgb
19
5
26
22
3
0
9
1
1
0
4
0
click bell
44
3
96
24
80
9
83
0
58
3
92
8
Table 2: Success rates (%) after single-task fine-tuning on 15 RoboTwin 2.0 tasks across different model architectures. Each model is compared under its corresponding baseline’s training and evaluation configuration. Baseline ( Chen et al., 2025 ) is the original policy; Ours integrates full POD-RoPE. Rand. denotes Randomized conditions; bold marks the best result per task and condition across the six configurations.
Task
Fast-WAM
Fast-WAM + POD-RoPE
Δ (pp)
Bottle opening
7/20 (35%)
12/20 (60%)
+25.0
Straw insertion
6/20 (30%)
9/20 (45%)
+15.0
Average (2 tasks)
32.5%
52.5%
+20.0
Table 3: Real-robot task results on Sharpa North. Both methods use the same architecture; the baseline uses standard RoPE. Both use the 20,000-step checkpoint and are evaluated for 20 trials per task; success rates are in parentheses. Δ denotes the success-rate difference in percentage points.
Figure 1: Bottle-opening and straw-insertion examples on Sharpa North. Each row shows six moments from one successful training demonstration, ordered from left to right; times are relative to the start of that recording. The images illustrate the task process. Table 3 reports the real-robot evaluation results of the two policies.
Cross-view Reconstruction
h16 Conditional Reconstruction
Dataset
Method
MSE ↓
PSNR ↑
SSIM ↑
LPIPS ↓
MSE ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Sharpa North
Standard RoPE
0.1931
16.264
0.5682
0.5668
0.3078
15.204
0.5364
0.5937
Sharpa North
P-RoPE
0.1129
18.825
0.6476
0.2728
0.2114
17.160
0.5931
0.3678
Sharpa North
POD-RoPE
0.1022
19.376
0.6779
0.2371
0.1853
17.808
0.6279
0.3164
RoboTwin
Standard RoPE
0.1378
22.064
0.7474
0.3921
0.4716
17.733
0.6917
0.5085
RoboTwin
P-RoPE
0.1335
22.459
0.7517
0.3451
0.3961
18.104
0.6964
0.4720
Table 4: Cross-view reconstruction and h16 conditional temporal extrapolation with different positional encodings on three datasets. At h16 , the target occurs 16 time steps after the reference observations, with each step spanning four original synchronized sampling intervals: approximately 2.15, 4.27, and 2.13 seconds on Sharpa North, RoboTwin, and AGIBot, respectively. Bold marks the best result within each dataset and evaluation setting.
Dataset
Method
AbsRel ↓
SqRel ↓
RMSE ↓
RMSE-log ↓
δ1↑
δ2↑
δ3↑
RGBD
Standard RoPE
0.1753
0.2523
0.9085
0.2716
0.7273
0.8741
0.9407
P-RoPE
0.1231
0.2129
0.6839
0.2115
0.8523
0.9264
0.9463
RayRoPE
0.1221
0.2303
0.6668
0.2071
0.8563
0.9244
0.9429
POD-RoPE (Q/K-only)
0.1307
0.1982
0.7041
0.2051
0.8133
0.9194
0.9491
POD-RoPE
0.1243
0.2094
0.6753
0.2033
0.8403
0.9227
0.9454
Scenes11
Standard RoPE
0.1185
0.2060
0.9395
0.2104
0.8660
0.9422
0.9700
Table 5: Depth estimation with different positional encodings on UniMatch/DeMoN. All methods use the same backbone, training, and evaluation settings; bold marks the best results.
Configuration
PSNR ↑
SSIM ↑
LPIPS ↓
Standard RoPE
18.138
0.6576
0.6014
Physical Point (Q/K-only)
21.832
0.7206
0.3194
Physical Point (Q/K+V/O)
22.172
0.7331
0.2997
POD-RoPE (Q/K-only)
22.366
0.7465
0.2821
Distribution + direct-average V/O
21.993
0.7255
0.3341
POD-RoPE
22.916
0.7524
0.2655
Table 6: Component ablation on Sharpa North head-to-wrist reconstruction under matched settings. Direct-average V/O keeps distribution Q/K fixed. Bold marks the best results.
Figure 2: Computation cost and reconstruction quality on an H20. (a) Six-layer reconstruction-block latency at 256×320 , B1, T1, and 960 tokens. Light and dark segments show forward and backward-only time; totals give forward+backward time, using medians over 30 measurements. (b) RE10K PSNR versus forward+backward latency of the six-layer LVSM block at 256×320 , B1, three views, and 960 tokens. RayRoPE takes 100.942 ms and is shown at the 100-ms boundary. The panels use separate model-block benchmarks, not end-to-end timings; their absolute latencies are not pooled. Full protocols and values are in Appendix B and Table 8 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Case
Image size
B
T
Tokens
Standard RoPE
P-RoPE
RayRoPE
POD-RoPE (Q/K-only)
POD-RoPE
R0
128×160
1
1
240
4.907 10.552
13.395 39.429
41.789 102.995
4.717 12.028
7.234 16.496
R1
256×320
1
1
960
5.390 11.581
13.812 38.569
39.319 100.942
4.577 11.737
6.537 17.343
R2
336×448
1
1
1764
6.037 16.795
14.019 38.312
41.616 105.185
6.624 17.461
7.736 19.718
B2
256×320
2
1
1920
5.454 15.653
13.340 37.914
41.385 106.076
6.087 16.423
7.270 19.861
B4
256×320
4
1
3840
9.099 27.395
16.545 47.976
48.219 107.367
9.938 28.333
11.840 32.045
T3
256×320
1
3
2880
9.683 28.712
15.705 45.628
– –
10.394 29.611
11.894 32.890
Appendix
Table 7: Median per-case model-block latency in milliseconds. Each cell lists forward time above forward+backward time; bold marks the minimum in each case and pass.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Plücker Raymap
23.442
0.7490
0.1390
GTA
24.308
0.7790
0.1260
P-RoPE
24.503
0.7838
0.1118
RayRoPE
26.055
0.8312
0.0842
POD-RoPE (Q/K-only)
24.825
0.7970
0.1020
POD-RoPE
25.223
0.8061
0.0994
Appendix
Table 8: LVSM novel-view synthesis on RE10K with 80k steps and n=6474 . Bold denotes the best result; pale blue identifies POD-RoPE variants.
Task
Baseline
Ours
Clean
Rand.
Clean
Rand.
adjust_bottle
100
100
100
100
beat_block_hammer
99
97
100
98
blocks_ranking_rgb
100
100
100
100
blocks_ranking_size
94
98
95
94
click_alarmclock
100
100
100
100
Appendix
Table 9: Fast-WAM success rates (%) on all 50 RoboTwin 2.0 tasks, using the methods and settings of Table 1 .
Condition
Processing
base
Re-encode under this suite’s baseline; target times and scoring differ from normal reconstruction and temporal h0 .
res336
Resize from 256×320 to 336×448 , changing pixel count and aspect ratio.
crop70; fov_mixed_a/b
Center-crop to 70% and resize, or apply camera-specific crop ratios; use only the available image area.
native_fov70/90/130
AGIBot only: rectify raw fisheye images to virtual pinhole views with 70/90/130-degree FOV.
stride5/8; irregular
Sample history at stride 5/8 or irregular intervals; use sufficiently long clips.
res336_stride5/8
Combine res336 with the corresponding history sampling stride.
Appendix
Table 10: Input-change conditions. Each transformation updates effective intrinsics and actual times consistently.
Condition
Sharpa North
RoboTwin
AGIBot
base
+0.751 [+0.663, +0.837]
-0.594 [-0.687, -0.505]
+0.406 [+0.219, +0.567]
res336
+4.174 [+3.911, +4.453]
+3.328 [+3.101, +3.560]
+2.456 [+2.058, +2.803]
crop70
+1.898 [+1.775, +2.025]
+0.568 [+0.418, +0.725]
+1.442 [+1.258, +1.625]
fov_mixed_a
+2.507 [+2.392, +2.624]
+0.319 [+0.191, +0.450]
+1.300 [+1.078, +1.534]
fov_mixed_b
+1.388 [+1.274, +1.503]
+0.584 [+0.448, +0.725]
+1.580 [+1.365, +1.771]
stride5
-0.671 [-0.846, -0.509]
-0.582 [-0.700, -0.476]
+0.393 [+0.220, +0.551]
Appendix
Table 11: PSNR differences for POD-RoPE − P-RoPE under all transfer conditions, with trajectory-level paired 95% confidence intervals. RoboTwin aggregates three target directions. Each transfer condition is paired with base on the same eligible subset. Dashes indicate undefined conditions.
Model / variant
Address
Bands
Scale
Norm.
V/O
Reconstruction
Space-time
B
S
L2
Yes
π0
Space-time
B
S
None
Yes
RDT
Space-time
B
S
Input LN
Yes
ACT
Space-time
B
S
None
Yes
Fast-WAM (sim., vision)
Space-time
B
S
Input LN
Yes
Fast-WAM (sim., action)
Sequence
A
A
Input LN
Yes
Appendix
Table 12: Experimental integration configurations for POD-RoPE. Q/K denotes distribution-based positional encoding and V/O denotes content transformations. L2 and RMSNorm precede the positional operator; input LN precedes projection.
Experiment
Backbone
Global batch
Updates
Reconstruction
24 / 1152 / 8 / 4608
2,048
2,500
Component ablation
6 / 1152 / 8 / 1024
512
2,500
UniMatch
6 blocks / 144 ch. / 1 head
16
100,000
RE10K/LVSM
6 / 1152 / 8 / 1024
8
80,000
Appendix
Table 13: Training budgets for visual experiments. Backbone entries list layers / width / heads / FFN width. The ablation and main reconstruction experiments use different backbones.
Figure 3: Synchronized cross-view reconstruction examples. Rows show Sharpa North, RoboTwin, and AGIBot from top to bottom. Each model reconstructs a third camera view from two simultaneous reference observations. Columns show Reference 1, Reference 2, the VAE-decoded target, and reconstructions from Standard RoPE, P-RoPE, and POD-RoPE. Metrics below the images are PSNR (dB), SSIM, and LPIPS, respectively; higher is better for the first two and lower is better for the last. Bold marks the best value for each example. Appendix E.2 specifies the scoring windows.
Dataset
Train / val. trajectories
Eval. samples (normal / h16 )
Target task
Sharpa North
281 / 50
200 / 200
Head–wrist cross-view
RoboTwin
2,212 / 271
1,800 / 1,728
Head and both wrists as targets
AGIBot
180 / 20
80 / 80
Head–hand cross-view
Appendix
Table 14: Data and tasks for embodied reconstruction. Sample counts give the ordinary reconstruction / h16 evaluation totals per method. Splits are made by trajectory. Models are trained independently on each dataset, and all three methods share the split within a dataset.
Figure 4: Temporal extrapolation to actual targets with fixed references, showing Sharpa North, RoboTwin, and AGIBot from left to right. All points use the same step-2500 checkpoint. Evaluated horizons are h=0,1,2,4,8,16 ; connecting lines are visual guides only. The same h corresponds to different actual delays across datasets. Curves show trajectory-balanced means; RoboTwin aggregates three target directions, as in the main table. Numerical results at h16 are reported in Table 4 .
Model
Global batch
Budget
Training / checkpoint
Execution
π0
32
30,000 updates
4 GPUs, FSDP; final EMA
50 actions per call
RDT
64
10,000 updates
4 GPUs; final checkpoint
Native qpos
ACT
8
6,000 epochs
1 GPU; best validation
Chunk 50; temporal aggregation
Appendix
Table 15: Single-task policy training and checkpoint selection. Both evaluation conditions use the same checkpoint.
Relative position embedding has become a standard mechanism for encoding positional information in Transformers. However, existing formulations are typically limited to a fixed geometric space, namely 1D sequences or regular 2D/3D grids, which restricts their applicability to many computer vision tasks that require geometric reasoning across camera views or between 2D and 3D spaces. To address this limitation, we propose URoPE, a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces. For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors and projects them into the query image plane. Standard 2D RoPE can then be applied using the projected pixel coordinates. URoPE is a parameter-free and intrinsics-aware relative position embedding that is invariant to the choice of global coordinate systems, while remaining fully compatible with existing RoPE-optimized attention kernels. We evaluate URoPE as a plug-in positional encoding for transformer architectures across a diverse set of tasks, including novel view synthesis, 3D object detection, object tracking, and depth estimation, covering 2D-2D, 2D-3D, and temporal scenarios. Experiments show that URoPE consistently improves the performance of transformer-based models across all tasks, demonstrating its effectiveness and generality for geometric reasoning. Our code is available on our project website: https://urope-pe.github.io/.
Yichen Xie, Depu Meng, Chensheng Peng +4
Applied Intuition · University of California, Berkeley
Rotary Positional Embedding (RoPE) is widely used to encode relative positions in Transformers. In multi-view settings, however, patch coordinates are defined independently in each image: the same 3D point can therefore have different coordinates across views, making them insufficient for establishing cross-view correspondences. We introduce WorldRoPE, a probabilistic world-centric RoPE for multi-view attention. At each layer, WorldRoPE back-projects each patch token into a set of volumetric Gaussian primitives in a shared world frame. These primitives represent multiple plausible 3D supports for a token, jointly capturing alternative depth hypotheses, depth uncertainty, and the finite spatial extent of its patch frustum. When a key token's support is projected into a query view, it induces a distribution over where that key may appear in the query image. WorldRoPE integrates the rotary operator over this distribution in closed form, combining rotations at the projected primitive centers according to their mixture weights with covariance-dependent attenuation. This makes attention less sensitive to uncertain key locations in the query view, without rasterizing or warping token features. The formulation reduces to deterministic RoPE when the support collapses to a point. Across layers, the predicted geometry guides cross-view attention, while the updated features refine the geometry used at the next layer. Experiments on novel view synthesis, depth estimation, and 3D geometry reconstruction demonstrate the effectiveness of WorldRoPE for multi-view Transformers.
Mijeong Kim, Kyujin Lim, Bohyung Han
ECE, Seoul National University · IPAI, Seoul National University
We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).