POD-RoPE: Rotary Position Encoding over Physical Observation Distributions for Embodied AI

Xiaodi Huang; Zilin Wang; Jinpeng Liu; Pengchuan Xiao; Zhenlei Shao; Zhang Zhang

cs.ROOct 24, 2026

POD-RoPE: Rotary Position Encoding over Physical Observation Distributions for Embodied AI

Authors: Xiaodi Huang, Zilin Wang, Jinpeng Liu, Pengchuan Xiao, Zhenlei Shao, Zhang Zhang

Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Sharpa · National University of Singapore

Abstract

Embodied perception requires positional representations that capture both physical geometry and the spatial and temporal extent of observations. We introduce POD-RoPE (Physical Observation Distribution RoPE), which represents token observation footprints as distributions over camera centers, viewing directions, and sampling times. Under independent sampling, expected rotations encode relative relationships between distributions; when each distribution collapses to a single address, the construction recovers pointwise RoPE. The same rotation statistics support Q/K matching and coherence-dependent V/O transformations, allowing observation geometry to guide both information selection and content aggregation. Per-token pre-aggregation avoids increasing sequence length or processing pairs of observation nodes. We evaluate POD-RoPE across four robot policy architectures, simulation and real-world manipulation, and complementary reconstruction, depth-estimation, and novel-view-synthesis tasks. Integrating POD-RoPE into Fast-WAM improves mean success from 91.83% to 92.88% across all 50 RoboTwin 2.0 tasks, and from 32.5% to 52.5% on two real-robot tasks on Sharpa North. Controlled reconstruction ablations support the contributions of distribution encoding and V/O transformations. In H20 model-block benchmarks, POD-RoPE achieves median forward-plus-backward speedups of $1.91\times$ over P-RoPE and $5.34\times$ over RayRoPE, with a median latency overhead of 17.4% over Standard RoPE. These results support physical observation distributions as an effective and computationally efficient basis for geometry-aware attention.

Figures & tables

	Baseline		Ours
Task	Clean	Rand.	Clean	Rand.
adjust_bottle	100	100	100	100
beat_block_hammer	99	97	100	98
blocks_ranking_rgb	100	100	100	100
blocks_ranking_size	94	98	95	94
click_alarmclock	100	100	100	100

Table 1: Fast-WAM success rates (%) on 50 RoboTwin 2.0 tasks. Baseline ( Yuan et al., 2026 ) is Fast-WAM; Ours integrates POD-RoPE into Fast-WAM. Both use identical training and evaluation settings. Light blue shading identifies our method; the best average results are bold.

Task	$\pi_{0}$				RDT				ACT
	Baseline		Ours		Baseline		Ours		Baseline		Ours
	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.
adjust bottle	90	56	100	100	81	75	97	88	97	23	99	26
beat block hammer	43	21	51	33	77	37	89	33	56	3	77	0
blocks ranking rgb	19	5	26	22	3	0	9	1	1	0	4	0
click bell	44	3	96	24	80	9	83	0	58	3	92	8

Table 2: Success rates (%) after single-task fine-tuning on 15 RoboTwin 2.0 tasks across different model architectures. Each model is compared under its corresponding baseline’s training and evaluation configuration. Baseline ( Chen et al., 2025 ) is the original policy; Ours integrates full POD-RoPE. Rand. denotes Randomized conditions; bold marks the best result per task and condition across the six configurations.

Task	Fast-WAM	Fast-WAM + POD-RoPE	$\Delta$ (pp)
Bottle opening	7/20 (35%)	12/20 (60%)	+25.0
Straw insertion	6/20 (30%)	9/20 (45%)	+15.0
Average (2 tasks)	32.5%	52.5%	+20.0

Table 3: Real-robot task results on Sharpa North. Both methods use the same architecture; the baseline uses standard RoPE. Both use the 20,000-step checkpoint and are evaluated for 20 trials per task; success rates are in parentheses.

\Delta

denotes the success-rate difference in percentage points.

Figure 1: Bottle-opening and straw-insertion examples on Sharpa North. Each row shows six moments from one successful training demonstration, ordered from left to right; times are relative to the start of that recording. The images illustrate the task process. Table 3 reports the real-robot evaluation results of the two policies.

		Cross-view Reconstruction				$h16$ Conditional Reconstruction
Dataset	Method	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Sharpa North	Standard RoPE	0.1931	16.264	0.5682	0.5668	0.3078	15.204	0.5364	0.5937
Sharpa North	P-RoPE	0.1129	18.825	0.6476	0.2728	0.2114	17.160	0.5931	0.3678
Sharpa North	POD-RoPE	0.1022	19.376	0.6779	0.2371	0.1853	17.808	0.6279	0.3164
RoboTwin	Standard RoPE	0.1378	22.064	0.7474	0.3921	0.4716	17.733	0.6917	0.5085
RoboTwin	P-RoPE	0.1335	22.459	0.7517	0.3451	0.3961	18.104	0.6964	0.4720

Table 4: Cross-view reconstruction and

h16

conditional temporal extrapolation with different positional encodings on three datasets. At

h16

, the target occurs 16 time steps after the reference observations, with each step spanning four original synchronized sampling intervals: approximately 2.15, 4.27, and 2.13 seconds on Sharpa North, RoboTwin, and AGIBot, respectively. Bold marks the best result within each dataset and evaluation setting.

Dataset	Method	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	RMSE-log $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
RGBD	Standard RoPE	0.1753	0.2523	0.9085	0.2716	0.7273	0.8741	0.9407
	P-RoPE	0.1231	0.2129	0.6839	0.2115	0.8523	0.9264	0.9463
	RayRoPE	0.1221	0.2303	0.6668	0.2071	0.8563	0.9244	0.9429
	POD-RoPE (Q/K-only)	0.1307	0.1982	0.7041	0.2051	0.8133	0.9194	0.9491
	POD-RoPE	0.1243	0.2094	0.6753	0.2033	0.8403	0.9227	0.9454
Scenes11	Standard RoPE	0.1185	0.2060	0.9395	0.2104	0.8660	0.9422	0.9700

Table 5: Depth estimation with different positional encodings on UniMatch/DeMoN. All methods use the same backbone, training, and evaluation settings; bold marks the best results.

Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Standard RoPE	18.138	0.6576	0.6014
Physical Point (Q/K-only)	21.832	0.7206	0.3194
Physical Point (Q/K+V/O)	22.172	0.7331	0.2997
POD-RoPE (Q/K-only)	22.366	0.7465	0.2821
Distribution + direct-average V/O	21.993	0.7255	0.3341
POD-RoPE	22.916	0.7524	0.2655

Table 6: Component ablation on Sharpa North head-to-wrist reconstruction under matched settings. Direct-average V/O keeps distribution Q/K fixed. Bold marks the best results.

Panel of Figure 2 — Figure 2: Computation cost and reconstruction quality on an H20. (a) Six-layer reconstruction-block latency at $256\times 320$ , B1, T1, and 960 tokens. Light and dark segments show forward and backward-only time; totals give forward+backward time, using medians over 30 measurements. (b) RE10K PSNR versus forward+backward latency of the six-layer LVSM block at $256\times 320$ , B1, three views, and 960 tokens. RayRoPE takes 100.942 ms and is shown at the 100-ms boundary. The panels use separate model-block benchmarks, not end-to-end timings; their absolute latencies are not pooled. Full protocols and values are in Appendix B and Table 8 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Case	Image size	B	T	Tokens	Standard RoPE	P-RoPE	RayRoPE	POD-RoPE (Q/K-only)	POD-RoPE
R0	$128\times 160$	1	1	240	4.907 10.552	13.395 39.429	41.789 102.995	4.717 12.028	7.234 16.496
R1	$256\times 320$	1	1	960	5.390 11.581	13.812 38.569	39.319 100.942	4.577 11.737	6.537 17.343
R2	$336\times 448$	1	1	1764	6.037 16.795	14.019 38.312	41.616 105.185	6.624 17.461	7.736 19.718
B2	$256\times 320$	2	1	1920	5.454 15.653	13.340 37.914	41.385 106.076	6.087 16.423	7.270 19.861
B4	$256\times 320$	4	1	3840	9.099 27.395	16.545 47.976	48.219 107.367	9.938 28.333	11.840 32.045
T3	$256\times 320$	1	3	2880	9.683 28.712	15.705 45.628	– –	10.394 29.611	11.894 32.890

Appendix

Table 7: Median per-case model-block latency in milliseconds. Each cell lists forward time above forward+backward time; bold marks the minimum in each case and pass.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Plücker Raymap	23.442	0.7490	0.1390
GTA	24.308	0.7790	0.1260
P-RoPE	24.503	0.7838	0.1118
RayRoPE	26.055	0.8312	0.0842
POD-RoPE (Q/K-only)	24.825	0.7970	0.1020
POD-RoPE	25.223	0.8061	0.0994

Appendix

Table 8: LVSM novel-view synthesis on RE10K with 80k steps and

n=6474

. Bold denotes the best result; pale blue identifies POD-RoPE variants.

Task	Baseline		Ours
Task	Clean	Rand.	Clean	Rand.
adjust_bottle	100	100	100	100
beat_block_hammer	99	97	100	98
blocks_ranking_rgb	100	100	100	100
blocks_ranking_size	94	98	95	94
click_alarmclock	100	100	100	100

Appendix

Table 9: Fast-WAM success rates (%) on all 50 RoboTwin 2.0 tasks, using the methods and settings of Table 1 .

Condition	Processing
base	Re-encode under this suite’s baseline; target times and scoring differ from normal reconstruction and temporal $h0$ .
res336	Resize from $256\times 320$ to $336\times 448$ , changing pixel count and aspect ratio.
crop70; fov_mixed_a/b	Center-crop to 70% and resize, or apply camera-specific crop ratios; use only the available image area.
native_fov70/90/130	AGIBot only: rectify raw fisheye images to virtual pinhole views with 70/90/130-degree FOV.
stride5/8; irregular	Sample history at stride 5/8 or irregular intervals; use sufficiently long clips.
res336_stride5/8	Combine res336 with the corresponding history sampling stride.

Appendix

Table 10: Input-change conditions. Each transformation updates effective intrinsics and actual times consistently.

Condition	Sharpa North	RoboTwin	AGIBot
base	+0.751 [+0.663, +0.837]	-0.594 [-0.687, -0.505]	+0.406 [+0.219, +0.567]
res336	+4.174 [+3.911, +4.453]	+3.328 [+3.101, +3.560]	+2.456 [+2.058, +2.803]
crop70	+1.898 [+1.775, +2.025]	+0.568 [+0.418, +0.725]	+1.442 [+1.258, +1.625]
fov_mixed_a	+2.507 [+2.392, +2.624]	+0.319 [+0.191, +0.450]	+1.300 [+1.078, +1.534]
fov_mixed_b	+1.388 [+1.274, +1.503]	+0.584 [+0.448, +0.725]	+1.580 [+1.365, +1.771]
stride5	-0.671 [-0.846, -0.509]	-0.582 [-0.700, -0.476]	+0.393 [+0.220, +0.551]

Appendix

Table 11: PSNR differences for POD-RoPE

-

P-RoPE under all transfer conditions, with trajectory-level paired 95% confidence intervals. RoboTwin aggregates three target directions. Each transfer condition is paired with base on the same eligible subset. Dashes indicate undefined conditions.

Model / variant	Address	Bands	Scale	Norm.	V/O
Reconstruction	Space-time	$\mathcal{B}$	S	L2	Yes
$\pi_{0}$	Space-time	$\mathcal{B}$	S	None	Yes
RDT	Space-time	$\mathcal{B}$	S	Input LN	Yes
ACT	Space-time	$\mathcal{B}$	S	None	Yes
Fast-WAM (sim., vision)	Space-time	$\mathcal{B}$	S	Input LN	Yes
Fast-WAM (sim., action)	Sequence	A	A	Input LN	Yes

Appendix

Table 12: Experimental integration configurations for POD-RoPE. Q/K denotes distribution-based positional encoding and V/O denotes content transformations. L2 and RMSNorm precede the positional operator; input LN precedes projection.

Experiment	Backbone	Global batch	Updates
Reconstruction	24 / 1152 / 8 / 4608	2,048	2,500
Component ablation	6 / 1152 / 8 / 1024	512	2,500
UniMatch	6 blocks / 144 ch. / 1 head	16	100,000
RE10K/LVSM	6 / 1152 / 8 / 1024	8	80,000

Appendix

Table 13: Training budgets for visual experiments. Backbone entries list layers / width / heads / FFN width. The ablation and main reconstruction experiments use different backbones.

Figure 3: Synchronized cross-view reconstruction examples. Rows show Sharpa North, RoboTwin, and AGIBot from top to bottom. Each model reconstructs a third camera view from two simultaneous reference observations. Columns show Reference 1, Reference 2, the VAE-decoded target, and reconstructions from Standard RoPE, P-RoPE, and POD-RoPE. Metrics below the images are PSNR (dB), SSIM, and LPIPS, respectively; higher is better for the first two and lower is better for the last. Bold marks the best value for each example. Appendix E.2 specifies the scoring windows.

Dataset	Train / val. trajectories	Eval. samples (normal / $h16$ )	Target task
Sharpa North	281 / 50	200 / 200	Head–wrist cross-view
RoboTwin	2,212 / 271	1,800 / 1,728	Head and both wrists as targets
AGIBot	180 / 20	80 / 80	Head–hand cross-view

Appendix

Table 14: Data and tasks for embodied reconstruction. Sample counts give the ordinary reconstruction /

h16

evaluation totals per method. Splits are made by trajectory. Models are trained independently on each dataset, and all three methods share the split within a dataset.

Figure 4: Temporal extrapolation to actual targets with fixed references, showing Sharpa North, RoboTwin, and AGIBot from left to right. All points use the same step-2500 checkpoint. Evaluated horizons are

h=0,1,2,4,8,16

; connecting lines are visual guides only. The same

h

corresponds to different actual delays across datasets. Curves show trajectory-balanced means; RoboTwin aggregates three target directions, as in the main table. Numerical results at

h16

are reported in Table 4 .

Model	Global batch	Budget	Training / checkpoint	Execution
$\pi_{0}$	32	30,000 updates	4 GPUs, FSDP; final EMA	50 actions per call
RDT	64	10,000 updates	4 GPUs; final checkpoint	Native qpos
ACT	8	6,000 epochs	1 GPU; best validation	Chunk 50; temporal aggregation

Appendix

Table 15: Single-task policy training and checkpoint selection. Both evaluation conditions use the same checkpoint.

Positional Encoding

Positional encoding gives Transformers signals about token order, distance, and spatial or temporal relationships.

See all 117 papers

URoPE: Universal Relative Position Embedding across Geometric Spaces
Apr 20, 2026Yichen Xie, Depu Meng, Chensheng Peng +4Rotary Positional Embeddings Relative Positional Encoding
Applied Intuition · University of California, Berkeley
WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention
Oct 12, 2026Mijeong Kim, Kyujin Lim, Bohyung HanRotary Positional Embeddings Multi-View 3D Reconstruction
ECE, Seoul National University · IPAI, Seoul National University
RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
Jan 21, 2026Yu Wu, Minsik Jeon, Jen-Hao Rick Chang +2Multi-View 3D Reconstruction Novel View Synthesis
Carnegie Mellon University · Apple

Apr 20, 2026J/K move · Enter open · S save

URoPE: Universal Relative Position Embedding across Geometric Spaces

Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu+3

Applied Intuition · University of California, Berkeley

Rotary Positional Embeddings Relative Positional Encoding Positional Encoding

Open paper AskPDF arXiv

Relative position embedding has become a standard mechanism for encoding positional information in Transformers. However, existing formulations are typically limited to a fixed geometric space, namely 1D sequences or regular 2D/3D grids, which restricts their applicability to many computer vision tasks that require geometric reasoning across camera views or between 2D and 3D spaces. To address this limitation, we propose URoPE, a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces. For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors and projects them into the query image plane. Standard 2D RoPE can then be applied using the projected pixel coordinates. URoPE is a parameter-free and intrinsics-aware relative position embedding that is invariant to the choice of global coordinate systems, while remaining fully compatible with existing RoPE-optimized attention kernels. We evaluate URoPE as a plug-in positional encoding for transformer architectures across a diverse set of tasks, including novel view synthesis, 3D object detection, object tracking, and depth estimation, covering 2D-2D, 2D-3D, and temporal scenarios. Experiments show that URoPE consistently improves the performance of transformer-based models across all tasks, demonstrating its effectiveness and generality for geometric reasoning. Our code is available on our project website: https://urope-pe.github.io/.

Abstract

Positional Encoding

Explore similar work

URoPE: Universal Relative Position Embedding across Geometric Spaces

WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention

RayRoPE: Projective Ray Positional Encoding for Multi-view Attention