Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction

Yanbang Li; Bu He; Yi Tian; Yu-Chiao Huang; Chenyu Zhang; Ruoqu Chen; Yuanzhe Liu; Yuxiang Gao

cs.ROOct 16, 2026

Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction

Authors: Yanbang Li, Bu He, Yi Tian, Yu-Chiao Huang, Chenyu Zhang, Ruoqu Chen, Yuanzhe Liu, Yuxiang Gao,

Organizations: Tsinghua University · CocoMatrix · Shanghai Qizhi Institute · UIUC

Abstract

Generative policies can synthesize robot behaviors, yet controlling and editing them through diverse spatiotemporal references remains challenging. In the real world, human intent requires robot behaviors that satisfy constraints on where, when, and how actions are executed. Our key insight is that spatiotemporal control references, such as keypoints, flows, orientations, and keyposes, can be viewed as partial specifications of the same temporal Cartesian action space. We introduce the MAP (Masked Action Space) interface to unify these action references and formulate controllable manipulation as an action inpainting problem. Our Inpainting Policy (IP) reconstructs missing action dimensions of the end-effector, combining progress-guided alignment with multi-mask training supporting unseen action-constraint combinations. We further adopt a training-free RePaint variant that iteratively inpaints the action during denoising to reduce control error. On ManiSkill and LIBERO-10, our policy achieves average task success rates of 92.67% and 64.67%, respectively.Real-world experiments demonstrate spatiotemporal control with average positional and rotational errors of 44.3 mm and 7°, respectively, showing that our policy can use diverse references to guide executable robot behaviors.

Figures & tables

Figure 1: Unified spatiotemporal control via action inpainting. Top: To enable unified spatiotemporal control from heterogeneous model-generated and human-provided references, we introduce Masked Action Space (MAP) , which represents these references as partial specifications of temporal Cartesian actions. Progress values specify when constraints apply, instead of timestep. Bottom: Our Inpainting Policy (IP) predicts these masks through action inpainting, supporting potentially unlimited spatiotemporal reference types and unseen reference combinations.

Figure 2: Inpainting Policy . Given camera observations and proprioceptive information, a progress network anchors a local MAP window and masked action chunk in the temporal Cartesian action space. The window is encoded into a feature vector to condition the base model, which predicts missing action tokens from the masked action chunk. Multi-modal joint training enables the policy to generalize to unseen spatiotemporal modalities.

Figure 3: Simulation and real-world tasks.

Figure 4: Effect of spatiotemporal information quantity and structure. (a) We vary the visible ratio to examine how the number of references affects task success. (b) We compare mask types with approximately equal numbers of spatiotemporal tokens, alongside a base policy (a DiT policy), to assess how constraint structure influences the performance of Inpainting Policy. Here, $r$ denotes the visible ratio, i.e., the proportion of action information that is not masked in MAP.

	Multi-task (LIBERO-10)		Single-task (ManiSkill)
Method	Avg. SR (%) $\uparrow$	CE(all) $\downarrow$	Lift Peg SR (%) $\uparrow$	Pick Cube SR (%) $\uparrow$	Poke Cube SR (%) $\uparrow$	Avg. SR (%) $\uparrow$	CE(all) $\downarrow$
DP	60.00	—	86.33	69.33	69.67	75.11	—
DiT Policy	59.67	—	94.67	87.33	93.00	91.67	—
STC	50.67	0.060	72.67	65.67	91.67	76.67	0.041
IP	64.67	0.045	100.00	90.33	87.67	92.67	0.023
IP-RePaint	51.33	0.026	95.80	82.13	62.20	80.04	0.014

Table 1: Unified control via joint training. Each policy checkpoint is evaluated across reference types using the average top-three picks of success rate and control error (CE). ManiSkill uses a separate policy for each task, whereas LIBERO-10 uses a shared multitask policy.

Figure 5: Mask-type breakdown in joint training. Left: We compare success rates across reference types, including the unseen mixed mask Mix0 , to evaluate controllability and compositional generalization. Right: Control error decreases as training progresses.

Figure 6: Real-world spatiotemporal editing. Here, $p_{r}$ denotes the progress associated with a user-provided reference, whereas $p$ denotes the robot’s current execution progress. The rollouts and corresponding MAP masks illustrate how these references guide execution.

Encoder	Feature dimension	Top-3 SR (%) $\uparrow$	Last evaluation step
1DConv	32	12.00	155k
2DConv	32	19.00	180k
2DConv	64	14.67	175k
2DConv	128	22.00	275k

Table 2: MAP encoder and feature-dimension comparison on PickCube. Training budgets, batch sizes, and evaluation counts are not fully matched. Bold marks the highest recorded Top-3 SR.

Forward length	Top-3 SR (%) $\uparrow$	200k SR (%) $\uparrow$	200k end SR (%) $\uparrow$	200k CE(all) $\downarrow$
52	70.22 $\bm{\pm}$ 4.25	63.33 $\bm{\pm}$ 3.79	36.33 $\bm{\pm}$ 3.06	0.02317
64	61.33 $\pm$ 3.00	55.67 $\pm$ 5.03	33.33 $\pm$ 4.93	0.03281
70	60.22 $\pm$ 3.40	55.33 $\pm$ 2.89	32.33 $\pm$ 4.73	0.03198

Table 3: MAP window-length comparison on PickCube-v1. SR statistics summarize three training seeds; end SR measures final-step success. CE(all) is the aggregate error at 200k. Bold marks the highest mean success rates and lowest CE.

STPM	Dimension / layers / heads	Obs. / gap	Checkpoint selection	SR (%) $\uparrow$
v3	512 / 4 / 8	1 / 1	Sequence best	74.00 $\bm{\pm}$ 0.00
v5	768 / 8 / 12	6 / 2	Endpoint best	70.67 $\pm$ 5.73
v6	544 / 5 / 8	6 / 2	Endpoint best	69.33 $\pm$ 3.40

Table 4: STPM configuration comparison on LIBERO-10 with a fixed 190k IP checkpoint. SR is mean

\pm

population standard deviation over three evaluation seeds. Architecture, observation history, and checkpoint selection vary together. Bold marks the highest mean SR.

Source demos	Training	Top-1 SR $\uparrow$	Top-1 step	Top-3 SR $\uparrow$	Top-5 SR $\uparrow$
500	Joint	56.00	180k	50.00	47.60
500	Two-stage	62.00	105k	56.00	54.40
1,000	Two-stage	100.00	150k	97.33	96.00

Table 5: Training-schedule comparison on LIBERO-10. All SR values are percentages; Top-1 step identifies the best evaluated checkpoint. Bold marks the higher SR within the 500-demonstration comparison. The 1,000-demonstration run is an independent experiment.

Progress source	SR $\uparrow$	CE(all) $\downarrow$	CE(success) $\downarrow$	CE(failed) $\downarrow$
STPM	83.00	0.02912	0.02675	0.04073
Rollout timestep	67.00	0.03983	0.03297	0.05377

Table 6: Progress-source comparison with a fixed PickCube IP checkpoint and 100 evaluation trajectories. SR is reported as a percentage. Bold marks the higher success rate or lower CE in each column. Successful and failed subsets can differ between modes.

Reference type	Coordinates per reference location	Visible ratio
2D point	2	$0.70$
3D point	3	$7/15$
Orientation	3	$7/15$
7D pose	7	$0.20$
Random	—	$0.20$

Table 7: Visible ratios for the budget-matched mask comparison, with

N_{\mathrm{ST}}=1.4T

. Structured-mask ratios are defined over reference locations; the random-mask ratio is defined over scalar action entries.

Method	LiftPeg Upright	Pick Cube	Poke Cube
DP	86.33	69.33	69.67
DiT Policy	94.67	87.33	93.00
STC	72.67	65.67	91.67
IP	100.00	90.33	87.67
IP-RePaint	95.80	82.13	62.20

Table 8: Per-task ManiSkill success rates (%). Each task uses a separate policy trained with 500 demonstrations.

Policy	Reference	$K$	Checkpoint	SR $(\%)\uparrow$	Norm. CE $\downarrow$	Positional error (mm) $\downarrow$
IP	3D position	4	20k	100.00	0.078249	32.60
IP	2D position	16	15k	80.00	0.069302	59.16
IP	3D position	4	15k	87.50	0.063748	36.53
IP	3D position	2	15k	12.50	0.175191	41.29
IP	2D position	4	15k	0.00	0.125435	66.29
IP	2D position	8	15k	100.00	0.048606	56.35

Table 9: Cup arrangement results by reference configuration. IP denotes Inpainting Policy without RePaint;

K

is the number of reference points. Checkpoints are identified by their training step, with k denoting 1,000 updates; a dash indicates an unreported or inapplicable value. Positional error is the 3D distance from the last recorded TCP position to the reference endpoint, averaged equally over all rollouts in each configuration, including failures, and indicating the positional error of the cup.

Policy	$K$	Checkpoint	SR $(\%)\uparrow$	Norm. CE $\downarrow$	Angle error $(^{\circ})\downarrow$
IP	4	30k	93.33	0.094315	7.8133
IP	8	30k	100.00	0.101605	7.6867
IP	16	30k	100.00	0.090101	7.7567
DP	—	50k	43.48	—	—

Table 10: Controlled test-tube pouring results by reference count. All IP rows use the orientation0.01 / STPM configuration without RePaint;

K

is the number of orientation references. DP receives no reference. The DP success labels remain unverified; angle errors are means for the corresponding reference configurations.

Task	Policy	SR $(\%)\uparrow$	Norm. CE $\downarrow$
Cup	IP	77.19	0.086105
Cup	DP	50.00	—
Water	IP	97.78	0.095340
Water	DP	43.48	—

Table 11: Aggregate real-world results across recorded reference configurations. Cup denotes cup arrangement; Water denotes controlled test-tube pouring. IP excludes RePaint. The Water DP success labels remain unverified.

Robot Motion Generation

Robot motion generation creates feasible, task-directed trajectories for robots to move, manipulate objects, and interact with people and environments.

See all 198 papers

Refine Actions If Needed: Adaptive Action Chunk Refinement for Robot Manipulation
Oct 14, 2026Hanlin Liu, Zichen Zhou, Jiacheng Zhang +6Robot Policy Learning Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute of Technology · Shanghai Jiao Tong University +2
Predict in Chunks, Revise in Spans: History-conditioned Action Regeneration for Robotic Manipulation
Oct 10, 2026Chenyang Wang, Yuntian Wang, Jiaying Tang +3Visuomotor Policy Learning Robot Imitation Learning
Soochow University, China · Tongji University, China
AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Apr 13, 2026Liaoyuan Fan, Zetian Xu, Chen Cao +3World Action Models Robotic Manipulation
INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China

Oct 14, 2026J/K move · Enter open · S save

Refine Actions If Needed: Adaptive Action Chunk Refinement for Robot Manipulation

Hanlin Liu, Zichen Zhou, Jiacheng Zhang, Liuyuan Na+5

The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute of Technology · Shanghai Jiao Tong University+2

Robot Policy Learning Robotic Manipulation Action Chunking

Open paper AskPDF arXiv

Computationally lightweight manipulation policies predict executable actions with low inference latency, but prediction errors can limit task success. Iterative generative policies can achieve higher task success rates, but repeated network evaluations increase inference latency. Refining action proposals from computationally lightweight policies can combine these strengths, but the amount of refinement needed varies across states. We introduce RAIN (Refine Actions If Needed), a framework that predicts action errors at different refinement levels to decide whether and how much to refine each executable proposal, before performing any refinement. We train an ensemble to predict these errors by comparing action proposals and their refined versions with expert actions from offline demonstrations, without requiring explicit labels of refinement needs. Calibration balances uncertainty, computation, and temporal consistency. We evaluate RAIN on eight ManiSkill tasks and four real-robot tasks. In simulation, our policy achieves 66.6% success, approaching the best evaluated diffusion-policy success rate with 5.71 times lower policy latency. On the real robot, success increases from 55.0% with Action Chunking with Transformers (ACT) to 72.5%, with comparable average successful completion time. Matched controls show that dynamic routing maintains the strongest observed fixed-refinement success level with 26.6% fewer refiner evaluations. Project page: https://rain-policy.github.io/.

Abstract

Robot Motion Generation

Explore similar work

Refine Actions If Needed: Adaptive Action Chunk Refinement for Robot Manipulation

Predict in Chunks, Revise in Spans: History-conditioned Action Regeneration for Robotic Manipulation

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps