DynaSlots: Slot-Centric 3D Dynamics Pretraining for Robot Manipulation

Meng Li; Lijiang Chen; Qi Zhao; Jun Jiang; Longhao Zou; Shuchang Lyu; Guangliang Cheng

cs.ROOct 23, 2026

DynaSlots: Slot-Centric 3D Dynamics Pretraining for Robot Manipulation

Authors: Meng Li, Lijiang Chen, Qi Zhao, Jun Jiang, Longhao Zou, Shuchang Lyu, Guangliang Cheng

Organizations: Department of Electronics and Information Engineering, Beihang University, Beijing, China. · Pengcheng Laboratory, Shenzhen, China. · School of Artificial Intelligence, University of Science and Technology Beijing, Beijing, China. · Department of Computer Science, University of Liverpool, Liverpool, UK.

Abstract

Robot manipulation involves localized changes: picking up a cup moves the cup while leaving the surrounding scene largely unchanged. Yet existing 3D visual pretraining methods, including recent dynamics-aware variants, model state transitions at the scene level, compressing the motion of everything in a scene into a single global latent action. This conflates semantically distinct transitions in scenes with multiple moving parts, obscures dependencies between changing regions, and leaves the learned representation vulnerable to task-irrelevant scene content. We introduce DynaSlots, an action-free, label-free pretraining framework that decomposes each point cloud frame into a small set of temporally consistent slots and models dynamics at this per-slot level rather than the scene level. A temporal slot propagation module promotes consistent slot assignments across frames; an inverse dynamics model then extracts one latent action per slot from feature differences, and a diffusion-based forward dynamics model, paired with a self-supervised interaction head, predicts each slot's future state while learning pairwise proximity-and-motion relations. Experiments on Meta-World, Adroit, and a RealMan RM65-B robot demonstrate that DynaSlots outperforms prior scene-level baselines. It achieves superior success rates in simulation and exhibits strong robustness to scene clutter and unseen objects in real-world manipulation. Code is available at https://zhucezhanghao12345-source.github.io/DynaSlots-Anonymous/.

Figures & tables

Fig. 1: Scene-level versus slot-centric 3D dynamics pretraining. (a) Potential limitations of scene-level modeling (b) DynaSlots uses temporally aligned soft slots, assigns one latent action to each slot, and models pairwise proximity-and-motion relations across slots.

Fig. 2: DynaSlots framework. (a) Three point-cloud frames are encoded by a shared Point Transformer and propagated by a temporal slot binder into $K{=}6$ aligned soft slots. The IDM extracts per-slot latent actions from feature differences, and the cross-slot encoder conditions a diffusion FDM on current slot features, centroids, and actions. An EMA target branch supplies noised future slot features, while an auxiliary interaction head predicts pairwise proximity-and-motion relations. The displayed losses are noise prediction, slot VICReg, relation BCE, and assignment entropy. (b) Previous slots query current point tokens through competitive assignment, weighted aggregation, and residual refinement, repeated three times per frame. (c) Two observed point clouds and the robot state produce slot condition tokens for the diffusion policy, which predicts the action sequence using cross-attention and FiLM.

Category	Method	MetaWorld					Adroit
Category	Method	Easy(3)	Med.(6)	Hard(2)	V.Hard(3)	Mean S.R.	Door	Pen	Mean S.R.
Large-scale	CLIP [ 31 ]	29.3	14.0	5.0	55.3	24.9 ( $\downarrow$ 55.9)	61.0	84.0	72.5 ( $\downarrow$ 15.5)
	DINOv2 [ 27 ]	34.0	14.0	5.0	55.3	25.9 ( $\downarrow$ 54.9)	76.0	84.0	80.0 ( $\downarrow$ 8.0)
	PointNet [ 30 ]	71.3	49.7	23.0	55.3	51.7 ( $\downarrow$ 29.1)	80.0	72.0	76.0 ( $\downarrow$ 12.0)
Static 3D	PointMAE [ 28 ]	77.3	61.3	42.0	70.0	63.9 ( $\downarrow$ 16.9)	64.0	76.0	70.0 ( $\downarrow$ 18.0)
Static 3D	PointDif [ 46 ]	83.3	58.0	41.0	58.7	61.1 ( $\downarrow$ 19.7)	76.0	78.0	77.0 ( $\downarrow$ 11.0)

TABLE I: Comparison on simulation benchmarks (success rates in %), using the evaluation schedule and baseline suite of [ 23 ] . Parentheses give gaps to the column best in pp (

\downarrow

Fig. 3: Slot assignment on a representative frame ( $K{=}6$ ). Colored points mark where each slot has high assignment response; light-gray points are bound to other slots.

Fig. 4: Four real-world manipulation tasks: (a) putting the milk on the white plate; (b) putting the ball on the white plate; (c) placing the ball from the white plate into the bowl; and (d) putting the chocolate on the plate. Each panel shows representative stages of execution.

Variant	MetaWorld	Adroit
w/o Temporal Slot Propagation	68.5 ( $\downarrow$ 12.3)	76.0 ( $\downarrow$ 12.0)
w/o Interaction Head ( $\lambda_{\mathrm{int}}{=}0$ )	78.2 ( $\downarrow$ 2.6)	85.5 ( $\downarrow$ 2.5)
w/o Reverse Consistency ( $\lambda_{\mathrm{reverse}}{=}0$ )	79.0 ( $\downarrow$ 1.8)	86.5 ( $\downarrow$ 1.5)
$K=4$ slots	74.5 ( $\downarrow$ 6.3)	82.5 ( $\downarrow$ 5.5)
$K=8$ slots	77.5 ( $\downarrow$ 3.3)	84.5 ( $\downarrow$ 3.5)
DynaSlots (ours, $K{=}6$ )	80.8	88.0

TABLE II: Ablation on DynaSlots: mean success rates (%) and drops from the full model (pp, in parentheses).

Method	Target-task data					RH20T pretrain + target-task policy training
Method	Task (a)	Task (b)	Task (c)	Task (d)	Avg.	Task (a)	Task (b)	Task (c)	Task (d)	Avg.
DP3 [ 42 ]	62.0 $\pm$ 13.0	66.0 $\pm$ 18.2	54.0 $\pm$ 11.4	48.0 $\pm$ 13.0	57.5 $\pm$ 7.9	54.0 $\pm$ 15.2	58.0 $\pm$ 17.9	48.0 $\pm$ 13.0	44.0 $\pm$ 11.4	51.0 $\pm$ 6.8
PointDif [ 46 ]	58.0 $\pm$ 16.4	64.0 $\pm$ 11.4	52.0 $\pm$ 13.0	46.0 $\pm$ 18.2	55.0 $\pm$ 7.9	52.0 $\pm$ 13.0	56.0 $\pm$ 13.4	46.0 $\pm$ 8.9	40.0 $\pm$ 15.8	48.5 $\pm$ 7.2
FVP [ 17 ]	60.0 $\pm$ 10.0	64.0 $\pm$ 11.4	54.0 $\pm$ 8.9	50.0 $\pm$ 12.2	57.0 $\pm$ 6.0	54.0 $\pm$ 11.4	60.0 $\pm$ 12.2	50.0 $\pm$ 10.0	42.0 $\pm$ 13.0	51.5 $\pm$ 6.0
DynaMo-3D [ 9 ]	66.0 $\pm$ 8.9	72.0 $\pm$ 11.0	60.0 $\pm$ 10.0	56.0 $\pm$ 8.9	63.5 $\pm$ 5.2	60.0 $\pm$ 10.0	64.0 $\pm$ 8.9	54.0 $\pm$ 11.4	50.0 $\pm$ 10.0	57.0 $\pm$ 5.4
AFRO [ 23 ]	72.0 $\pm$ 11.0	78.0 $\pm$ 8.4	68.0 $\pm$ 8.4	64.0 $\pm$ 8.9	70.5 $\pm$ 4.8	66.0 $\pm$ 8.9	72.0 $\pm$ 11.0	60.0 $\pm$ 10.0	54.0 $\pm$ 8.9	63.0 $\pm$ 5.4

TABLE III: Real-world results on tasks (a)–(d) in Fig. 4 . Entries are mean success rate (%)

\pm

sample standard deviation (pp) across five initial-layout scores, with ten trials per layout. The Avg. is computed from the four-task suite score for each layout. The RH20T column uses RH20T pretraining followed by target-task policy training.

Method	Success rate (%) $\uparrow$		Drop (pp) $\downarrow$
	No clutter	Cluttered
DP3 [ 42 ]	$54.0\pm 11.4$	$28.0\pm 14.8$	$26.0$
PointDif [ 46 ]	$52.0\pm 13.0$	$26.0\pm 13.4$	$26.0$
FVP [ 17 ]	$54.0\pm 8.9$	$30.0\pm 15.8$	$24.0$
DynaMo-3D [ 9 ]	$60.0\pm 10.0$	$36.0\pm 13.4$	$24.0$
AFRO [ 23 ]	$68.0\pm 8.4$	$40.0\pm 12.2$	$28.0$

TABLE IV: Robustness to scene clutter on task (c); drops in pp.

Method	Success rate (%) $\uparrow$		Drop (pp) $\downarrow$
	Original	Unseen
DP3 [ 42 ]	$54.0\pm 11.4$	$38.0\pm 14.8$	$16.0$
PointDif [ 46 ]	$52.0\pm 13.0$	$36.0\pm 13.4$	$16.0$
FVP [ 17 ]	$54.0\pm 8.9$	$40.0\pm 10.0$	$14.0$
DynaMo-3D [ 9 ]	$60.0\pm 10.0$	$48.0\pm 11.0$	$12.0$
AFRO [ 23 ]	$68.0\pm 8.4$	$54.0\pm 11.4$	$14.0$

TABLE V: Robustness to unseen objects on task (c); drops in pp.

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 321 papers

Action-Effect Memory Pretraining for Robot Manipulation
Jun 10, 2026Yijing Zhou, Qiwei Liang, Sitong Zhuang +5Representation Learning Robotic Manipulation
Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
May 28, 2026Jusuk Lee, Seungjae Lee, Jonghun Shin +6Cross-Modal Learning Cross-Modal Representation Learning
Seoul National University · University of Maryland, College Park · Georgia Institute of Technology
InternW0- $Δ$ : A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
Sep 25, 2026Xingyu Miao, Zizun Li, Baole Fang +45World Models for Robotics Robot Foundation Models

Jun 10, 2026J/K move · Enter open · S save

Action-Effect Memory Pretraining for Robot Manipulation

Yijing Zhou, Qiwei Liang, Sitong Zhuang, Jiaxi Li+4

Hong Kong University of Science and Technology (Guangzhou) · Shenzhen University

Representation Learning Robotic Manipulation Dynamic Robotic Manipulation Memory-Augmented Agents Robot Learning

Open paper AskPDF arXiv

We present AEM, an Action-Effect Memory pretraining framework for robot manipulation that learns compact temporal representations from vision-action history. Unlike prior robot representation pretraining methods that mainly focus on single-frame visual encoding, AEM targets the temporal nature of manipulation, where the current observation alone is often insufficient under partial observability. AEM models manipulation as an action-driven interaction process by interleaving visual and action features and applying masked modeling to recover missing content from incomplete histories, thereby learning action-conditioned state evolution. The Mamba-encoded output of the final vision token is used as a compact history representation, serving as the global context for decoding and downstream control. This design preserves a single-vector temporal bottleneck while keeping inference efficient. We evaluate AEM with Diffusion Policy and Flow Policy. AEM consistently improves manipulation performance in both simulation and real-world settings, outperforming baselines across clean scenes, cluttered and random scenes, and non-Markovian tasks. Ablation studies further show that history-aware pretraining surpasses single-frame pretraining and direct frame stacking, while reducing inference latency and computational cost.

Abstract

Robot Learning

Explore similar work

Action-Effect Memory Pretraining for Robot Manipulation

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

InternW0-ΔΔΔ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data

InternW0- $Δ$ : A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data