Imitation Learning (IL) enables robots to acquire manipulation skills from expert demonstrations. Diffusion Policy (DP) models multi-modal expert behaviors but degrades when naively increasing stacked observation horizons, limiting long-horizon manipulation. We propose Self-Evolving Gated Attention (SEGA), a temporal module that maintains a time-evolving latent state via gated attention, enabling efficient recurrent updates that accumulate long-term context into a compact latent representation while filtering irrelevant temporal information. Integrating SEGA into DP yields Self-Evolving Diffusion Policy (SeedPolicy), which resolves the temporal modeling bottleneck and extends the effective temporal horizon with moderate overhead. On the RoboTwin 2.0 benchmark with 50 manipulation tasks, SeedPolicy outperforms DP and other IL baselines. Averaged across both CNN and Transformer backbones, SeedPolicy achieves 36.8% relative improvement in clean settings and 169% relative improvement in randomized challenging settings over the DP. Compared to vision-language-action models such as RDT with 1.2B parameters, SeedPolicy achieves stronger performance in the clean setting with one to two orders of magnitude fewer parameters, demonstrating strong efficiency. These results establish SeedPolicy as a state-of-the-art imitation learning method for long-horizon robotic manipulation. Code is available at: https://github.com/Youqiang-Gui/SeedPolicy.
Figures & tables
Figure 1: Horizon scaling analysis. (a) DP shows a counter-intuitive performance drop as the observation horizon grows, indicating that naive frame stacking fails to scale. (b) Temporal attention can initially improve performance, but scaling it to longer horizons incurs quadratic cost and diminishing returns. (c) In contrast, our approach improves task success by recurrently extending the effective temporal receptive field without increasing the per-step observation window.
Figure 2: Overview of the SeedPolicy framework. The system takes current RGB images and joint poses as input, encoding them via a ResNet Encoder and MLP. The core Self-Evolving Gated Attention (SEGA) module (blue box) recursively updates a time-evolving latent state ( Statet ) to capture long-term spatiotemporal dependencies while generating enhanced observation features ( EObst ). These context-rich features are then fed into the Action Expert, a transformer-based diffusion model, to predict a sequence of future actions.
Figure 3: (a) SEGA employs a dual-stream design: the State Update stream (top) evolves the latent state ( Statet−1 ) by integrating new observations, while the State Retrieval stream (bottom) utilizes historical context to generate enhanced observation features ( EObst ). (b) The Self-Evolving Gate (SEG) dynamically computes a gating signal directly from the cross-attention logits. It selectively fuses the intermediate evolved state ( Inter⋅St ) with the previous state, ensuring only semantically relevant information is preserved while filtering out noise.
Figure 4: Performance comparison across varying task length. A consistent trend emerges in both architectures: as the task length increases, the performance gap between SeedPolicy and the baseline progressively widens. This validates the architecture-agnostic effectiveness of our approach, demonstrating that the advantage of our explicit temporal modeling becomes increasingly significant in long-horizon scenarios compared to fixed-window baselines.
Method
RDT ( Liu et al. 2024 )
ACT ( Zhao et al. 2023 )
DP ( Chi et al. 2025 )
SeedPolicy
Transformer
CNN
Transformer
CNN
Avg. Success (Easy)
34.50%
29.74%
33.10%
28.04%
40.08%
42.76%
Avg. Success (Hard)
13.72%
1.74%
1.44%
0.64%
4.28%
1.54%
# Parameters
1.2 B
80 M
20.61 M
96.80 M
33.36 M
147.26 M
Table 1: Average success rates and parameter counts over 50 tasks. Gray indicates non-comparable VLA results.
Task
Transformer
CNN
Ep.Time
Task
Transformer
CNN
Ep.Time
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Click Alarmclock
58
61
61
64
2
Place Mouse Pad
0
1
0
1
5
Click Bell
83
85
54
91
2
Place Shoe
12
23
23
26
5
Beat Block Hammer
72
72
42
61
3
Rotate Qrcode
8
23
13
27
5
Grab Roller
84
89
98
93
3
Scan Object
2
9
9
11
5
Lift Pot
74
71
39
80
3
Open Laptop
56
45
49
47
6
Table 2: Per-task performance comparison between Baseline and Ours on 50 tasks in the "clean" setting. Tasks are sorted by episode time ( Ep.Time ) (average duration of expert demonstrations in seconds). Higher success rates are highlighted in bold .
Figure 5: Qualitative failure cases in simulation. We compare the successful execution of SeedPolicy (top row) against representative failure modes of the DP across three tasks: (a) Put Bottles Dustbin ("clean" setting), (b) Handover Mic ("hard" setting), and (c) Grab Roller ("hard" setting). Red circles highlight critical errors, including execution stagnation (getting stuck) and spatial positioning failures (collisions or air grabs). Additional visualizations are provided in the Appendix.
Task
DP
SeedPolicy
Looping Place-Retrieval
14
38
Sequential Picking
10
26
Bottle Handover
16
54
Food Replacement
18
60
Cover and Reveal
16
58
Table 3: Real-world success rates (%) across five tasks.
Figure 6: Qualitative failure analysis in real-world scenarios. We visualize the successful execution of SeedPolicy (top rows) compared to common baseline failures in (a) Looping_Place-Retrieval and (b) Bottle_Handover . Red circles highlight critical errors. Failure Case 1 illustrates execution stagnation caused by Perceptual Aliasing (misinterpreting the returned block as the initial state in (a)) or overfitting to pauses (in (b)). Failure Case 2 demonstrates spatial precision errors (air grabs and collisions) attributed to the lack of explicit depth information. Additional visualization analysis is provided in the Appendix.
Task
DP + ARMT-style
DP + MemoryVLA-style
SeedPolicy
Move Playingcard Away
56
64
68
Turn Switch
50
52
54
Place Object Stand
20
26
28
Dump Bin Bigbin
47
50
52
Place Container Plate
38
51
60
Place Empty Cup
15
30
32
Table 4: Comparison of different memory mechanisms on ten manipulation tasks.
Task
Architecture Design
DP
+ Temp. Att.
+ State
+ Gating
CA (SeedPolicy)
FFN
Turn Switch (Short)
51
51
51
54
53
Place Empty Cup (Medium)
24
26
28
32
21
Stack Bowls Two (Long)
33
48
65
73
70
Table 5: Component-wise Ablation of SeedPolicy.
Figure 7: Visualization Analysis of the Self-Evolving Gating Mechanism.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Section
Description
Related Work
An extended review of prior studies related to imitation learning for robotic manipulation and gating mechanisms in sequence modeling.
Hyperparameter Analysis
A comprehensive sensitivity analysis on the key hyperparameters of the Self-Evolving Gated Attention (SEGA) module.
Hardware & Data Collection and Processing
Detailed specifications of the hardware configurations (DOS-W1) and the data collection and processing pipeline used to curate expert demonstrations.
Additional Results
Additional quantitative and qualitative comparisons on both the RoboTwin 2.0 simulation benchmark and real-world robotic setups.
Open-Loop Evaluation
Open-loop evaluations on the training set to verify the model’s capacity and trajectory reconstruction quality.
Appendix
Table 6: Overview of the Appendix Content. This table outlines the supplementary materials provided to support the main paper.
Task
State
30
60
90
Grab Roller
74
89
80
Dump Bin Bigbin
43
52
47
Open Microwave
73
80
79
Handover Mic
80
92
86
Stack Bowls Two
56
73
73
Appendix
Table 7: Ablation study on latent state sequence length ( Ns ). We report the success rates (%) on six representative tasks from RoboTwin 2.0. The results demonstrate that Ns=60 achieves superior performance while maintaining a smaller parameter size compared to Ns=90 .
Task
Network Depth
2
4
6
8
Scan Object
3
2
9
2
Beat Block Hammer
56
35
72
47
Move Can Pot
59
63
71
52
Dump Bin Bigbin
44
43
52
44
Handover Mic
83
77
92
86
Appendix
Table 8: Ablation study on the depth of attention blocks. We report the success rate (%) on seven representative tasks. The model achieves the best balance and performance at depth 6.
Hyperparameter
SeedPolicy-Transformer
SeedPolicy-CNN
Optimization & Training
Optimizer
AdamW
AdamW
Learning Rate
1×10−4
1×10−4
LR Scheduler
Cosine Decay
Cosine Decay
LR Warmup Steps
500
500
Batch Size
128
128
Appendix
Table 9: Hyperparameter configurations for SeedPolicy. We report the settings for both Transformer and CNN backbones used in our experiments.
Component
Parameter
Value
Dual Arms
Degrees of Freedom (DoF)
2×(6 Arm+1 Gripper)
Payload Capacity
1.5 kg / arm
Working Radius
647 mm
Repeatability
±0.1 mm
Mobile Chassis
Drive Type
Differential Drive
Dimensions ( L×W )
700×620 mm
Appendix
Table 10: Hardware Specifications of the DOS-W1 Platform. The system integrates dual 7-DoF manipulators with a high-payload differential mobile base, featuring an adjustable lift mechanism for versatile workspace adaptation.
Figure 8: The DOS-W1 Mobile Manipulation Platform. As illustrated, the system integrates dual 7-DoF robotic arms with a differential drive mobile chassis and a vertical lift mechanism. A front-view RGB camera is mounted on the mast for visual perception.
Task
Transformer
CNN
Ep.Time
Task
Transformer
CNN
Ep.Time
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Click Alarmclock
4
6
5
10
2
Place Mouse Pad
0
0
0
0
5
Click Bell
1
1
0
3
2
Place Shoe
0
0
0
0
5
Beat Block Hammer
0
9
0
0
3
Rotate Qrcode
1
1
0
0
5
Grab Roller
0
51
0
0
3
Scan Object
0
0
0
0
5
Lift Pot
1
5
0
1
3
Open Laptop
2
3
0
1
6
Appendix
Table 11: Per-task performance comparison between Baseline and Ours on 50 tasks in the "hard" setting. Tasks are sorted by episode time ( Ep.Time ) (average duration of expert demonstrations in seconds). Higher success rates are highlighted in bold .
Task
DP
DP+State
SeedPolicy
+FFN gate
(Ours)
Turn Switch
12
30
35
Place Empty Cup
0
3
6
Stack Bowls Two
0
18
25
Appendix
Table 12: Success rates (%) of models trained and evaluated in the “Hard” setting.
Task
50 Demos
100 Demos
200 Demos
Turn Switch
54
55
57
Place Empty Cup
32
67
87
Stack Bowls Two
73
91
94
Appendix
Table 13: Success rates (%) of SeedPolicy under varying numbers of expert demonstrations.
Task
DP
ACT
π0.5
SeedPolicy
Swap Blocks
11
2
24
26
Cover Blocks
0
0
0
15
Battery Try
9
19
16
33
Appendix
Table 14: Success rate (%) comparison on three minute-level manipulation tasks.
Task
DP
ACT
Ours-Img
Ours-Voxel
Coffee D2
26
33
80
79
Kitchen D1
50
61
83
89
Mug Cleanup D1
23
31
76
77
Appendix
Table 15: Success rate (%) comparison on MimicGen. Img and Voxel denote image-based and voxel-based observation modalities, respectively.
Figure 9: More qualitative visualization of failure cases in simulation. We compare the successful execution of SeedPolicy (top row) against representative failure modes of the DP across three tasks: (a) Stack Bowls Three ("clean" setting), (b) Stack Blocks Two ("clean" setting), and (c) Shake Bottle ("hard" setting). Red circles highlight critical errors, including execution stagnation (getting stuck) and spatial positioning failures (collisions or air grabs).
Figure 10: More qualitative failure analysis in real-world scenarios. We visualize the successful execution of SeedPolicy (top rows) compared to common baseline failures in Sequential_Picking . Red circles highlight critical errors. Failure Case 1 illustrates execution stagnation caused by Perceptual Aliasing (misinterpreting the placed block as the initial state). Failure Case 2 demonstrates spatial precision errors (air grabs) attributed to the lack of explicit depth information.
Task
RDT ( Liu et al. 2024 )
MemoryVLA ( Shi et al. 2026 )
π0.5 ( Black et al. 2025 )
SeedVLA
Turn Switch
35
40
42
50
Handover Mic
90
69
58
91
Put Object Cabinet
33
38
54
60
Stack Bowls Two
76
81
93
94
Blocks Ranking Size
0
27
36
45
Appendix
Table 16: Compatibility study on RoboTwin 2.0 with VLA-based policies.
Backbone
Wins
Losses
Ties
Two-sided p -value
Transformer
38
5
7
2.5×10−7
CNN
41
6
3
1.8×10−7
Appendix
Table 17: Statistical significance analysis on RoboTwin 2.0 clean-setting tasks. We perform a paired two-sided exact sign test over per-task success rates between SeedPolicy and the corresponding DP baseline. Ties are excluded when computing the p-value.
Vision-based imitation learning has enabled impressive robotic manipulation skills, but action imitation alone provides limited supervision of the geometric consequences of robot behavior. To address this limitation, we introduce Implicit Scene Supervision (ISS) Policy, a 3D visuomotor diffusion policy with a DiT backbone that predicts continuous action sequences from point-cloud observations. ISS augments action diffusion with a supervised robot motion predictor that maps generated actions and robot-state context to end-effector motion, and then uses the predicted motion together with gripper intent to forecast future point-cloud representations. By explicitly modeling the intermediate transition from action to robot motion, ISS encourages the policy to capture how its actions affect the surrounding 3D scene. We further introduce asymmetric gradient routing to separate direct motion regression from scene-level policy supervision, together with a change-balanced objective that accounts for variations in scene-change magnitude. These auxiliary objectives provide dynamics-aware geometric supervision using only expert demonstrations, without requiring additional annotations or auxiliary modules at inference time. ISS Policy achieves state-of-the-art performance on single-arm manipulation tasks in MetaWorld and dexterous manipulation tasks in Adroit, while real-world dual-arm experiments further demonstrate its effectiveness on physical robotic manipulation. The resulting framework preserves the scalable DiT backbone and standard diffusion-policy control interface. Code and videos will be released.
Diffusion-based imitation learning has shown strong promise for robot manipulation. However, most existing policies condition only on the current observation or a short window of recent observations, limiting their ability to resolve history-dependent ambiguities in long-horizon tasks. To address this, we introduce DSSP, a history-conditioned Diffusion State Space Policy that enables efficient, full-history conditioning for robot manipulation. Leveraging the continuous sequence modeling properties of State Space Models (SSMs), our history encoder effectively compresses the entire observation stream into a compact context representation. To ensure this context preserves critical information regarding future state evolution, the encoder is optimized with a dynamics-aware auxiliary training objective. This high-level context representation is then seamlessly fused with recent state observations to form a hierarchical conditioning mechanism for action generation. Furthermore, to maintain architectural consistency and minimize GPU memory overhead, we also instantiate the diffusion backbone itself using an SSM. Extensive experiments across simulation benchmarks and real-world manipulation tasks show that DSSP achieves state-of-the-art performance with a significantly smaller model size, demonstrating superior efficiency of the hierarchical conditioning in capturing crucial information as the history length increases.
Imitation learning enables robots to acquire manipulation skills from demonstrations by mapping observations to actions. Existing approaches predict either short-horizon continuous action sequences or discrete keyposes. However, continuous prediction methods suffer from compounding errors due to short prediction horizons and struggle with multi-modal action distributions, whereas keypose-based methods necessitate an external planner, constraining real-time applicability. To address these challenges, we introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms. SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose, enabling long-horizon prediction with real-time refinement. Furthermore, we leverage the capability of diffusion models and DDIM inversion to propose a Dynamic Temporal Ensembling mechanism, which allows the policy to efficiently respond to dynamic environments and mitigate discontinuities caused by inconsistent multi-modal sampling. SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios, indicating its strong ability to reason over extended temporal dependencies while maintaining real-time adaptability and control stability.
Haidong Cao, Wenjun Cao, Quanhao Li +5
Institute of Trustworthy Embodied AI, Fudan University, China · Shanghai Key Laboratory of Multimodal Embodied AI, China