Organizations: Shenzhen University of Advanced Technology · Southern University of Science and Technology · Primotion · New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · Shenzhen University · Nanjing University
A robot must choose an action before it can see the outcome. In a recorded demonstration, later camera frames show what happened after the robot moved. Hindsight Self-Distillation (HSD) uses these later frames to teach a policy what to extract from current images. During training, a teacher sees both the current and later frames, while a student sees the current frames. They share a visual encoder and an action predictor. Each branch summarizes its images with a small set of feature vectors. These visual summaries supply all visual input to the action predictor. The teacher learns from demonstrated actions. For each noisy action sequence, the student learns from either the demonstration or the teacher's denoising prediction. It also learns to match the teacher's visual summaries. We analyze which aspects of the teacher's predictions the student can recover from its inputs. At deployment, the student encodes the current images once and reuses its summaries at every denoising step. HSD achieves average success rates of 98.50% on LIBERO and 92.31% on RoboTwin 2.0.
Figures & tables
Figure 1: Learn with hindsight, act from the present. The demonstration records the scene before and after the robot moves (left). During training, the teacher uses the current and later frames to guide what the student extracts from the current images (center). At deployment, the student chooses its actions from the current images, robot state, and instruction (right). Circles and arrows illustrate the interaction.
Figure 2: Hindsight Self-Distillation. Gray and blue backgrounds denote the training-only teacher and deployable student, which share the Video DiT and Action DiT weights. Teacher queries (orange) summarize current and future observations; student queries (purple) summarize the current observation. Branch-specific summary keys and values provide the sole visual input to each action stream. Both streams receive Aτ at the same flow time τ . LT uses ground-truth flow; LS selects ground-truth flow or a detached teacher prediction per chunk. Lsum aligns normalized terminal summaries. Encoders, nonvisual conditions, and auxiliary video co-training are omitted. Deployment retains only the student.
Figure 3: Attention within and across branches. Rows are queries and columns are keys and values. Shaded cells allow attention. Of the two summary groups, only the teacher queries QT read the later visual tokens zf . Each action stream Ab reads its own summaries and action tokens.
Method
PT
Spatial
Object
Goal
Long
Avg.
Previously reported results
Diffusion Policy [ 5 ]
No
78.3
92.5
68.3
50.5
72.40
MDT [ 32 ]
No
78.5
87.5
73.5
64.8
76.10
OpenVLA-OFT [ 14 ]
No
94.3
95.2
91.7
86.5
91.90
π0 [ 3 ]
Yes
96.8
98.8
95.8
85.2
94.20
π0.5 [ 2 ]
Yes
98.8
98.2
98.0
92.4
96.90
Table 1: LIBERO success rates (%). PT marks additional embodied pretraining. DP, MDT, and π0 scores follow Kim et al. [14] ; MDT uses 2% language annotations. π0.5 scores follow Kim et al. [15] . Published averages retain source rounding. The lower block reports our experiments. Bold marks the highest displayed scores, including ties.
Figure 5
Method
PT
Clean
Randomized
Avg.
Previously reported results
π0 [ 3 ]
Yes
65.92
58.40
62.16
π0.5 [ 2 ]
Yes
82.74
76.76
79.75
Motus [ 1 ]
No
72.80
77.00
74.90
Motus [ 1 ]
Yes
88.66
87.02
87.84
LingBot-VA [ 18 ]
Yes
91.99
91.11
91.55
Table 2: RoboTwin 2.0 success rates (%) after joint training on clean and randomized scenes. PT marks additional embodied pretraining. Motus and LingBot-VA scores follow their CVPR and RSS versions, respectively. Avg. is the mean of the two displayed conditions. Bold marks column maxima.
Method
Spatial
Object
Goal
Long
Overall
Fast-WAM (released)
51.62
69.58
36.70
40.29
49.43
Student-only
54.58
73.71
41.30
44.74
53.48
No explicit transfer
68.28
78.99
51.87
52.76
62.83
Action only
70.98
75.97
53.22
52.88
63.10
Summary only
61.53
78.28
45.89
51.13
59.08
HSD (both)
70.27
79.27
50.37
55.62
63.71
Table 3: Fast-WAM and HSD ablations on LIBERO-Plus. Entries are success rates (%) over one trial per variant. Overall pools all 10,030 variants. Bold marks the highest score in each column.
Figure 6: Inference latency. Mean milliseconds per action chunk on one NVIDIA H20 (BF16, batch size 1), using model-specific inference settings. Lower is better.
Variant
Spatial
Object
Goal
Long
Avg.
Student-only
98.6
99.8
97.4
92.4
97.05
No explicit transfer
98.6
99.4
97.6
93.8
97.35
Action only
98.0
99.2
97.8
95.0
97.50
Summary only
98.6
99.4
97.0
93.4
97.10
HSD (both)
98.6
99.2
98.4
97.8
98.50
Table 4: LIBERO success rates (%) for ablations of the teacher branch, action-level transfer, and summary-level transfer.
K
Spatial
Object
Goal
Long
Avg.
8
97.6
98.8
98.8
96.4
97.90
16
98.2
100.0
97.8
94.8
97.70
32
98.6
99.2
98.4
97.8
98.50
Table 5: Summary-token count on LIBERO. Both branches use K tokens. Entries are success rates (%); bold marks the highest score in each column.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Setting
LIBERO
RoboTwin 2.0
Video initialization
Wan2.2-TI2V-5B
Wan2.2-TI2V-5B
Optimizer
AdamW
AdamW
Learning rate
10−4
10−4
Training epochs
10
5
Global batch size
128
512
Learning-rate decay
Cosine
Cosine
Appendix
Table S1 : HSD training and evaluation settings on LIBERO and RoboTwin 2.0.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Language provides a natural interface for humans to specify and reason about goals, but training language-conditioned robotic policies and reasoning typically requires expensive manual annotations. We propose Action Self-Distillation (ASD), a method for learning language-conditioned reasoning for robotic navigation without human-annotated language or reasoning labels. ASD uses an open-source VLM, which contains broad world knowledge and reasoning capabilities, as both a teacher network and a student policy. During training, the self-teacher receives privileged trajectory feedback and infers the underlying navigation goal together with goal-oriented reasoning. The student is then trained to condition on the inferred navigation goal, predict future ego-frame trajectory, and provide a description of the scene semantics, plan, and reasoning, from onboard observations alone. Across 13 challenging indoor and outdoor real-world navigation tasks, our method improves success rate over the strongest baseline by 30% in indoor settings, 30% on multi-stage tasks, and achieves 95% success rate on outdoor tasks, validating that privileged trajectory feedback can enable self-distillation for scalable generalist navigation policies without human annotations.
Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only on targeted action spans. Experiments on BFCL v3 and AppWorld show that our method improves over the dense per-turn feedback baseline by up to 18.80 percent while achieving 2.26× lower time per training step, suggesting that selecting where to distill is a key factor for both effective and efficient long-horizon agent training.
Large-scale action pretraining equips Embodied Foundation Models (EFMs) with strong manipulation capabilities, with particular strengths in generalization and closed-loop robustness, but reproducing such pretraining requires substantial robot data and computation. To transfer these capabilities from pretrained EFMs to policies without action pretraining, we introduce TILDE (Teacher-In-the-Loop Distillation from EFMs), which synergizes on-policy distillation at student-induced states with learning from teacher-executed recovery trajectories. We further propose AutoDAgger, an autonomous data aggregation pipeline in which a robotic reward model is utilized to monitor task progress and trigger teacher intervention, thereby collecting recovery experience without human monitoring or manual control. AutoDAgger accelerates recovery-trajectory collection by more than 2× compared with the human-in-the-loop method. By unifying learning from demonstrations, on-policy teacher supervision, and teacher's corrective behavior, TILDE improves task performance by up to 4.1× over behavior cloning across three real-world tasks and supports effective cross-architecture distillation.