AutodidactWAM: Cross-Modal Self-Distillation from Generated Video to Robot Actions

Sergei Kurchev; Iaroslav Kolomiets; Miguel Altamirano Cabrera; Artem Lykov; Dzmitry Tsetserukou

cs.ROOct 6, 2026

AutodidactWAM: Cross-Modal Self-Distillation from Generated Video to Robot Actions

Authors: Sergei Kurchev, Iaroslav Kolomiets, Miguel Altamirano Cabrera, Artem Lykov, Dzmitry Tsetserukou

Organizations: Skolkovo Institute of Science and Technology (Skoltech), Bolshoy Boulevard 30, bld. 1, Moscow 121205, Russia. · MWS R&D Center.

Abstract

World-action models (WAMs) such as Cosmos 3 jointly generate future video and robot actions from an observation and instruction. Adapting one such model with a lightweight LoRA fine-tune to a previously unseen robot, a Unitree G1 humanoid with five-fingered BrainCo hands, exposes a video-action asymmetry: the video renders plausible task executions, while the co-generated action is systematically mis-targeted. We evaluate closed-loop real-robot trials at three cumulative stages: pre-grasp, grasp, and pick-and-place. The native action succeeds only approximately 17%, 10%, and 7% of the time, respectively, and performs worse on held-out objects. We propose AutodidactWAM, a hand-pose estimator trained without teleoperation, followed by inverse kinematics, that runs on the model's generated video to recover action estimates. Paired with the native prediction, these recovered actions provide preferred targets for fine-tuning only the action-related layers, while the generated video is teacher-forced. We compare supervised relabeling with a rectified-flow adaptation of Diffusion-DPO. After one-time embodiment adaptation, self-distillation requires no additional task-specific teleoperation. The recovered-action gate reaches approximately 75%, 47%, and 42% pre-grasp, grasp, and pick-and-place success, compared with 17%, 10%, and 7% for the native action. A hybrid objective combining preference supervision, supervised target fitting, and Cartesian trajectory anchoring (DPO+SFT+DTW) performs best: on Oreo, the training object, it reaches 90% pre-grasp and 20% full-task success; on a held-out object, it reaches 80% and 30%. Plain Flow-DPO reaches 0% success despite 1.000 validation preference accuracy, indicating that the combination of training objectives, rather than the contrastive objective alone, drives the observed gains.

Figures & tables

Fig. 1: Cosmos generates its own action supervision. 1. generate a self-distillation pair: one anchor frame, prompt, and seed are passed through Cosmos 3; the generated video becomes the teacher for the next step. 2. One rollout, two action trajectories: (A) RoboHaMeR ${+}$ IK read the generated video itself to recover an action trajectory $\hat{a}$ — the preferred supervision; (B) the same joint generation’s native action tokens $a^{-}$ remain mis-targeted despite sharing the same rollout. 3. Fine-tune action-related layers: the preferred target $a^{+}=0.8\,\hat{a}+0.2\,a_{0}$ and the rejected $a^{-}$ , together with the teacher-forced video, self-distill only the action-related layers, with every other weight frozen and no teleoperation demonstrations involved. 4. Physical outcome: before fine-tuning the native prediction approaches the object but fails contact and leaves it behind; after fine-tuning the same model’s output moves toward $a^{+}$ and can complete the grasp in successful trials, and the updated policy seeds the next round.

Fig. 2: The AutodidactWAM pipeline. 1, generate a self-distillation pair (one joint Cosmos 3 rollout): from anchor frame $I_{0}$ and prompt/seed, the round- $k$ policy $\theta_{k-1}$ (A) jointly produces video $V$ (B) and native action $a^{-}$ ; $V$ is passed through RoboHaMeR ${+}$ IK, $\Phi(V)$ , to recover the pose (C) and the trajectory $\hat{a}$ , mixed with the round-starting prediction $a_{0}$ into the target $a^{+}=0.8\,\hat{a}+0.2\,a_{0}$ ; the pair is refreshed from the updated policy every $K$ steps (Sec. III-B ). 2, optimize the action-related layers only (teacher-forced video, shared noise): the EMA reference $\theta_{\mathrm{ref}}$ and the trainable policy $\theta_{k,i}$ (LoRA ${+}$ action interface) score $(a^{+},a^{-})$ , giving flow-matching residuals $\Delta_{\theta},\Delta_{\mathrm{ref}}$ ; the combined objective $\alpha_{\mathrm{DPO}}\mathcal{L}_{\mathrm{DPO}}+\alpha_{\mathrm{SFT}}\mathcal{L}_{\mathrm{SFT}}+\alpha_{\mathrm{DTW}}\mathcal{L}_{\mathrm{DTW}}$ (Eq. ( 2 )) is backpropagated into the policy, which EMA-updates the reference; the updated policy (D) is deployed for real-robot execution.

Pre-grasp	Pre-DPO	Extracted $\hat{a}$	SFT	DPO	DPO+SFT +DTW
Oreo	0.2	0.95	0.7	0.0	0.9
Pink antistress	0.1	0.8	0.5	0.0	0.8
Blue ball	0.2	0.5	0.2	0.0	0.25
Grasp	Pre-DPO	Extracted $\hat{a}$	SFT	DPO	DPO+SFT +DTW
Oreo	0.1	0.8	0.15	0.0	0.3
Pink antistress	0.1	0.4	0.25	0.0	0.3

TABLE I: Closed-loop pilot: success rate over

N\!=\!20

real-robot trials per (object, condition).

Fig. 3: Training-time signal vs. the on-policy task metric, for the three completed runs of Sec. VI – VII (shared $x$ -axis, iterations $0$ – $500$ ). Top: train DPO loss (solid) and validation preference accuracy (dashed) for DPO+SFT+DTW and plain DPO; both saturate accuracy at $1.000$ for many rounds before breaking down. Middle: SFT loss for plain-SFT and DPO+SFT+DTW’s auxiliary term, shown only to confirm both decrease while the headline metric below diverges. Bottom: headline task-wrist DTW/path (mm) — DPO+SFT+DTW is the only run separating from its round-1 base by more than noise ( $13.99$ mm thread-best at iteration $350$ ); plain DPO ranges $25.5$ – $75.1$ mm with no trend despite its saturated accuracy above; plain SFT improves once then degrades monotonically. Direct evidence for the training-signal/task-metric divergence of Sec. VII .

Explore similar work

Aug 9, 2026cs.RO

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.

Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7

Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute

Efficient World-Action Model Video Diffusion Models World Models Robot Policies Additive-Residual Selective Invariance

Ask PDF

Jun 3, 2026cs.LG

Flash-WAM: Modality-Aware Distillation for World Action Models

World-action models (WAMs) jointly generate future video and robot actions through iterative diffusion, achieving strong performance on manipulation benchmarks but requiring tens of denoising steps, a cost that precludes real-time control. Step distillation has emerged as the natural remedy, but off-the-shelf methods break down in the joint video-action setting because video and action streams use different SNR-shifted noise schedules and reach training with substantially different marginal noise distributions, an asymmetry that single-modality distillation methods cannot accommodate. We introduce \textbf{Flash-WAM}, a modality-aware step-distillation framework inspired by consistency distillation that selects the consistency function for each modality to match its noise regime: a linear-gradient-scaling parametrization for the action stream's low-noise regime, paired with a variance-preserving parametrization for the video stream's high-noise regime, grounded in a structural analysis of the consistency-function family that characterizes the achievable gradient scaling under the consistency boundary condition. Instantiated on LingBot-VA, Flash-WAM compresses inference to a single step in each modality. On RoboTwin 2.0, this reduces per-chunk latency from

8.1

seconds to

348

ms on NVIDIA L40S, a

23{\times}

speedup that enables real-time inference. Flash-WAM preserves task success on simulation benchmarks (

85.5\%

RoboTwin 2.0,

95.7\%

LIBERO) and substantially recovers real-world performance (

60\%

average on a Unitree G1 humanoid robot), while naive consistency distillation drops to

24\%

at the same step budget.

Arman Akbari, Ci Zhang, Arash Akbari +6

1Northeastern University · University of Georgia · 3EmbodyX Inc.

Multimodal Action Distributions

Ask PDF

Sep 29, 2026cs.CV

EVO-WAM: Evolving World Action Models through Video-Action Verification

Improving robot policies on new tasks without collecting additional expert demonstrations remains a central challenge in robot learning. World action models (WAMs) use broad video priors to jointly predict future videos and actions, offering a potential source of supervision for adapting to new tasks. However, generated videos may fail to depict task completion, and even visually successful videos may be paired with inconsistent actions that lead to execution failure. We propose EVO-WAM, a framework that adapts WAMs to unseen tasks by learning from their own generated video-action trajectories, without executing candidate actions in an external environment. First, we augment WAM training with state prediction and anchored multi-frame context to enable complete autoregressive rollouts without external execution feedback. Second, we identify reliable training experience by selecting task-completing prefixes with a vision-language model and verifying their video-action consistency with an inverse dynamics model. Third, we iteratively train the WAM on verified prefixes and generate new rollouts with the updated model. On seven unseen RoboTwin 2.0 tasks, EVO-WAM increases average success rates from 26.9% to 68.0% for Cosmos3 and from 28.5% to 46.4% for DreamZero, reaching approximately

2.5\times

and

1.6\times

their initial success rates. On three unseen long-horizon composite tasks in the real world, it improves Cosmos3's average success rate from 20.0% to 76.7%, a gain of 56.7 percentage points. Project Page: https://evo-wam.github.io/.

Shiyang Zhou, Xionghao Wu, Wenbo Li +11

HITSZ · SLAI · THU +9

Efficient World-Action Model World Models Robot Systems Long-Horizon Task Planning

Ask PDF