From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data

Linfang Zheng; Zikai Ouyang; Chen Wang; Jia Pan; Wei Zhang

cs.ROApr 4, 2026

From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data

Authors: Linfang Zheng, Zikai Ouyang, Chen Wang, Jia Pan, Wei Zhang

Organizations: The University of Hong Kong, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · LimX Dynamics

Abstract

Video is a scalable observation of physical dynamics: it captures how objects move, how contact unfolds, and how scenes evolve under interaction -- all without requiring robot action labels. Yet translating this temporal structure into reliable robotic control remains an open challenge, because video lacks action supervision and differs from robot experience in embodiment, viewpoint, and physical constraints. This survey reviews methods that exploit non-action-annotated temporal video to learn control interfaces for robotic manipulation. We introduce an interface-centric taxonomy organized by where the video-to-control interface is constructed and what control properties it enables, identifying three families: direct video-action policies, which keep the interface implicit; latent-action methods, which route temporal structure through a compact learned intermediate; and explicit visual interfaces, which predict interpretable targets for downstream control. For each family, we analyze control-integration properties -- how the loop is closed, what can be verified before execution, and where failures enter. A cross-family synthesis reveals that the most pressing open challenges center on the robotics integration layer -- the mechanisms that connect video-derived predictions to dependable robot behavior -- and we outline research directions toward closing this gap.

Figures & tables

Figure 1: Video-based manipulation interfaces. This survey organizes the literature by how video-derived temporal structure is connected to robot control through three recurring interface families: direct video–action policies, latent-action intermediates, and explicit visual interfaces (e.g., subgoal images, trajectories, or poses). These families differ in how explicitly that structure is exposed to the robot’s control loop, providing the conceptual basis for the taxonomy and comparative analyses that follow.

Figure 2: Three families of video-based manipulation interfaces. Each column shows how video-derived temporal structure is connected to robot actions through predictive modules and control modules. Direct video–action policies (left) keep this interface implicit within a shared backbone; latent-action methods (center) route it through a learned action-like intermediate; and explicit visual interfaces (right) predict interpretable targets that a downstream controller or policy module tracks. From left to right, the interface becomes more explicit and the stages of prediction and control become more cleanly separated.

Figure 3: Design space of video-based manipulation methods. Methods are positioned by interface explicitness (x-axis) and distance from robot actions (y-axis; higher indicates that the control interface is farther from low-level actions, e.g., subgoals, trajectories, or poses, rather than direct action prediction). Marker shape indicates family (direct video

\rightarrow

action, latent-action, explicit interfaces). Marker size qualitatively reflects the typical amount of action-labeled robot data used for grounding/training (low/medium/high). Plotted methods are representative anchors, not exhaustive.

Subcategory	Methods
\sagesf Family I: Direct Video–Action
Joint video–action generators	Main: GR-1 ( Wu et al., 2024 ) , GR-2 ‡ ( Cheang et al., 2024 ) , PAD ( Guo et al., 2024 ) , UWM ( Zhu et al., 2025 ) , UVA ( Li et al., 2025b ) , Cosmos Policy ( Kim et al., 2026b ) , Fast-WAM ‡ ( Yuan et al., 2026 ) Other: LDA-1B ( Lyu et al., 2026 ) , DreamZero † ( Ye et al., 2026 ) , X-WAM ‡ ( Guo et al., 2026 )
Frozen predictive-video features	VidMan ( Wen et al., 2024b ) , VPP ( Hu et al., 2025 )
Latent-state world models ( boundary )	APV ( Seo et al., 2022 ) , ContextWM ( Wu et al., 2023 )
\sagesf Family II: Latent-Action
Latent actions as standalone control interfaces	CLASP ( Rybkin et al., 2019 )

Table 1: Method taxonomy. We group methods by where the video-to-control interface is defined: (i) direct video–action methods keep the interface implicit in shared features, (ii) latent-action methods expose abstract action codes, and (iii) explicit visual interfaces expose human-interpretable subgoals, plans, or trajectories. Within each subgroup, Main methods are discussed in detail and appear in the detailed comparison tables, while Other methods are covered through shorter block-style mentions. Boundary cases are marked explicitly.

Figure 4: Control-loop abstractions for video-based manipulation. (a) Direct methods map observations directly to executable actions through a policy, leaving the connection between visual change and control implicit. (b) Latent-action methods introduce an intermediate latent variable $z_{t}$ between observations and robot commands. In standalone interfaces, $z_{t}$ is selected by a latent policy or planner and grounded to actions; in instruction-conditioned variants, $z_{t}$ may instead be explicit, internal, or training-only. The dotted box denotes an integrated latent-control interface whose components may be separate, fused, or co-trained depending on the method. (c) Explicit-interface methods predict a human-interpretable intermediate target, such as a goal image, trajectory, waypoint, flow, or keypoint, which can be verified or revised before a controller converts it into actions.

Figure 5: Taxonomy of direct video–action modeling approaches. Left: Joint video–action generators jointly predict future video and actions using a shared backbone, typically trained end-to-end on paired observation–action data. Middle: Video-prediction-pretrained action policies first learn a video prediction model for representation learning (stage 1), then train a policy on top of the learned features (stage 2), with video prediction serving as auxiliary supervision. Right: Latent-state world models from action-free video (boundary case) pretrain a latent world model from passive video via reconstruction and latent prediction objectives (stage 1), and subsequently learn action-conditioned dynamics and policies using model-based reinforcement learning in latent space with robot interaction data (stage 2). Unlike direct video–action generators, actions are not predicted directly from video but are produced via latent-space control, motivating their inclusion as a boundary case. Note: Some joint generators in (a) are trained in a single stage, jointly learning video and action prediction.

Method	Architecture	Video–Action Coupling	Training	Video at Inference	Execution
Joint video–action generators
GR-1 ( Wu et al., 2024 )	AR Transformer	Joint (shared GPT)	Two-stage	Optional (action-only decoding)	Stepwise
GR-2 ( Cheang et al., 2024 )	AR Transformer	Joint (shared GPT)	Two-stage	Optional (action-only decoding)	Chunked
PAD ( Guo et al., 2024 )	Diff. Transformer	Joint denoising, masked modalities	Two-stage	Optional (action-only denoising)	Receding
UWM ( Zhu et al., 2025 )	Diff. Transformer	Coupled diff. via timesteps	End-to-end	Optional	Chunked
UVA ( Li et al., 2025b )	Latent + diff. heads	Joint latent, decoupled decoding	End-to-end	No (action head only)	Chunked

Table 2: Structural comparison of direct video–action methods. Methods are grouped by architectural paradigm: joint methods generate video and actions in a shared model, and frozen-feature methods pre-train a video predictor and train a separate action head on its frozen predictive features. “Video at Inference” indicates whether explicit video generation is required during deployment: Optional means the method can bypass video generation and decode actions directly.

Method	Video Pre-training Data	Action Supervision	Video $\rightarrow$ Action Transfer	Task Scope	Real-Robot Scope
Joint video–action generators
GR-1 ( Wu et al., 2024 )	Ego4D (egocentric human video)	Robot demos	Shared token space; video features $\rightarrow$ action tokens	Tabletop and articulated manip.	Object transport; articulated manip.
GR-2 ( Cheang et al., 2024 )	Web-scale video + robot video	Multi-robot demos	Shared transformer; video–language features $\rightarrow$ action tokens	General tabletop manip.	105 tabletop tasks + bin picking
PAD ( Guo et al., 2024 )	Robot demos + action-free video	Robot demos	Joint denoising over video+action tokens (masked on video-only data)	Tabletop manip.	4 Panda task groups
UWM ( Zhu et al., 2025 )	Internet video + robot video	Robot demos	Coupled diffusion; modality-specific timesteps	Tabletop manip.	5 Franka DROID tasks
UVA ( Li et al., 2025b )	Robot video (action-free acceptable)	Robot demos	Shared encoder; separate video/action heads	General manip.	3 ARX X5 UMI tasks

Table 3: Training sources and transfer mechanisms of direct video–action methods. “Video Pre-training Data” describes the source of action-free video for learning visual dynamics. “Action Supervision” specifies how action labels are obtained. “Video

\rightarrow

Action Transfer” summarizes how video representations contribute to action prediction. “Task Scope” summarizes the broad task family reported in the original papers. “Real-Robot Scope” summarizes the real-robot experiments reported (or their absence).

Method	Metric (as reported)	Setting / Protocol Note	Source
CALVIN ( Mees et al., 2022 ) : long-horizon language-conditioned manipulation (ABC $\rightarrow$ D)
GR-1 ( Wu et al., 2024 )	SR@1–5 † (%): 85.4 / 71.2 / 59.6 / 49.7 / 40.1; Avg.Len ‡ : 3.06	Train on 100% ABC, test on D	Table 1
VidMan ( Wen et al., 2024b )	SR@1–5 (%): 91.5 / 76.4 / 68.2 / 59.2 / 46.7; Avg.Len: 3.42	Train on 100% ABC, test on D	Table 1
VPP ( Hu et al., 2025 )	SR@1–5 (%): 96.5 / 90.9 / 86.6 / 82.0 / 76.9; Avg.Len: 4.33	Train on 100% ABC, test on D	Table 1
MetaWorld ( Yu et al., 2020 ) : multi-task manipulation suite (50 tasks)
PAD ( Guo et al., 2024 )	Avg. SR (%) (50 tasks): 72.5	Single text-conditioned policy; 50 traj. per task for training	Table 1

Table 4: Reported quantitative snapshots for direct video–action methods. These results are not a leaderboard: papers differ in robot/data budgets, pretraining corpora, embodiments, and evaluation protocols. We use the numbers only to indicate evidence type and maturity—within-paper claims, weakly comparable shared-benchmark slices (e.g., CALVIN, LIBERO), and deployment-feasibility evidence. Entries are reported as stated in the original papers; “Source” gives the original location.

Method	Discovery Data	Grounding Workflow	Grounding Data	Task Scope	Real-Robot Scope
Latent actions as standalone control interfaces
CLASP ( Rybkin et al., 2019 )	Robot video (BAIR pushing)	Small latent $\rightarrow$ action map; plan over latents (image-goal MPC)	Small labeled set	Tabletop manip.	–
Latent actions in instruction-conditioned policy learning
LAPA ( Ye et al., 2025 )	Robot + human manip. video	Latent-action prediction as pretraining; head replacement + fine-tune	Robot demos	Tabletop manip.	3 Franka tabletop tasks, 15 objects
Moto ( Chen et al., 2025c )	Cross-embodiment robot + human video	Autoregressive latent-token pretraining; co-train action head	Robot demos	Long-horizon manip.	3 FANUC tabletop tasks
UniVLA ( Bu et al., 2025 )	Cross-embodiment robot + human video	Task-centric latent tokens; lightweight decoder $\rightarrow$ actions	Small robot demos	Manip., navigation	4 Piper-arm long-horizon tasks

Table 5: Discovery and grounding workflows of latent-action methods. “Discovery Data” indicates the action-free video used to learn latent actions. “Grounding Workflow” describes the procedure used to establish the latent-to-action or latent-conditioned policy connection. “Grounding Data” specifies the action-labeled supervision or interaction required to connect latents to executable controls. “Task Scope” summarizes the broad task family reported in the original papers. “Real-Robot Scope” summarizes the real-robot experiments reported (or their absence).

Figure 6: Generic latent-action pipeline. Latent-action methods typically follow a two-stage logic. Stage 1 : action-free video is used to infer a latent action $z_{t}$ from an observed transition $(o_{t},o_{t+H})$ through an inverse-dynamics-style encoder, constrain it with a bottleneck, and train a forward model to predict the future observation or representation. Stage 2 : the learned latent-action interface is connected to control using action-labeled robot data: a frozen encoder can label transitions with $z_{t}$ , a planner or policy can select latent actions from the current observation and task context, and a grounding mechanism maps or couples $z_{t}$ to executable robot actions $a_{t}$ . The grounding block is drawn as a mapping module for clarity, but may also be implemented through head replacement, persistent latent-token decoding, or joint latent–action generation.

Figure 7: Latent actions induce action-relevant visual change (example adapted from Genie ( Bruce et al., 2024 ) ). Each row is a different scene. The first column is the input frame, and columns 2–4 show future observations predicted by applying different latent actions to it. Although unlabeled, the latent actions produce consistent, interpretable effects (e.g., end-effector or object motion in different directions), illustrating an emergent alignment between the learned latent-action space and robot-action-relevant visual change.

Method	Latent	Bottleneck	Prediction Space	Grounding Mechanism	Deployment Role
Latent actions as standalone control interfaces
CLASP ( Rybkin et al., 2019 )	Continuous	$\beta$ -VAE / IB	Pixels (next-frame / rollout)	MLP latent $\rightarrow$ action (labeled rollouts)	Used for planning (MPC)
Latent actions in instruction-conditioned policy learning
LAPA ( Ye et al., 2025 )	Discrete	VQ	Pixels / features (fixed-horizon)	Replace latent head with action head; fine-tune	Pretraining only
Moto ( Chen et al., 2025c )	Discrete	VQ	Pixels / features (fixed-horizon)	Add action head; co-train with autoregressive latent action prediction	Pretraining + co-training
UniVLA ( Bu et al., 2025 )	Discrete	Two-stage VQ	DINO ( Oquab et al., 2024 ) features (short-horizon)	Lightweight latent $\rightarrow$ action decoder	Action tokens

Table 6: Structural comparison of latent-action methods. “Latent” and “Bottleneck” describe the form and capacity constraint of the learned latent-action variable. “Prediction Space” indicates the observation or representation space used by the latent-action discovery model. “Grounding mechanism” describes the architectural connection between the latent interface and executable actions. “Deployment role” indicates whether latent actions persist as a test-time control interface or serve only as training-time supervision.

Method	Metric (as reported)	Setting / Protocol Note	Source
LIBERO ( Liu et al., 2023 ) /CALVIN ( Mees et al., 2022 ) manipulation benchmarks — task success
UniVLA ( Bu et al., 2025 )	LIBERO Avg. SR (%) over Sp/Ob/Go/Lo: 95.2 (full pretrain); 92.5 (Bridge-V2); 88.7 (human-video)	Four suites, 50 traj. per task for training; result avg. over Sp/Ob/Go/Lo	Table I
LAPA ∗ ( Ye et al., 2025 )	LIBERO Avg. SR (%) over Sp/Ob/Go/Lo: 65.7	Four suites, 50 traj. per task for training; result avg. over Sp/Ob/Go/Lo	UniVLA Table I
HiLAM ( Kim et al., 2026a )	LIBERO-Lo SR (%): 94; Sp/Ob/Go are reported in Fig. 3a bar chart without exact numeric values	Four LIBERO suites, 10 tasks per suite and 50 demos per task; exact data-efficiency values are reported for Lo	Fig. 3a–b; Sec. 4.2.1
RotVLA ( Li et al., 2026 )	LIBERO SR (%) over Sp/Ob/Go/Lo: 98.2 / 99.6 / 98.4 / 96.4; Avg. 98.2	Four LIBERO suites	Table 1
ALAM ( Tang et al., 2026 )	LIBERO SR (%) over Sp/Ob/Go/Lo: 99.2 / 99.6 / 99.0 / 94.4; Avg. 98.1	Four LIBERO suites; reported as $\pi_{0}$ + ALAM with a 3B flow-matching backbone	Table 2

Table 7: Reported quantitative snapshots for latent-action methods. These results are not a leaderboard: papers differ in robot embodiments, simulation/real-robot settings, data regimes, and evaluation protocols. We use the numbers only to indicate evidence type and maturity—within-paper claims, weakly comparable shared-benchmark slices, and deployment-feasibility evidence. Entries are reported as stated in the original papers; “Source” gives the original location.

Figure 8: Explicit visual interface–based methods for robotic manipulation. Sub-figure (a–b) These methods decouple perception and control by first predicting a human-interpretable interface (e.g., subgoal images, trajectories, or pose sequences) from visual observations and optional language, and then mapping the interface to robot actions with a downstream policy. Some approaches additionally apply an interface transfer step (e.g., 2D trajectories $\rightarrow$ 6D object poses) before policy execution. Sub-figure (c) Representative explicit interfaces used in prior work, parentheses denote interface format (e.g., RGB, RGB-D) where methods in the same group differ; examples are reproduced/adapted from ( Mendonca et al., 2023 ; Black et al., 2024b ; Zhang et al., 2025a ; Wen et al., 2024a ; Xu et al., 2024a ; Wang et al., 2023a ; Yuan et al., 2024 ; Wang et al., 2025a ; Shi et al., 2025 ) . Methods in this section typically pretrain the interface predictor on video without action labels, and learn the downstream policy from robot data.

Method	Video Source	Robot Data	Cross-E.	Task Scope	Real-Robot Scope
Frame-based interfaces
UniPi ( Du et al., 2023 )	Internet text-video + robot video-text finetuning	Robot demos	✓	General	Limited (reimpl. by SuSIE)
Gen2Act ( Bharadhwaj et al., 2025 )	Internet (pretrained generator; frozen)	Robot demos + paired generated human videos	✓	General	multi-setting generalization test
AVDC ( Ko et al., 2024 )	Robot video (action-free); diffusion predictor finetuned in-domain	None (zero-shot; grasp required)	✓	Rigid objects	Real robot + Meta-World sim
RIGVid ( Patel et al., 2026 )	Pretrained video generator (frozen)	None (zero-shot; online tracking)	✓	Rigid objects	4 tasks; zero-shot
Dreamitate ( Liang et al., 2024 )	Collected stereo human tool-use video	None (zero-shot after pose tracking)	Limited	Tool manip.	4 tool tasks

Table 8: Training sources, deployment requirements, and real-robot evaluation scope of explicit visual interface methods. “Video Source” = data for the interface predictor (action-free unless noted); “Robot Data” = supervision needed to ground the interface on a robot; “Real-Robot Scope” = scope of real-robot experiments reported in the paper. Quantitative results on shared benchmarks (CALVIN ( Mees et al., 2022 ) , LIBERO ( Liu et al., 2023 ) ) are in Table 10 .

Method	Predicted Interface	Transferred To	Input	Action Derivation	Loop
Frame-based interfaces
UniPi ( Du et al., 2023 )	Video plan	—	RGB	Learned (inverse dynamics)	Open
Gen2Act ( Bharadhwaj et al., 2025 )	Human video plan	—	RGB	Learned (video-conditioned policy)	Closed
AVDC ( Ko et al., 2024 ) †	Video plan	Object 6D poses	RGB-D	Geometric (flow + PnP ( Lepetit et al., 2009 ) )	Open
RIGVid ( Patel et al., 2026 ) †‡	Video plan (sampled & filtered)	Object 6D poses	RGB-D	Geometric (6D pose tracking + retargeting)	Closed
Dreamitate ( Liang et al., 2024 )	Stereo video plan	Tool 6D poses	Stereo	Geometric (MegaPose ( Labbé et al., 2022 ) )	Open

Table 9: Structural comparison of explicit visual interface methods. Methods are grouped by interface type: frame-based interfaces predict future visual states, while trajectory-based interfaces predict lower-dimensional motion signals. “Transferred To” indicates whether an explicit interface-transfer step is applied; “Loop” indicates open-loop (execute plan once) or closed-loop (replan or update targets during execution). See footnotes for marker definitions.

Method	Metric (as reported)	Setting / Protocol Note	Source
CALVIN ( Mees et al., 2022 ) : long-horizon language-conditioned (ABC $\rightarrow$ D)
UniPi † ( Du et al., 2023 )	SR@1–5 ‡ (%): 56 / 16 / 08 / 08 / 04	Train on 100% ABC, test on D	SuSIE Table 1
SuSIE ( Black et al., 2024b )	SR@1–5 (%): 87 / 69 / 49 / 38 / 26	Train on 100% ABC, test on D	Table 1
CLOVER ( Bu et al., 2024 )	SR@1–5 (%): 96 / 84 / 71 / 58 / 45; Avg.Len § : 3.53	Train on 100% ABC, test on D	Table 1
LIBERO ( Liu et al., 2023 ) : multi-suite manipulation (Spatial / Object / Goal / Long)
UniPi † ( Du et al., 2023 )	SR (%) over Sp/Ob/Go/Lo: 69.2 / 59.8 / 11.8 / 5.8	Reported by ATM; 10 demos + 50 action-free videos per task	ATM Table V

Table 10: Reported quantitative snapshots for explicit visual interface methods. These results are not a leaderboard: papers differ in data regimes, sensors, and evaluation protocols. We use the numbers only to indicate evidence type and maturity—within-paper claims, weakly comparable shared-benchmark slices, and deployment-feasibility evidence. Entries are reported as stated in the original papers; “Source” gives the original location.

Axis	Direct Video–Action	Latent-Action	Explicit Visual Interfaces
What action-free video supervises	Shared dynamics-aware representations via temporal prediction	A bottleneck variable (latent action) summarizing transition-causing change	A visualizable interface (subgoal image / video plan / traj. / poses)
Interface at deployment	None exposed; link lives in hidden features and the action head	Abstract latent-action interface; may be exposed for planning/control or retained mainly as a training signal	Explicit and inspectable: plans, subgoals, tracks, flows, poses
Action production	Predict actions directly in native control space	Ground latent actions to robot actions, or transfer their supervision into a native-action policy	Map interface to robot actions via controller, optionally after interface transfer
Typical training factorization	Joint/interleaved video+action on mixed data; boundary: action-free pretrain then RL grounding	Stage 1: discover latent actions from obs.-only data; Stage 2: grounding, policy training, or head transfer with limited action-labeled data	Predictor from action-free video; controller trained separately (demos, sim, RL, or zero-shot geometry)
Inference-time requirements	Often bypasses video gen.; some models can still produce rollouts	Usually no video gen.; infer/select latent actions, decode persistent tokens/skills, or output native actions	Depends on interface: may need generative prediction + post-processing; or lightweight predictors; execution via tracking/controller
Benefits	End-to-end in native action space; simple interface; scales with mixed data; efficient when video bypassed	Modular discovery vs. grounding; compact action-like interface or supervision signal; reduced action supervision	Transparency / debuggability; natural cross-embodiment transfer; predictor/controller improve independently; interface transfer makes embodiment-agnostic targets explicit

Table 11: Family-level comparison across three design philosophies. Direct video–action models keep the video–action link implicit in shared representations; latent-action methods introduce an intermediate action abstraction that may be used at deployment or as a training signal; explicit visual-interface methods expose structured, human-interpretable targets for a downstream controller.

Family	Loop closure	Pre-exec. verification	Main control risk	Common mitigation
Direct video–action	Native-action decoding (stepwise, chunked, or receding-horizon); no inspectable intermediate	Low	Shared representation may be visually predictive yet not controllable; feasibility enters through action outputs	Robot-action grounding; frequent re-decoding to limit drift; bypass/amortize video generation for control rate; no explicit feasibility checks
Latent-action	Latent planning/search (e.g., MPC over latents) or latent-conditioned policy; some variants output native actions	Medium-low	Codes may be predictive but not identifiable or executable; risk enters at latent prediction and grounding alignment	Information bottlenecks, disentanglement/contrastive objectives, native-action fine-tuning; online monitoring of latent rollouts when retained
Explicit visual interfaces	Controller tracks a subgoal, trajectory, or pose target; open-loop, periodic replanning, or closed-loop tracking	Medium-high	Target may be visually plausible yet physically infeasible; risk enters through predicted visual/geometric targets and the transfer pipeline	Closed-loop tracking and replanning, residual correction, candidate filtering; reachability/collision checks (still rarely integrated)

Table 12: Control-level comparison across the three families. Summary of loop closure, pre-execution verification, dominant control risk, and common mitigation strategies; § 8.2 unpacks each column.

Robot Learning from Video

See all 32 papers

May 18, 2026

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models. However, most existing approaches rely on large collections of robot demonstrations, which are costly to obtain and tightly coupled to specific embodiments. Human videos, by contrast, are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation. Yet, embodiment differences and the frequent absence of task-aligned annotations make their direct use in VLA models challenging. This survey provides a unified view of how human videos are transformed into effective knowledge for VLA models. We categorize existing approaches into four classes based on the action-related information they derive: (i) latent action representations that encode inter-frame changes; (ii) predictive world models that forecast future frames; (iii) explicit 2D supervision that extracts image-plane cues; and (iv) explicit 3D reconstruction that recovers geometry or motion. Beyond this taxonomy, we highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency, thereby informing future research directions. A curated list of papers and resources is available at https://github.com/AaronFengZY/HumanCentricToVLA-Survey.

Zhiyuan Feng, Qixiu Li, Huizhi Liang +12

1Tsinghua University · 5Microsoft Research Asia · 2HKUST +4

Language-Conditioned Robot Manipulation Cross-Embodiment Robot Learning Vision-Language-Action Models Robot Imitation Learning

Ask PDF

May 12, 2026

From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation

Video generation models offer a promising imagination mechanism for robot manipulation by predicting long-horizon future observations, but effectively exploiting these imagined futures for action execution remains challenging. Existing approaches either condition policies on predicted frames or directly decode generated videos into actions, both suffering from a mismatch between visual realism and control relevance. As a result, predicted observations emphasize perceptual fidelity rather than action-centric causes of state transitions, leading to indirect and unstable control. To address this gap, we propose MoLA (Mixture of Latent Actions), a control-oriented interface that transforms imagined future videos into executable representations. Instead of passing predicted frames directly to the policy, MoLA leverages a mixture of pretrained inverse dynamics models to infer a mixture of latent actions implied by generated visual transitions. These modality-aware inverse dynamics models capture complementary semantic, depth, and flow cues, providing a structured and physically grounded action representation that bridges video imagination and policy execution. We evaluate our approach on simulated benchmarks (LIBERO, CALVIN, and LIBERO-Plus) and real-world robot manipulation tasks, achieving consistent gains in task success, temporal consistency, and generalization.

Yajie Li, Bozhou Zhang, Chun Gu +5

School of Data Science, Fudan University · Shanghai Innovation Institute · Imperial College London +1

Latent Action Learning Long-Horizon Robotic Manipulation Robotic Manipulation

Ask PDF

Oct 6, 2026

IronMan: Information-Constrained Video-Action Learning for Robot Manipulation

Video Action Models (VAMs) couple visual dynamics modeling with action generation for robot manipulation. However, video representations are not naturally suited to action generation, as exposing the action policy to excessive visual detail can impair its generalization ability. Therefore, we introduce IronMan (Information-constRained videO-actioN learning for robot MANipulation), a robust video-action learning framework built on the information bottleneck principle. The core principle of this framework is to impose information constraints that suppress irrelevant visual information while preserving action-relevant dynamics cues. IronMan employs a dynamics-aware bottleneck that distills noisy, entangled one-step video features into compact world representations. Extensive simulation and real-world experiments demonstrate strong in-distribution (ID) performance and out-of-distribution (OOD) robustness while maintaining efficient inference. IronMan achieves success rates of 99.0% on LIBERO and 79.4% on RoboTwin clean2clean, outperforming all the evaluated baselines. Under OOD shifts, IronMan achieves a success rate of 79.1% on LIBERO-Plus, exceeding the strongest baseline by 10.4 percentage points. Project page: https://youngsoul0731.github.io/ironman-project-page/

Yuanshuo Zhang, Wenzhe Zhao, Zixing Lei +2

Shanghai Jiao Tong University · Joy Future Academy, JD

Information Bottleneck Generalization in Robotic Manipulation Visuomotor Policy Learning Robotic Manipulation

Ask PDF

Abstract

Robot Learning from Video

Explore similar work

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation

IronMan: Information-Constrained Video-Action Learning for Robot Manipulation