VioLA: Learning Generalist Humanoid Control Policies from Human Data

Mert Albaba; Jens Beißwenger; Anna Manasyan; Daniel Marta; Michael J. Black; Wieland Brendel; Andreas Krause; Georg Martius

cs.ROOct 8, 2026

VioLA: Learning Generalist Humanoid Control Policies from Human Data

Authors: Mert Albaba, Jens Beißwenger, Anna Manasyan, Daniel Marta, Michael J. Black, Wieland Brendel, Andreas Krause, Georg Martius, +1 more

Organizations: Vesoma · ETH Zürich · MPI-IS · ELLIS Institute · University of Tuebingen

Abstract

Teaching a humanoid to follow instructions with its whole body runs into two obstacles. Its action space is large and tightly coupled: legs, arms, and fingers must move together while the robot keeps its balance, which makes joint-level actions hard to learn. And humanoid demonstrations are scarce, so current humanoid generalist policies do not follow new instructions out of the box and are fine-tuned on teleoperated demonstrations of each task before deployment. Human demonstrations exist in far larger numbers, but a person's motion is not a robot command. We remove both obstacles by changing what the generalist policy predicts. We introduce VioLA, a generalist humanoid policy that predicts body and hand motion latents instead of joint commands. A pretrained body- and hand-controller execute these latents on the robot. Their corresponding motion encoders map human and robot motion into the same latent spaces. A human recording is therefore labeled in the policy's action space, and the training demonstration pool contains 140.6 million frames, 93.2% of them human. As a result, VioLA follows locomotion instructions on the real robot zero-shot, without task-specific fine-tuning, reaching 100% success where GR00T N1.7 and $Ψ_0$ reach 16.7% and 0%, respectively. It also reaches 88.6% manipulation success without task-specific fine-tuning. The same approach works across two VLA and one world-action model backbones. A generalist policy trained on human demonstrations alone performs locomotion tasks on the real robot zero-shot. Code and checkpoints will be released.

Figures & tables

Figure 1: VioLA. Human and robot motion encoders produce targets in shared body and hand latent spaces. The generalist policy predicts motion latents from vision, language, and proprioception; frozen robot controllers execute its predictions. Annotated human motion therefore supervises the policy in the same action representation as robot motion.

Figure 2: VioLA overview. Left: frozen motion encoders construct body and hand action targets from human or robot demonstrations. Center: the generalist policy predicts motion latents from an egocentric image, an instruction, and proprioception. It is the only component updated during training. Right: a frozen body controller and hand decoder convert the predicted latents into robot joint commands. Human and robot encoders produce targets for these same decoders.

Panel of Figure 3 — Figure 3: One policy follows locomotion and manipulation instructions without task-specific fine-tuning. VioLA completes 30/30 locomotion trials (top) and 31/35 manipulation trials (bottom). Released GR00T N1.7 succeeds only at standing still; $\Psi_{0}$ succeeds only at chair rotation. Bars show success over five trials per task, with one standard error. Photos illustrate the tasks.

Figure 4: From body motion to object interaction. VioLA bends down, closes a laptop, and places a dumpling toy on the desk (first three rows). In a failed coat execution, VioLA grasps the coat but then drops it (last row; failure marked in red). Black and gold curves show executed body and hand latents projected onto their first principal components (63–91% and 47–87% of their variance). Projections are fitted per recording and scaled separately within each row. Gray bands mark successive action chunks; triangles align image timestamps with the curves.

Figure 5: Complex movements have compact descriptions in motion tokens. Each dot is the number of distinct body (black) or hand (gold) motion tokens over one complete recording, after grouping similar predictions as described in Appendix A.10 . Horizontal marks show the mean and vertical whiskers one standard error across five recordings per task. Numbers below give mean counts and durations. Chair rotation and laptop closing require tens of distinct body and hand tokens over roughly ten seconds of execution.

Figure 6: Human demonstrations teach robot behavior. Left: the human-only policy follows “walk to the yellow couch” and “squat down.” Right: locomotion success with human-then-mixed training (30/30 trials), robot-only training (24/30), and human-only training (12/30). All three are VioLA variants with the same GR00T backbone and frozen motion modules. Error bars show one standard error over trials. The robot-only and human-then-mixed variants each receive 200k training updates.

Figure 7: VioLA with two VLA backbones and one WAM backbone. The variants use the same training data and budget and are evaluated without task-specific fine-tuning. Bars report success and progress over 30 locomotion trials, 35 manipulation trials, or all 65. VioLA reaches 100% locomotion success with either GR00T or DiT4DiT, while GR00T performs best on manipulation. The final panel shows measured prediction delay. Error bars show one standard error over trials for performance and over recorded episodes for delay.

Figure 8: RTC reduces jumps when a new motion chunk takes over. Each row compares VioLA recordings with RTC on (gold) and off (gray). Curves show the L2 change in the 64-coordinate body latent between consecutive 20 ms control steps. Vertical lines mark chunk switches in the corresponding condition’s color. RTC-off traces show sharp spikes near these switches, whereas RTC-on traces remain close to their between-switch variation. Each recording is aligned to the start of policy execution and shown for 5.1 s; the initial idle-to-policy transition is excluded.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Episodes	Frames	Hours	Targets
Humanoid Everyday	4,064	2,962,764	16.46	Body and hands
PSI	835	1,431,454	7.95	Body and hands
UnifoLM-WBT	2,854	4,454,240	24.75	Body and hands
LeVERB	3,555	713,279	3.96	Body only
EgoSuite	20,680	131,010,575	727.84	Body and hands
Total	31,988	140,572,312	780.96

Appendix

Table 1: Converted demonstration pool. Humanoid Everyday, PSI, and UnifoLM-WBT contain real-G1 demonstrations; LeVERB contains simulated G1 demonstrations, and EgoSuite contains human recordings. All are converted to 50 Hz, with hand targets where available. Hours are frames divided by 50 Hz. Human data are 64.6% of episodes and 93.2% of frames. Sampling weights, not corpus size, determine exposure to each source during training.

Figure 9: Dataset size and VioLA training schedule. The first two bars show each source’s share of episodes and frames in the converted pool. The last two show sampling probabilities during 150k steps of human-only training and 50k steps of uniform sampling over all five corpora. Human recordings supply 93.2% of available frames; the training schedule determines how often they are sampled.

Backbone family	$H$	State conditioning	Motion-target scaling
$\pi_{0.5}$	50	110 values, tokenized	Identity
GR00T N1.7	40	46 values, state encoder	Identity
DiT4DiT	50	32 values, state encoder	Identity

Appendix

Table 2: Backbone-specific interfaces for VioLA. All variants predict the same 128 body and hand coordinates without target normalization. Chunk length and state conditioning follow each backbone.

Figure 10: Bottle carrying fails at release. One VioLA trial from a third-person camera. The robot takes the bottle from a person, walks to the table, and opens its hand while the bottle is tilted; the bottle then falls off the table. Times are seconds from the start of the clip. The task scores 0/5 and is reported separately from the seven manipulation tasks in Figure 3 .

Backbone	Episodes	Chunks	Delay (ms)	Replan interval (ms)
GR00T	161	3,871	$160.2\pm 1.8$	$559.9\pm 1.8$
$\pi_{0.5}$	83	2,554	$208.6\pm 0.6$	$608.4\pm 0.6$
DiT4DiT	57	1,035	$248.5\pm 2.5$	$648.0\pm 2.4$

Appendix

Table 3: Measured chunk timing. Mean

\pm

one standard error across recorded episodes. Delay measures the time from requesting to receiving a prediction; the replanning interval measures time between successive chunk arrivals. The episode counts exceed the scored-trial counts because the recording archive includes additional runs.

Figure 11: Threshold calibration. Each episode is summarized by the 95th percentile of framewise error, averaged over 14 body links or six fingertip proxies. The gray line shows the median of the calibration episodes’ errors; shading spans their minimum and maximum. Gold dots show held-out episodes; triangles mark values above the displayed range. Red horizontal lines mark the tracking-error references (2.57 cm body, 1.44 cm hand). Vertical dashed lines mark the reported thresholds of 8 and 32, chosen finer than the permitted 16 and 96 to reduce wrist and individual-finger deviations.

Explore similar work

WOLF-VLA: Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning
Jun 24, 2026Melya Boukheddimi, Omar Adjali, Daniel Sontag +1Robotic Control Humanoid Robot Locomotion
Robotics Innovation Center, DFKI, 28359 Bremen, Germany · Interactive Machine Learning, DFKI, 26129 Oldenburg, Germany · University of Oldenburg, 26129 Oldenburg, Germany +1
Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments
Jun 30, 2026Xiaopeng Lin, Ruoqi Yang, Shijie Lian +14Humanoid Robot Control Cross-Embodiment Robot Learning
The Hong Kong University of Science and Technology (Guangzhou) · DeepCybo · ZGCA +4
PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation
May 27, 2026Yutai Li, Shaohui Peng, Jiaming Guo +8Long-Horizon Robotic Manipulation Robot Skill Learning
State Key Lab of Processors, Institute of Computing Technology, CAS · Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, CAS · University of Chinese Academy of Sciences +3

Jun 24, 2026, cs.ROJ/K move · Enter open · S save

WOLF-VLA: Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning

Melya Boukheddimi, Omar Adjali, Daniel Sontag, Frank Kirchner

Robotics Innovation Center, DFKI, 28359 Bremen, Germany · Interactive Machine Learning, DFKI, 26129 Oldenburg, Germany · University of Oldenburg, 26129 Oldenburg, Germany+1

Robotic Control Humanoid Robot Locomotion Vision-Language-Action Models Language-Conditioned Robot Control

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently demonstrated strong generalization in robotic manipulation, yet their applicability to whole-body, contact-rich humanoid locomotion remains severely underexplored due to data scarcity, the absence of dynamically consistent demonstrations, and the difficulty of encoding optimality and safety in learning-based pipelines. This work introduces a unified framework WOLF-VLA that integrates whole-body optimal-control (OC) motion synthesis with large-scale multi-modal dataset to train VLAs capable of generating humanoid locomotion policies directly from natural-language instructions. We construct a comprehensive dataset of dynamically feasible humanoid trajectories across six locomotion-related task families, each parameterized by environmental variations, object colors, placements, and visual distractors. We train a VLA model using the collected joint trajectories, ego-centric visual observations and natural language instruction, yielding a policy that exhibits strong reasoning and robustness to initial-condition variability, and competitive performance across several tasks and environment settings. A systematic ablation study demonstrates the impact of each modality on the model performance. The full dataset, model checkpoints, and benchmarking simulation suite will be openly released, establishing a reproducible dynamically consistent benchmark for whole-body humanoid locomotion rich VLA control and enabling future research in scalable transfer of instruction-driven locomotion policies.

Abstract

Explore similar work

WOLF-VLA: Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning

Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments

PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation