One-shot Human Video-Guided Adaptation for Vision-Language-Action Models
Authors: Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao, Jin Chen, Che Sun, Zhi Gao, Yunde Jia
Organizations: Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science and Technology, Beijing Institute of Technology. · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University. · AInnovation Technology Group Co., Ltd.
Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.
Figures & tables
Fig. 1: Overview of the proposed one-shot human-video-guided VLA adaptation framework. Our framework consists of a phase-aware human video representation module and a video-conditioned adaptation module, which respectively extract task-relevant guidance from a human demonstration in the context of the current robot observation and incorporate the resulting guidance into the VLA policy for action generation. Phase-level auxiliary supervision is used only during training and is not required at inference time.
Split
# Tasks
Task Shift
Source
13
Source training
Target-A1
6
Near-task shift
Target-A2
10
Object / interaction / affordance shift
Target-B
10
Compositional task shift
TABLE I: RoboTwin task splits used in our experiments.
Hyperparameter
Value
Human support frames Ns
8
GridS tokens per frame S
32
Learnable query tokens M
4
Caption loss weight λcap
0.1
Action gate initialization
0.1
Batch size
128
TABLE II: Key hyperparameters used in our experiments.
Method
Source
Target-A1
Target-A2
Target-B
Overall
π0.5
64.4
34.5
2.5
4.0
28.4
Direct Video Prefix
43.5
25.2
1.5
1.4
19.1
CapPro
68.9
44.8
29.0
20.7
42.6
TABLE III: Success rates (%) on the RoboTwin task splits. All methods use the same Source training data. Results are averaged over the clean and randomized evaluation settings.
Variant
Source
Target-A1
Target-A2
Target-B
Target Avg.
Full CapPro
68.9
44.8
29.0
20.7
29.5
w/o GridS
w/o Phase-Aware Caption
76.5
39.9
5.9
3.1
12.7
w/o Caption Loss
w/o Query-to-Action Guidance
w/o Human Video Progress
TABLE IV: Ablation study of the proposed components. Results are reported as task success rates (%), averaged over the clean and randomized evaluation settings. Target Avg. denotes the average over all 26 target tasks.
Split
Task
π0.5
CapPro
Source
Coke → box
3/5 (60%)
Source
Marker → holder
5/5 (100%)
Source
Fold short-sleeve garment
5/5 (100%)
Target-A
Beverage → box
5/5 (100%)
Target-A
Cable → box
4/5 (80%)
Target-B
Fold pants
0/5 (0%)
TABLE V: Real-world manipulation results. Each task is evaluated over five trials. Results are reported as successful trials and success rates.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for robotic manipulation by leveraging pre-trained vision-language representations. However, current VLA training methods suffer from two critical limitations: poor generalization to novel environments and low training efficiency requiring extensive demonstrations. We introduce Agentic-VLA, an agentic training framework that enables VLAs to efficiently adapt online through three key innovations: (1) Adaptive Reward Synthesis, which dynamically generates and adjusts reward functions based on the VLA's current capabilities and task complexity, decomposing complex tasks into learnable sub-goals for curriculum learning; (2) Language-Guided Exploration, where a critic model provides structured guidance for systematic exploration rather than random sampling; and (3) Experience Memory,which stores and retrieves task-relevant policy weights for warm-starting adaptation to similar tasks. We evaluate Agentic-VLA on the LIBERO benchmark, achieving substantial improvements: +12.3% on long-horizon tasks, +28.5% in 1-shot learning, and enabling cross-task transfer from 0% to 31.2% without task-specific demonstrations. Our framework also demonstrates 2.4x faster convergence compared to existing online adaptation methods. Beyond LIBERO, Agentic-VLA retains its advantage on the dual-arm RoboTwin 2.0 benchmark, including under its randomized Hard setting. These results establish Agentic-VLA as a significant step toward truly adaptive VLA systems capable of continuous learning in deployment.
Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models. However, most existing approaches rely on large collections of robot demonstrations, which are costly to obtain and tightly coupled to specific embodiments. Human videos, by contrast, are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation. Yet, embodiment differences and the frequent absence of task-aligned annotations make their direct use in VLA models challenging. This survey provides a unified view of how human videos are transformed into effective knowledge for VLA models. We categorize existing approaches into four classes based on the action-related information they derive: (i) latent action representations that encode inter-frame changes; (ii) predictive world models that forecast future frames; (iii) explicit 2D supervision that extracts image-plane cues; and (iv) explicit 3D reconstruction that recovers geometry or motion. Beyond this taxonomy, we highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency, thereby informing future research directions. A curated list of papers and resources is available at https://github.com/AaronFengZY/HumanCentricToVLA-Survey.
Zhiyuan Feng, Qixiu Li, Huizhi Liang +12
1Tsinghua University · 5Microsoft Research Asia · 2HKUST +4
Training generalist Vision-Language-Action(VLA) models typically requires massive, diverse robotic datasets with high-fidelity action annotations. While egocentric human manipulation videos are abundant and capture significant environmental diversity, the absence of action labels makes them difficult to use in conventional training paradigms. To address this, we propose a latent-action-based framework designed to extract general action priors from unlabeled human videos. The architecture features a Hybrid Disentangled VQ-VAE that decouples motion dynamics from environmental backgrounds through physical masks, enabling the construction of a cross-embodiment action codebook. By pre-training on human videos with the codebook, the VLM backbone learns deep representations of action intent. For adaptation to specific embodiments, we introduce an intent-perception decoupling strategy where the VLM predicts the action intent while a separate frozen visual encoder provides state-specific features to the action expert, thereby reducing action hallucinations. Results in simulation and real-world environments show that our method, pre-trained exclusively on unlabeled human videos, performs competitively with state-of-the-art VLA models trained on massive annotated datasets, requiring only 50 trajectories for downstream adaptation.
Runze Xu, Yiluo Zhang, Jian Wang +2
Department of Electronic Engineering, Tsinghua University · Tianfu Jiangxi Laboratory