GAE: General Action Expert for Real-Time Humanoid Teleoperation

Yuefan Wang; Huaicheng Zhou; Xiao He; Zhijie He; Mingchuan Yang; Huayi Zhang; Li Chai; Jinxin Liu

cs.ROSep 28, 2026

GAE: General Action Expert for Real-Time Humanoid Teleoperation

Authors: Yuefan Wang, Huaicheng Zhou, Xiao He, Zhijie He, Mingchuan Yang, Huayi Zhang, Li Chai, Jinxin Liu, +1 more

Organizations: Westlake Robotics · Westlake University

Abstract

Humanoid avatars extend human physical presence beyond the body, enabling people to participate in social, service, and labor activities through remotely operated robots. This requires teleoperation systems capable of realizing diverse and dynamic whole-body behaviors while maintaining responsive human-robot synchronization. We present General Action Expert(GAE), a unified learning framework for general-purpose, low-latency humanoid whole-body teleoperation. To cover diverse human behaviors, GAE builds a large-scale human motion dataset from heterogeneous sources, including videos, animations, and motion capture, followed by standardization and augmentation. GAE then addresses the noise and embodiment mismatch in human motions with a two-stage training paradigm: a privileged generator policy first tracks human motion references in simulation and rolls out feasible humanoid trajectories; a deployable executor policy then learns to track these generated trajectories under curriculum domain randomization. For responsive human-robot synchronization, GAE introduces a latency-conditioned anticipation mechanism that adaptively compensates for end-to-end delay during real-time teleoperation. Simulation and real-world experiments on Unitree G1 and Westlake O1 robots demonstrate that GAE enables humanoids to smoothly mirror diverse, agile, and expressive human behaviors. Project website: https://wangyf0928.github.io/gae-wlrobotics/

Figures & tables

Fig. 2 : Overview of GAE framework. GAE consists of three stages: 1) curating a large-scale multi-source human motion dataset, 2) training the controller in simulation through a generator-executor paradigm, 3) teleoperating real-world robots with latency-conditioned synchronization.

Fig. 4 : Synchronization evaluation in real world.

Fig. 5 : ”free-style” produced by self-referential control. (a) Cradling sway; (b) Wide arm swings.

Penalty
Term	Equation	Weight
Torque limits	$\sum_{j}[\|\tau_{j}^{\rm c}/\bar{\tau}_{j}\|-0.75]_{+}$	$-1.0$
DoF position limits	$\sum_{j}([q_{j}^{min}-q_{j}]_{+}+[q_{j}-q_{j}^{max}]_{+})$	$-1.0\!\times\!10^{1}$
DoF velocity limits	$\sum_{j}[\|\dot{q}_{j}/\bar{v}_{j}\|-\rho]_{+}$	$-1.0$
Torque	$\sum_{j}\|\beta_{j}^{\tau}\tau_{j}^{\rm c}\|$	$-1.0\!\times\!10^{-4}$
Action rate	$\\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\\|_{2}^{2}$	$-1.0\!\times\!10^{-2}$

TABLE II : Reward designs for RL training.

Domain Rand Term	Range	Type
Friction	$[0.3,1.6]$	Scale
Body link mass	$[0.5,1.5]$	Scale
Torso link mass (kg)	$[0,2]$	Add
Body link CoM (m)	$[-0.03,0.03]$	Add
Joint stiffness	$[0.75,1.25]$	Scale
Action delay	$[0,3]$	–

TABLE III : Domain randomization parameters.

Training hyperparameter	Value
Num steps per env	16
Num learning epochs	1
Num mini-batches	1
Gamma	0.97
Lambda	0.9
Clip parameter	0.2

TABLE IV : Training hyperparameters.

Fig. 6 : SMPL body models used for shape fitting.

Fig. 7 : Stitch on motions. (a) Walking; (b) Running; (c) Transition from walking to running.

Fig. 8 : Recomposition on motions. (a) Drinking; (b) Walking; (c) Drinking while walking.

Fig. 9 : Open-world teleoperation. Two Unitree G1 robots follow separate motion-capture operators during a live fighting performance, reproducing expressive behaviors.

Fig. 10 : VR-based teleoperatopn. The operator’s head, hands, and feet are tracked by a VR system and reconstructed into whole-body motion. GAE follows the reconstructed motion to produce smooth robot movements.

Fig. 11 : Long-distance teleoperation. An operator in Beijing remotely controls the humanoid robot in Hangzhou over a distance of approximately 1,300 km.

Fig. 12 : Robustness to command interruption. When the motion target is unexpectedly frozen at an unstable frame(left), GAE autonomously settles the robot into a nearby balanced posture(right) without falling or losing control.

Fig. 13 : Robustness to physical disturbances. GAE maintains stable whole-body control under a strong external impact to the pelvis and an additional payload of approximately 5kg.

Fig. 14 : Cross-embodiment teleoperation. Unitree G1, Westlake O1, and ENGINEAI PM01 are simultaneously controlled by the same human operator, demonstrating the transferability of the GAE framework across different humanoid embodiments.

Explore similar work

MATE: Multi-Agent Virtual Teleoperation Platform for Humanoid Collaboration Data Collection
Sep 22, 2026Yichuan Yu, Youzhuo Wang, Yiming Ren +6Teleoperation Multi-Robot Systems
ShanghaiTech University · Nanyang Technological University
Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments
Jun 30, 2026Xiaopeng Lin, Ruoqi Yang, Shijie Lian +14Egocentric Video Teleoperation
The Hong Kong University of Science and Technology (Guangzhou) · DeepCybo · ZGCA +4
SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy
Apr 20, 2026Wei Yao, Haohan Ma, Hongwen Zhang +6Human-To-Robot Transfer Synagent
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +2

Sep 22, 2026, cs.ROJ/K move · Enter open · S save

MATE: Multi-Agent Virtual Teleoperation Platform for Humanoid Collaboration Data Collection

Yichuan Yu, Youzhuo Wang, Yiming Ren, Di Feng+5

ShanghaiTech University · Nanyang Technological University

Teleoperation Multi-Robot Systems Strong Imitation Learning

Open paper AskPDF arXiv

Humanoid robots require diverse embodied experiences to acquire complex loco-manipulation and collaborative skills. However, existing humanoid data pipelines primarily focus on individual agents, while physical multi-robot collaboration remains difficult to scale due to costly hardware, dedicated spaces, and repeated resets. In this work, we introduce MATE, a Multi-Agent virtual TEleoperation platform for humanoid collaboration data collection that enables multiple geographically distributed operators to simultaneously control whole-body humanoids in a shared physics-based environment. MATE removes the need for multiple physical robots and co-located operation while preserving physically coupled interactions among humanoids, objects, and environments. Using MATE, we construct a multi-humanoid collaboration dataset comprising 24.1 hours of coordinated behavior across 2,500 joint episodes and five long-horizon tasks, including object handover, relay delivery, environment interaction, and cooperative transport. To improve learning from these interaction-rich demonstrations, we introduce EAIS, an Execution-Aligned Interaction Sampling strategy that computes sampling signals within an execution-aligned prefix and prioritizes task-progressing and interaction-critical behaviors. We evaluate MATE with representative imitation learning and vision-language-action policies across diverse collaboration tasks. Experiments demonstrate efficient data collection, effective policy learning, and zero-shot transfer from virtual demonstrations to a physical humanoid without real-world fine-tuning. Project page: https://yerik-yu.github.io/MATE/

Abstract

Explore similar work

MATE: Multi-Agent Virtual Teleoperation Platform for Humanoid Collaboration Data Collection

Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments

SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy