AMBIT: Anticipatory Multimodal Body Recruitment for Bimanual Tracking on a Humanoid

Hanlong Li; Sihan Tan; Takeshi Ashizawa; Benjamin Yen; Kazuhiro Nakadai

cs.ROSep 27, 2026

AMBIT: Anticipatory Multimodal Body Recruitment for Bimanual Tracking on a Humanoid

Authors: Hanlong Li, Sihan Tan, Takeshi Ashizawa, Benjamin Yen, Kazuhiro Nakadai

Organizations: Institute of Science Tokyo, Tokyo, Japan

Abstract

A humanoid with 5-DoF arms cannot track generic bimanual end-effector trajectories with its arms alone; pelvis and waist motion must be recruited, but which motion, and when, is not uniquely determined. On a Unitree R1 in fixed double support, the set of dynamically valid recruitment strategies (pelvis pose and waist trajectories) for a task is a diverse continuous manifold, and a deterministic regressor trained on it mode-averages into strategies valid only 35% of the time, against 52% for a conditional variational autoencoder (CVAE) and 82% for the best of 16 CVAE samples. We introduce AMBIT: the CVAE proposes strategies from a preview of the commanded trajectory, a non-learned selector filters, ranks and verifies them, and a receding-horizon loop commits to one with hysteresis. The committed strategy is the reference of the same whole-body differential-IK QP a reactive tracker runs, which keeps authority over residual error. On 160 held-out episodes that admit a valid strategy, in full MuJoCo dynamics under a torque controller, AMBIT reaches 85% success at a 3 cm/15 deg tolerance against 74% for the tracker (disjoint confidence intervals) and recruits the body before the arms saturate in 48% of episodes against 35%. Because diversity is preserved, constraints unknown at training time are enforced by selection alone: under five zero-shot shifts AMBIT beats the warm-started tracker on every shift and matches a test-time re-optimisation baseline 17x more expensive. On a Unitree G1, with hyperparameters unchanged, the protocol reproduces the structure of the valid set and widens the gap over the tracker to 0.85 against 0.53. Five selected strategies execute on the externally supported physical R1, distinct in pelvis excursion and tracking the planned end-effector motion to a median of 11 mm by encoder forward kinematics, which establishes kinematic realisability, not balance.

Figures & tables

Fig. 1: One bimanual task, four bodies. With pelvis and waist frozen the 5-DoF arms saturate and the hands miss their targets (left); the same task admits several dynamically valid recruitment strategies that lean, lower or twist the body (right; MuJoCo, Unitree R1). We learn their conditional distribution, select among samples with a non-learned verifier, and execute the chosen one as the reference of a standard whole-body tracker.

model	precision (2 cm / 3 cm / 5 cm)	P( $\geq$ 1 valid of 16)	coverage	MMD 2
B2 regressor	0.15 / 0.35 [0.28, 0.42] / 0.54	0.35 [0.28, 0.42]	0.20	–
CVAE (ours)	0.33 / 0.52 [0.46, 0.57] / 0.72	0.82 [0.76, 0.87]	0.46	0.11
cond. flow	0.20 / 0.34 [0.30, 0.39] / 0.52	0.69 [0.62, 0.76]	0.43	0.08

TABLE I: E1/E2: per-sample precision at the strict/mild/loose tolerances, probability that at least one of 16 samples is valid, and oracle-relative coverage at the mild tolerance; 60 held-out episodes, 3 seeds, 95% CIs.

method	mild [CI]	strict	anticip.	bnd. err	sw.
ours (soft) + fallback	0.85 [0.81, 0.88]	0.74	48%	0.18 cm	1.7
replan every 1.0 s	0.85 [0.81, 0.88]	0.73	48%	0.18 cm	1.1
replan every 1.5 s (= latency)	0.81 [0.78, 0.85]	0.66	49%	0.18 cm	0.8
single plan at $t=0$	0.72 [0.68, 0.76]	0.56	49%	0.18 cm	0.0
ours (soft)	0.82 [0.79, 0.86]	0.66	48%	0.19 cm	2.6
ours (hard) + fallback	0.81 [0.77, 0.84]	0.29	50%	0.36 cm	2.3

TABLE II: E4, 160 held-out episodes

\times

3 seeds: success at the mild (3 cm, 15 ∘ ; 95% CI) and strict (2 cm, 10 ∘ ) tolerances, episodes recruiting before arm saturation, mean EE error around

t_{\mathrm{sat}}

, and strategy switches per episode.

shift	ours (CVAE)	ours (flow)	B1 warm-started	B4 re-opt.
torso obstacle	0.86 (0.74)	0.81 (0.72)	0.74 (0.51)	0.78 (0.58)
pelvis keep-out	0.87 (0.75)	0.82 (0.72)	0.74 (0.51)	0.78 (0.58)
waist range $\downarrow$	0.87 (0.75)	0.82 (0.72)	0.74 (0.51)	0.78 (0.58)
no forward lean	0.67 (0.58)	0.66 (0.59)	0.61 (0.46)	0.67 (0.52)
payload 0.75 kg	0.61 (0.55)	0.60 (0.55)	0.43 (0.36)	0.58 (0.46)

TABLE III: E3, post-selection success at the mild tolerance (strict in parentheses) on 160 held-out episodes

\times

3 seeds; the shift is known only at test time. Wall time per plan, candidates verified in parallel: 1.5, 1.6, 1.7 and 25 s for the four columns.

Fig. 6: One task per platform at the instant of the largest frozen-body miss (MuJoCo; a lateral task on the R1, a traversal on the G1). Frozen body: the arms saturate and the hands miss their targets (spheres); one valid oracle strategy brings both hands back within millimetres by moving the pelvis and, on the R1, the waist.

ep.	str.	pelvis	dur.	joint err. [mrad]		EE pos. err. [mm]			rot. [deg]
		[cm]	[s]	med.	max	mean	p95	max	mean
68	0	4.0	5.4	5.6	40	10.5	14.4	15.8	2.35
68	1	5.4	5.4	7.3	43	10.3	16.3	17.6	2.44
68	2	3.0	5.4	7.2	46	10.3	15.6	16.7	2.30
399	1	8.3	4.1	6.6	65	13.1	18.4	19.2	2.65
399	3	6.5	4.1	7.9	58	13.6	24.3	24.7	2.84

TABLE IV: Hardware execution (supported robot), trajectory phase. Pelvis: commanded excursion. End-effector error: forward kinematics of the encoders against the plan, both hands pooled.

Explore similar work

Jul 22, 2026cs.RO

What Matters in Humanoid General Motion Tracking? An Empirical Study

Humanoid general motion tracking requires policies that can follow diverse whole-body references while maintaining balance. Building such policies involves many practical design choices, and their individual effects are often hard to assess. We address this issue with an empirical study of common modeling and training factors used in recent humanoid motion-imitation pipelines. To make the study controlled and reproducible, we developed YAHMP, an open-source modular framework for training, evaluating, and deploying whole-body motion tracking policies on the Unitree G1. Within YAHMP, we define a nominal configuration and compare variants that differ in motion-command representation, observation history, action representation, actuation profile, hand-force randomization during training, and training approach. We evaluate the resulting policies on a test set of retargeted human motions and compare the nominal policy with TWIST2 as an external baseline trained on the same motion set. The results distinguish choices with clear tracking effects from choices that mainly change actuation effort, training complexity, or physical interaction capability. Finally, we deploy YAHMP policies zero-shot on the real Unitree G1, demonstrating diverse whole-body motion tracking, balance under external perturbations, and forceful interaction.

Fabio Amadio, Enrico Mingo Hoffman

HUCEBOT · Inria, Université de Lorraine, CNRS, Nancy, France

Humanoid Motion Zero-Shot Robustness Empirical Study Random

Ask PDF

Aug 13, 2026cs.RO

HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark

Humanoid motion tracking is central to teleoperation and whole-body imitation, yet evaluation often disagrees with what people perceive in videos. Kinematic errors average per-frame pose differences but miss the physical artifacts that matter most, particularly unstable support and incorrect contacts such as foot skating and mistimed touch-downs. Meanwhile, widely used test suites are small and lack the diversity needed to stress contact-rich, long-horizon behaviors. We introduce HumanTracker to make humanoid tracking evaluation both perceptually aligned and scalable. The HumanTracker benchmark contains approximately 153 hours of optical motion trajectories from multiple professional performers, organized into four motion families with text labels for fine-grained diagnosis. We further propose HumanScore, a preference-aligned metric trained on 12K motion pairs containing 24K motions. Across representative state-of-the-art trackers, HumanScore better predicts human preferences and reveals contact and stability failures that kinematic metrics often miss.

Dairu Liu, Zekun Qi, Jiayu Zeng +11

1Nankai University · 3Galbot · 2Tsinghua University +3

Human Motion Motion Imitation Motion Human-Annotated Benchmark

Ask PDF

Jun 15, 2026cs.RO

VENOM: Versatile Embodied Network for Omni-bodied Motion tracking

Achieving expert-level expressive full-body motion tracking across multiple humanoids solely from demonstration data remains a challenging and relatively an underexplored problem in humanoid robot learning. Cross-embodiment motion tracking policies are mostly trained by decoupling the control problem into upper and lower body control. This work proposes VENOM, a cross-embodiment full-body motion tracking model for humanoids in simulation. VENOM is a GPT-based motion tracker trained on multiple humanoid data that can track the entire body without the requirement to split into upper and lower body control. We curate a multi-humanoid motion tracking dataset called the VENOM dataset that contains states, actions, and rewards and train VENOM and the baselines on this dataset. In this letter, we evaluate VENOM's performance against baselines and show that we can achieve a stable motion tracker across different humanoids more capable than an MLP trained on multiple humanoid data with supervised learning alone, and also show that despite lack of reward feedback, VENOM closely matches the tracking capability of experts that were trained using asymmetric-actor critic reinforcement learning.

Siddharth Padmanabhan, Kazuki Miyazawa, Takato Horii

Graduate School of Engineering Science, University of Osaka, 1-3 Machikaneyama-cho, Toyonaka-shi, Osaka 560-8531, Japan

Training-Set Long-Tail Motions Human Motion Scalable Robot Learning

Ask PDF