EmbodyHOI: Embodiment-Aware Generation of Hand-Object Interactions for Humanoid Dexterous Manipulation

Chaitanya Chawla; Kartik Sharma; Ananya Bal; Laszlo A. Jeni; Guanya Shi

cs.ROOct 19, 2026

EmbodyHOI: Embodiment-Aware Generation of Hand-Object Interactions for Humanoid Dexterous Manipulation

Authors: Chaitanya Chawla, Kartik Sharma, Ananya Bal, Laszlo A. Jeni, Guanya Shi

Organizations: Carnegie Mellon University, United States

Abstract

Humanoid dexterous manipulation needs reference motions that are both semantically rich and kinematically feasible for the target robot. Text-conditioned hand-object interaction (HOI) generators supply diverse, language-specified demonstrations, but they synthesize motion in human (MANO) space. The embodiment gap is therefore deferred to post-hoc retargeting, where trajectories deployed on the robot can fail to maintain contact with objects and also degrade policy learning. Our key idea is that robot reachability should constrain the interaction trajectories while they are being generated. We present EmbodyHOI, which runs a differentiable inverse-kinematics (IK) solver for a specific humanoid embodiment inside the sampling loop of a frozen flow-matching HOI trajectory generator. We feed back gradients from the embodiment-aware IK solver into the HOI sampler and jointly optimize the position of the humanoid with respect to the object. Because guidance acts only at inference, one HOI trajectory generator trained on human hand poses transfers across humanoid embodiments without retraining. On ARCTIC trajectories with two contrasting humanoids (Unitree G1 and H1-2), our proposed guidance-based method cuts the retargeted wrist error from 8.9 cm to 0.6 cm while leaving text-to-HOI generation quality intact, and raises task success from 68% to 79% on 50 random G1 tasks and from 62% to 76% on a four-task H1-2 suite. Critically, cheaper alternatives for bridging the embodiment gap fail to beat EmbodyHOI. Post-hoc wrist refinement and compute-matched best-of-N selection both reach lower IK energy than our guidance yet train worse policies. For simulation results, see our website: https://embodyhoi.github.io/

Figures & tables

Fig. 2: EmbodyHOI makes a frozen HOI generator embodiment-aware at inference. Inside every unmasking step of the masked autoregressive generator, a differentiable IK solve on the target humanoid scores the decoded wrist trajectories and places the robot’s base; the gradient of the resulting IK energy is added to the flow-matching velocity field of the hand tokens, so the finger and object tokens generated in later steps form around reachable wrists. We then retarget the guided sample and train a PPO tracking policy on it with decoupled arm and hand rewards. Changing the robot changes only the IK module.

Method	Cost	$E\downarrow$	$\Delta_{\mathrm{wrist}}$ (cm)	Pen. $\downarrow$	SR % $\uparrow$
Unguided, fixed base	$t_{1}$	2.30	–	0.52	68.1 $\pm$ 0.3
+ base optimization	$t_{1}$	1.89	0	0.52	73.3 $\pm$ 0.5
+ IK guidance (ours)	$32\,t_{1}$	1.81	0.3	0.53	79.7 $\pm$ 0.4
Post-hoc IK	$t_{1}$ +IK	0.35	32.6	0.59	32.8 $\pm$ 0.3
Best-of-8 by $E$	$8\,t_{1}$	1.36	13.2	0.52	73.1 $\pm$ 0.6
Best-of-32 by $E$	$32\,t_{1}$	1.21	13.7	0.51	74.0 $\pm$ 0.6

TABLE I: Guidance during sampling trains the best policies, although two alternatives reach lower IK energy. Medians over 51 seed-matched prompts. Cost: sampling time per reference in native samples (

t_{1}{=}5

s);

\Delta_{\mathrm{wrist}}

: wrist displacement (cm) from the seed-matched unguided sample; Pen.: penetration loss; SR: success rate on the four-task suite, mean

\pm

s.e.m. over 5 seeds.

Fig. 3: Only guidance improves on base optimization. Success rate on the four-task suite against sampling cost per reference (Table I ). Base optimization lifts the unguided baseline at no sampling cost; best-of-

N

selection does not measurably improve on it even at 32 times the cost (

0.7\pm 0.8

points).

Fig. 4: Change in IK energy that guidance applies to each seed-matched sample. Guidance makes 83–97% of samples more reachable depending on the robot–generator pair; the remaining increases are small (median 1%), with two outliers on the G1 where sampling propagated the correction into a different trajectory (Sec. IV-B ).

Fig. 5: Base placement, not the IK weights, removes near-singular configurations. Worst-case conditioning of the G1 per arm over the ARCTIC prompts: minimum Jacobian singular value ( $\uparrow$ ) and maximum condition number ( $\downarrow$ , log scale). Unguided samples are solved at the fixed base; guided samples are solved at the optimized base under four IK-weight settings, whose distributions overlap.

	FID $\downarrow$	KID $\downarrow$	Div. $\uparrow$	Top-3 $\uparrow$	Match $\downarrow$	Contact % $\uparrow$	Pen. $<$ 1 cm % $\downarrow$	Jitter $\downarrow$	Wrist err. (cm) $\downarrow$	Wrist err. ( ∘ ) $\downarrow$
Unguided	0.667	0.001	19.77	0.852	3.99	83.5	12.2	0.287	8.9	20.2
IK-guided	0.545	0.003	19.85	0.872	3.97	83.1	11.5	0.293	0.6	4.0

TABLE II: Guidance removes most of the retargeting error with little change in generation quality. Left: generation quality of unguided and IK-guided MAD-HOI on the same 400 ARCTIC test prompts with one evaluator. Right: wrist tracking error of the retargeted G1 after PyRoki IK on the randomly sampled tasks.

	Reference $\rightarrow$ retarget / RL	Avg. SR % $\uparrow$
A	ARCTIC GT, fixed base $\rightarrow$ DexMachina-style	66.8 $\pm$ 0.3
B	Unguided, fixed base $\rightarrow$ DexMachina-style	68.2 $\pm$ 0.9
C	Unguided, optimized base $\rightarrow$ ours	73.6 $\pm$ 0.9
D	Guided (both groups) $\rightarrow$ ours (full)	79.3 $\pm$ 0.6
	D $-$ arm-conditioning group	76.4 $\pm$ 0.3
	D $-$ base-placement group	75.3 $\pm$ 0.3

TABLE III: Guidance, in-loop base placement, and the decoupled reward each raise success on the G1 four-task suite (box open, waffle-iron lift, mixer open+close, microwave lift+place); mean

\pm

s.e.m. over 5 seeds, 10 episodes per task. Rows A–D form a ladder: A

\to

B changes the reference source, B

\to

C changes the RL recipe (DexMachina-style [ 10 ] vs. ours) and adds post-hoc base optimization, and C

\to

D adds IK guidance with in-loop base placement. The lower half removes one component at a time from D. Row D and the IK-guidance row of Table I are independent training runs of the same configuration (79.3 vs. 79.7, within seed error).

Fig. 6: Decoupled arm and hand imitation rewards give the highest success rate on every task of the G1 four-task suite. Per-task success rate under four reward configurations: imitation only, object tracking only, one imitation term over all 39 DoFs, and our decoupled rewards.

Fig. 7: The same generator and IK objective place two humanoids differently: from the same initial placement, guidance moves the H1-2 base (left; 178 cm, 68.5 cm arms) away from the object to avoid over-extension and the G1 base (right; 132 cm, 45 cm arms) closer and higher to stay reachable.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Weight	Description
Wrist position	$w_{\mathrm{pos}}{=}250$	Tracks the target wrist position.
Wrist orientation	$w_{\mathrm{ori}}{=}5$	Tracks the target wrist orientation.
Rest-pose prior	$\lambda_{\mathrm{rest}}{=}10$	Encourages solutions near a nominal arm configuration.
Joint-limit penalty	$w_{\mathrm{joint}}{=}100$	Penalizes configurations outside joint limits.
Smoothness	$w_{\mathrm{smooth}}{=}1$	Penalizes deviation from the previous timestep’s solution.
Root prior	$w_{\mathrm{root}}{=}100$	Keeps the base near the current preferred placement $\mathbf{b}_{0}$ .

Appendix

TABLE IV: Components of the IK guidance objective. Wrist tracking is the primary task; the remaining terms regularize toward kinematically feasible arm and base configurations. The null-space term is a secondary optimization applied after the primary solve.

Parameter	Value
LM iterations per timestep	10
LM damping coefficient	$10^{-6}$
Gradient-norm termination threshold	$10^{-2}$
Implicit differentiation	enabled
Solve variables $\mathbf{u}^{t}$	arm joints $\mathbf{q}^{t}$ + base translation $\mathbf{b}^{t}$
Base orientation	fixed

Appendix

TABLE V: Optimization parameters of the IK guidance module.

Observation	Interpretation
$\{\mathbf{q}^{\mathrm{robot}}_{t-k}\}_{k=0}^{8}$	Robot joint positions over the past 9 steps (proprioception).
$\{\dot{\mathbf{q}}^{\mathrm{robot}}_{t-k}\}_{k=0}^{8}$	Robot joint velocities over the past 9 steps.
$\{\mathbf{a}_{t-k}\}_{k=0}^{8}$	Previous actions over the past 9 steps ( $n_{a}{=}39$ controlled DoFs).
$\phi_{t}$	Normalized episode phase $2t/T_{\max}-1$ .
$\hat{\mathbf{q}}^{\mathrm{robot}}_{t+\Delta}-\mathbf{q}^{\mathrm{robot}}_{t},\hskip 9.24994pt\Delta\in\{1,2,4,8,16\}$	Future reference joint offsets (retargeted trajectory minus current joints).
$T^{\mathrm{obj}}_{t,k},\ R^{\mathrm{obj}}_{t,k},\hskip 9.24994ptk\in\{1,2\}$	3D position and 4D orientation of both articulated object parts.

Appendix

TABLE VI: Policy observations for bimanual manipulation on articulated objects. The reference quantities

\hat{\cdot}

come from the generated (guided or unguided) trajectory, never from ground truth.

Reward term	Interpretation
$r_{\mathrm{task}}=\begin{aligned} &\left\\|T^{\mathrm{obj}}_{t}-\hat{T}^{\mathrm{obj}}_{t}\right\\|_{2}\cdot d_{\mathrm{rot}}(R^{\mathrm{obj}}_{t},\hat{R}^{\mathrm{obj}}_{t})\\ &\cdot-\beta_{a}\left(\alpha^{\mathrm{obj}}_{t}-\hat{\alpha}^{\mathrm{obj}}_{t}\right)^{2}\end{aligned}$	Rewards matching the reference object position, orientation, and articulation in the world frame.
$r_{\mathrm{imi,upper}}=\left\\|\mathbf{q}^{\mathrm{upper}}_{t}-\hat{\mathbf{q}}^{\mathrm{upper}}_{t}\right\\|_{2}^{2}$	Upper-body imitation stream: tracking of the 15 torso and arm joints.
$r_{\mathrm{imi,dex}}=\left\\|\mathbf{q}^{\mathrm{dex}}_{t}-\hat{\mathbf{q}}^{\mathrm{dex}}_{t}\right\\|_{2}^{2}$	Dexterous imitation stream: tracking of the 24 finger joints.
$r_{\mathrm{contact}}=\left\\|\mathbf{c}_{t}-\hat{\mathbf{c}}_{t}\right\\|_{2}$	Rewards matching simulated contact locations to the retargeted reference contacts on hand links.
$r_{\mathrm{act}}=\left\\|\mathbf{a}_{t}\right\\|_{2}$	Penalizes large actions.
$r_{\mathrm{act\text{-}rate}}=\left\\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\right\\|_{2}^{2}$	Penalizes large action rates.

Appendix

TABLE VII: Reward terms. The two imitation streams are kept separate rather than summed over all 39 DoFs; Table III shows this decoupling is worth 5.2 points of success rate.

Termination condition	Interpretation
$t\geq T_{\max}$	Ends the episode at the task-dependent horizon.
$\mathrm{NaN}$ in simulation state	Ends the episode on numerical failure in robot or object state.
$\max_{i}\ \tfrac{1}{2}\left(q^{\mathrm{upper}}_{t,i}-\hat{q}^{\mathrm{upper}}_{t,i}\right)^{2}>\epsilon_{\mathrm{upper}}$	Ends the episode when any arm or torso joint deviates too far: a large arm failure ends the rollout.
$\mathrm{mean}_{i}\ \tfrac{1}{2}\left(q^{\mathrm{dex}}_{t,i}-\hat{q}^{\mathrm{dex}}_{t,i}\right)^{2}>\epsilon_{\mathrm{dex}}$	Ends the episode on average finger deviation: an isolated finger error is not catastrophic.
$p^{\mathrm{obj}}_{t,z}<h_{\mathrm{table}}$	Ends the episode when the object falls below the table.
$\tfrac{1}{2}\left(\alpha^{\mathrm{obj}}_{t}-\hat{\alpha}^{\mathrm{obj}}_{t}\right)^{2}>\epsilon_{a}$	Ends the episode when object articulation departs the reference trajectory.

Appendix

TABLE VIII: Episode termination conditions. Thresholds are applied to the half-squared joint error, so

\epsilon_{\mathrm{upper}}{=}0.12

corresponds to a

0.49

rad deviation on the worst joint. The asymmetry between the maximum (arm) and the mean (hand) is the termination-side counterpart of the decoupled rewards.

Category	Parameter	Value
PPO / training
	Parallel environments $B$	8192
	PPO iterations	2000
	Rollout horizon (steps per env)	16
	Learning rate	$3{\times}10^{-4}$
	Discount $\gamma$ / GAE $\tau$	0.99 / 0.95

Appendix

TABLE IX: RL hyperparameters. The same values are used for every condition in Tables I and III and for both embodiments; only the reference trajectory differs between conditions. Domain randomization and observation/action noise are omitted.

Fig. 8: Worst-case conditioning of the Unitree H1-2 per arm over the ARCTIC prompts: minimum Jacobian singular value (

\uparrow

) and maximum condition number (

\downarrow

, log scale). Compare Fig. 5 for the G1.

Setting	Median $E\downarrow$	$\sigma_{\min}\uparrow$	FID $\downarrow$	Top-3 $\uparrow$
Default (ours)	1.81	0.082	0.545	0.872
$\lambda_{\mathrm{rest}}{=}0$	2.05	0.054	0.615	0.869
$\lambda_{\mathrm{manip}}{=}0$	1.94	0.082	0.650	0.861
$\lambda_{\mathrm{manip}}{\times}3$	1.78	0.086	0.869	0.835
$w_{\mathrm{root}}{\times}0.3$	1.88	0.083	0.810	0.830

Appendix

TABLE X: Sensitivity to IK objective weights on the same 400 ARCTIC test prompts. The default setting is the one used in all other experiments and was fixed before this sweep was run.

	% Reaching lower $E$ than EmbodyHOI
Method	Easy	Med.	Hard	Median $E\downarrow$	SR $\%\uparrow$
Best-of-8 by $E$	65%	88%	75%	1.36	73.1%
Best-of-32 by $E$	88%	94%	88%	1.21	74.0%
EmbodyHOI (ref.)	–	–	–	1.81	79.7%

Appendix

TABLE XI: Selection’s energy advantage does not depend on difficulty. 51 prompts split into % terciles by unguided optimized-base

E

(bin edges

E<

1.80, 1.80–3.00,

>

3.00; %

n{=}17

each).

Subset	$n$	Unguided SR %	Guided SR % $\uparrow$
All tasks	50	68.1	79.2
Well-conditioned ( $\sigma_{\min}\geq 0.1$ )	40	70.7	78.3
Hard ( $\sigma_{\min}<0.1$ )	10	57.9	82.1

Appendix

TABLE XII: Success rate on the 50 randomly sampled G1 tasks, split by

\sigma_{\min}

of the unguided reference; mean

\pm

s.e.m. over 5 seeds.

Stage	Runtime	Hardware
MAD-HOI generation (unguided)	5 s	A100
IK-guided generation (ours)	160 s	A100
Post-hoc IK refinement	55 s	A100
Best-of-32 selection	160 s	A100
Retargeting per sequence	50 s	CPU
PPO training per policy	$\approx$ 4 h	RTX 4090

Appendix

TABLE XIII: Approximate runtime per pipeline stage. Generation was measured on an NVIDIA A100 and PPO training on an RTX 4090. Runtimes are approximate and vary with sequence length and task.

Fig. 10: Additional articulated-object manipulation tasks executed on the Unitree G1 by policies trained from EmbodyHOI references, spanning opening, closing, lifting, and bimanual behaviors across the ARCTIC object set.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 84 papers

Jun 11, 2026

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present GenHOI, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.

Zhihai Bi, Qiang Zhang, Guoyang Zhao +8

The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1

Contact-Rich Robotic Manipulation Robot Motion Generation Robot Imitation Learning Humanoid Robotics Human-Object Interaction

Ask PDF

Sep 28, 2026

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object. We publicly release the code and the retargeted motion dataset.

Jihwan Shin, Adrià López Escoriza, Junzhe He +2

Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics Motion Retargeting Humanoid Robotics Human-Object Interaction

Ask PDF

Jul 2, 2026

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.

Mingyeong Song, Jungbin Cho, Jisoo Kim +5

Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1

HOI Generation Diffusion Models Hand-Object Interaction

Ask PDF

Abstract

HOI Generation

Explore similar work

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation