Generating photorealistic 3D hand-object interactions (HOIs) from text is important for applications like robotic grasping and AR/VR content creation. In practice, however, achieving both visual fidelity and physical plausibility remains difficult, as mesh extraction from text-generated Gaussians is inherently ill-posed and the resulting meshes are often unreliable for physics-based optimization. We present THOM, a training-free framework that generates physically plausible 3D HOI meshes directly from text prompts, without requiring template object meshes. THOM follows a two-stage pipeline: it first generates hand and object Gaussians guided by text, and then refines their interaction using physics-based optimization. To enable reliable interaction modeling, we introduce a mesh extraction method with an explicit vertex-to-Gaussian mapping, which enables topology-aware regularization. We further improve physical plausibility through contact-aware optimization and vision-language model (VLM)-guided translation refinement. Extensive experiments show that THOM produces high-quality HOIs with strong text alignment, visual realism, and interaction plausibility.
Figures & tables
Figure 2: Overall pipeline of THOM. In the first stage, object and hand Gaussians are separately generated from the text prompts. During the second stage, we jointly refine the HOI Gaussians and HOI parameters. At HOI initialization, we refine the hand translation with VLM-guided refinement. During HOI parameters optimization, we introduce physics-based optimization via distance-adaptive contact losses and reposition loss. For both the first and the second stage, Laplacian regularization is applied for topological consistency.
Table 1: Comparison with state-of-the-art text-to-3D and text-to-3D human-object interaction generation methods. *: Adapted to generate hand-object interactions.
Figure 4: Qualitative comparison with human-object interaction methods. *: Adapted to generate hand-object interactions.
Method
Max. pene. ↓
Mean. pene. ↓
Contact ratio ↑
Disp. ↓
G-HOP
1.30 ×10−2
9.52 ×10−3
0.15
1.00
Text2HOI
5.16 ×10−4
2.28 ×10−4
0.98
0.25
Ours
2.20 ×10−5
1.24 ×10−5
0.95
0.20
Table 2: Physics and simulation evaluation results with learning-based HOI generation methods.
w/o Llap
w/o Lhoi-phys
w/o Lrepos
w/o VLM
all
30.4
30.3
31.0
31.3
31.4
Table 3: Ablation results with CLIP score. Higher scores indicate better performance. Our complete method shows the best score.
Method
CLIP ↑
Pene. ↓
Downsample
30.9
7.3 ×10−5
Marching Cubes
26.8
2.3 ×10−5
SuGaR
30.4
3.8 ×10−5
Ours
31.4
2.2 ×10−5
Table 4: Ablation of object mesh reconstruction methods.
w/o phys. opt
w/o VLM
Ours (all)
13
25
61
Table 5: Ablation of physics optimization and VLM refinement using GPT-4o select metric. Values indicate the preference frequency (one sample excluded due to GPT-4o non-response).
Figure 5: VLM refinement results. Left: “A right hand calling a smartphone”. Middle: “A right hand using a hammer”. Right: “A right hand inspecting Paddington Bear”.
Figure 6: User preference study for THOM.
Method
CLIP ↑
T 3 -Align ↑
Contact ratio ↑
Disp. ↓
Baseline
30.4
1.9
0.98
0.25
+ phys. opt
31.3
2.0
0.95
0.20
+ VLM refine (full)
31.4
2.6
0.95
0.20
Table 6: Contribution of physics optimization and VLM-guided translation refinement.
Method
Llap
Lpene
Loc
Lhc
concise mesh
Lrepos
Lcons
VLM refine
CLIP ↑
w/o Llap
×
◯
◯
◯
◯
◯
◯
◯
30.4
w/o Lpene
◯
×
◯
◯
◯
◯
◯
◯
30.4
w/o Loc
◯
◯
×
◯
◯
◯
◯
◯
30.8
w/o Lhc
◯
◯
◯
×
◯
◯
◯
◯
30.9
w/o concise mesh
◯
◯
◯
◯
×
◯
◯
◯
31.0
w/o Lrepos
◯
◯
◯
◯
◯
×
◯
◯
31.0
Table 7: Comprehensive ablation results of our proposed method.
Figure 7: Diverse generation results of our method from various HOI prompts.
Figure 8: Visualization of hand and object contact maps. Input prompt: ”A right hand using a tambourine.”
Text-conditioned 3D generation has progressed rapidly for images and isolated objects, but producing a hand-object mesh remains challenging: the output must preserve language semantics, cross-view consistency, object geometry, articulated hand shape, and physically plausible contact. We present TextHOI-3D, a staged framework that uses generated multi-view observations as an explicit interface between text-conditioned visual generation and geometry-aware hand-object recovery. TextHOI-3D learns a compact VQ token space for fixed-camera hand-object observations, predicts multi-view visual tokens from text with a CLIP-conditioned visual autoregressive model, and recovers a unified hand-object mesh through prior initialization, multi-view joint optimization, and anti-penetration refinement. The design separates semantic generation from geometric recovery while keeping both stages connected by a discrete multi-view representation. On HO3D-derived evaluations, the multi-view setting reduces object CD from 17.26 mm to 4.92 mm and penetration volume from 5.3721 cm^3 to 0.2193 cm^3 compared with a single-view counterpart, while improving hand errors and surface F-scores. These results support multi-view visual tokens as an effective intermediate representation for text-driven 3D hand-object mesh creation.
Zixiong Hao, Zhencun Jiang
Technical University of Munich School of Computation, Information and Technology · Tongji University Shanghai Research Institute for Intelligent Autonomous Systems
Text driven hand object interaction (HOI) generation is gaining attention for immersive applications and robotics, yet producing physically plausible interactions remains challenging. Even when individual motions appear natural, small contact errors can cause conspicuous artifacts such as floating and interpenetration. Prior methods mitigate these issues using explicit contact cues or implicit grasp priors, but typically rely on multi stage pipelines and fail to model temporally evolving contact. We present JointHOI, a single stage diffusion framework that jointly generates 3D hand object motion and dynamic, distance based contact maps from text. By treating contact as an auxiliary inner modality, joint generation enables the model to learn contact motion coupling during training. At inference, contact guided sampling enforces consistency between generated contact maps and motion implied geometry, improving temporal stability and reducing penetration and floating. Experiments on GRAB and ARCTIC demonstrate consistent improvements in text adherence and physical plausibility over prior methods.
Mingyeong Song, Jungbin Cho, Jisoo Kim +5
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1
Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.
Agniv Sharma, Xianghui Xie, Tom Fischer +2
University of Tübingen · Technische Universit4t N4rnberg · Tübingen AI Center +1