cs.CVApr 3, 2026

THOM: Generating Physically Plausible Hand-Object Meshes From Text

Authors: Uyoung Jeong, Yihalem Yimolal Tiruneh, Hyung Jin Chang, Seungryul Baek, Kwang In Kim

Organizations: UNIST · University of Birmingham · POSTECH

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Generating photorealistic 3D hand-object interactions (HOIs) from text is important for applications like robotic grasping and AR/VR content creation. In practice, however, achieving both visual fidelity and physical plausibility remains difficult, as mesh extraction from text-generated Gaussians is inherently ill-posed and the resulting meshes are often unreliable for physics-based optimization. We present THOM, a training-free framework that generates physically plausible 3D HOI meshes directly from text prompts, without requiring template object meshes. THOM follows a two-stage pipeline: it first generates hand and object Gaussians guided by text, and then refines their interaction using physics-based optimization. To enable reliable interaction modeling, we introduce a mesh extraction method with an explicit vertex-to-Gaussian mapping, which enables topology-aware regularization. We further improve physical plausibility through contact-aware optimization and vision-language model (VLM)-guided translation refinement. Extensive experiments show that THOM produces high-quality HOIs with strong text alignment, visual realism, and interaction plausibility.

Figures & tables

Figure 2: Overall pipeline of THOM. In the first stage, object and hand Gaussians are separately generated from the text prompts. During the second stage, we jointly refine the HOI Gaussians and HOI parameters. At HOI initialization, we refine the hand translation with VLM-guided refinement. During HOI parameters optimization, we introduce physics-based optimization via distance-adaptive contact losses and reposition loss. For both the first and the second stage, Laplacian regularization is applied for topological consistency.
Figure 3: Qualitative comparisons of ProlificDreamer [ 68 ] , GaussianDreamerPro [ 76 ] , Hash3D [ 72 ] and ours.
MethodTimeCLIP ↑\uparrowT 3 -Align ↑\uparrow
DreamFusion [ 52 ]35 min.27.11.3
ProlificDreamer [ 68 ]3 h29.11.2
GaussianDreamerPro [ 76 ]1 h 30 min.28.51.4
Hash3D [ 72 ]25 min.27.61.9
InterFusion* [ 10 ]2 h22.51.5
DreamHOI* [ 85 ]3 h 20 min.27.81.6
Table 1: Comparison with state-of-the-art text-to-3D and text-to-3D human-object interaction generation methods. *: Adapted to generate hand-object interactions.
Figure 4: Qualitative comparison with human-object interaction methods. *: Adapted to generate hand-object interactions.
MethodMax. pene. ↓\downarrowMean. pene. ↓\downarrowContact ratio ↑\uparrowDisp. ↓\downarrow
G-HOP1.30 ×10−2\times 10^{-2}9.52 ×10−3\times 10^{-3}0.151.00
Text2HOI5.16 ×10−4\times 10^{-4}2.28 ×10−4\times 10^{-4}0.980.25
Ours2.20 ×10−5\times\bm{10^{-5}}1.24 ×10−5\times\bm{10^{-5}}0.950.20
Table 2: Physics and simulation evaluation results with learning-based HOI generation methods.
w/o Llap\mathcal{L}_{\text{lap}}w/o Lhoi-phys\mathcal{L}_{\text{hoi-phys}}w/o Lrepos\mathcal{L}_{\text{repos}}w/o VLMall
30.430.331.031.331.4
Table 3: Ablation results with CLIP score. Higher scores indicate better performance. Our complete method shows the best score.
MethodCLIP ↑\uparrowPene. ↓\downarrow
Downsample30.97.3 ×10−5\times 10^{-5}
Marching Cubes26.82.3 ×10−5\times 10^{-5}
SuGaR30.43.8 ×10−5\times 10^{-5}
Ours31.42.2 ×10−5\times\bm{10^{-5}}
Table 4: Ablation of object mesh reconstruction methods.
w/o phys. optw/o VLMOurs (all)
132561
Table 5: Ablation of physics optimization and VLM refinement using GPT-4o select metric. Values indicate the preference frequency (one sample excluded due to GPT-4o non-response).
Figure 5: VLM refinement results. Left: “A right hand calling a smartphone”. Middle: “A right hand using a hammer”. Right: “A right hand inspecting Paddington Bear”.
Figure 6: User preference study for THOM.
MethodCLIP ↑\uparrowT 3 -Align ↑\uparrowContact ratio ↑\uparrowDisp. ↓\downarrow
Baseline30.41.90.980.25
+ phys. opt31.32.00.950.20
+ VLM refine (full)31.42.60.950.20
Table 6: Contribution of physics optimization and VLM-guided translation refinement.
MethodLlap\mathcal{L}_{\text{lap}}Lpene\mathcal{L}_{\text{pene}}Loc\mathcal{L}_{\text{oc}}Lhc\mathcal{L}_{\text{hc}}concise meshLrepos\mathcal{L}_{\text{repos}}Lcons\mathcal{L}_{\text{cons}}VLM refineCLIP ↑\uparrow
w/o Llap\mathcal{L}_{\text{lap}}×\times◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc30.4
w/o Lpene\mathcal{L}_{\text{pene}}◯\bigcirc×\times◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc30.4
w/o Loc\mathcal{L}_{\text{oc}}◯\bigcirc◯\bigcirc×\times◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc30.8
w/o Lhc\mathcal{L}_{\text{hc}}◯\bigcirc◯\bigcirc◯\bigcirc×\times◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc30.9
w/o concise mesh◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc×\times◯\bigcirc◯\bigcirc◯\bigcirc31.0
w/o Lrepos\mathcal{L}_{\text{repos}}◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc◯\bigcirc×\times◯\bigcirc◯\bigcirc31.0
Table 7: Comprehensive ablation results of our proposed method.
Figure 7: Diverse generation results of our method from various HOI prompts.
Figure 8: Visualization of hand and object contact maps. Input prompt: ”A right hand using a tambourine.”
Figure 9: Comparison of concise mesh extraction methods. Input prompt: ”a hamburger.”
Figure 10: Qualitative results of our concise mesh extraction.
Table 17
Figure 11: Failure cases of our method. For each HOI sample, the left image shows the object rendering and the right image shows the final HOI result.
Figure 12: Exemplar VLM prompting inputs. Upper images: Input images for VLM prompting. Lower text: Text instruction for VLM prompting.
grabgrasptouchdrink
liftbrowseeatinspect
brushusecookshake
playcleanflysqueeze
setopenseecall
hand overpasspourswitch on
Table 10: Interaction types used in evaluation prompts.
λlap,μ\lambda_{\text{lap,}\bm{\mu}}1.0×1051.0\times 10^{5}λlap,c\lambda_{\text{lap,c}}1.0×1051.0\times 10^{5}
λlap,s\lambda_{\text{lap,s}}1.0×1051.0\times 10^{5}ss≈7.39\approx 7.39
λpene\lambda_{\text{pene}}10.0λhc\lambda_{\text{hc}}0.5
λoc\lambda_{\text{oc}}0.5λrepos\lambda_{\text{repos}}1.0
λcons\lambda_{\text{cons}}1.0HOI param. lr.0.01
Table 11: Hyperparameters used in our proposed framework.

Explore similar work

CardsList

  1. TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization

    Jun 10, 2026Zixiong Hao, Zhencun JiangHOI Generation3D Mesh Generation

    Technical University of Munich School of Computation, Information and Technology · Tongji University Shanghai Research Institute for Intelligent Autonomous Systems

  2. JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

    Jul 2, 2026Mingyeong Song, Jungbin Cho, Jisoo Kim +5HOI GenerationDiffusion Models

    Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1

  3. Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D

    Mar 12, 2026Agniv Sharma, Xianghui Xie, Tom Fischer +2HOI Generation3D Asset Generation

    University of Tübingen · Technische Universit4t N4rnberg · Tübingen AI Center +1