LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs
Organizations: College of Computing & Data Science, Nanyang Technological University, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Honda Research Institute Europe (HRI-EU), Offenbach am Main, Germany
Abstract
The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.
Figures & tables
| Text-to-3D Model / | Shap-E | Trellis | ||||
|---|---|---|---|---|---|---|
| LLM | Baseline | Ours | Improvement | Baseline | Ours | Improvement |
| Mistral 3.1 Small | 0.9557 | +14.46% | 0.8494 | +4.46% | ||
| GPT 3.5 | 0.9637 | +7.41% | 0.9531 | +11.32% | ||
| GPT 4o Mini | +45.12% | +62.89% | ||||
| Gemini 2.0 Lite | 0.9173 | +19.18% | 0.9634 | +106.74% | ||
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Generalize to Target Domain | Conform to Physics | |
| Method | ||
| Diffusion-SDF Li et al. [2023b] | ✓ | ✗ |
| Shap-E Jun and Nichol [2023] | ✓ | ✗ |
| Trellis Xiang et al. [2024] | ✓ | ✗ |
| LLaMA-Mesh Wang et al. [2024] | ✓ | ✗ |
| Fun3D Xu et al. [2025] | ✗ | ✓ |
| Provider | Model Name | # Params | Cut-Off Knowledge | Temperature |
|---|---|---|---|---|
| OpenAI | GPT 3.5 Turbo | undisclosed | Sep 2021 | 0.90 to 1.30 |
| OpenAI | GPT 4o Mini | undisclosed | Oct 2023 | 0.90 to 1.30 |
| Gemini 2.0 Flash Lite | undisclosed | Jun 2024 | 0.15 to 1.00 | |
| Mistral | Mistral 3.1 Small | 24B | Oct 2023 ( ALLMO.ai [] ) | 0.15 to 1.00 |
| Model Name/Variant | Reference | Type | No. Of Params |
|---|---|---|---|
| Shap-E | Jun and Nichol [2023] | Text-to-3D | 300M |
| Trellis Text Large | Xiang et al. [2024] | Text-to-3D | 1.1B |
3D Asset Generation
3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.
See all 176 papers