LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs
Authors: Melvin Wong, Yueming Lyu, Thiago Rios, Stefan Menzel, Yew-Soon Ong
Organizations: College of Computing & Data Science, Nanyang Technological University, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · Honda Research Institute Europe (HRI-EU), Offenbach am Main, Germany
The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.
Figures & tables
Figure 1: Most representative vehicles generated with various conventional LLM-to-3D models using LLM-generated prompts (left) and physically conforming novel vehicles generated with LLM-to-Phy3D (right).
Figure 2: Aerodynamic Visualization of two vehicles generated (a) without considering physical constraints and (b) with physical constraints. Vehicle shape has a direct impact on the aerodynamic (drag) performance. Some aerodynamically efficient (low drag) characteristics include attached streamline flow from the front to the rear, and small pressure differences between the front and rear. In contrast, an aerodynamically inefficient (high drag) vehicle exhibits opposing characteristics, including separated streamline flow and pronounced pressure differences.
Text-to-3D Model /
Shap-E
Trellis
LLM
Baseline
Ours
Improvement
Baseline
Ours
Improvement
Mistral 3.1 Small
0.8350
0.9557
+14.46%
0.8131
0.8494
+4.46%
GPT 3.5
0.8972
0.9637
+7.41%
0.8562
0.9531
+11.32%
GPT 4o Mini
0.6891
1.0000
+45.12%
0.6139
1.0000
+62.89%
Gemini 2.0 Lite
0.7697
0.9173
+19.18%
0.4660
0.9634
+106.74%
Table 1: Comparison of LLM-to-Phy3D and Baselines DPAR in Producing Physically Conform Target Domain 3D Artifacts
Figure 3: Search optimization score (Equation 3 ) of LLM-to-Phy3D with different LLM and text-to-3D generative models combinations.
Figure 4: Results of the aerodynamic profile of cars generated with LLM-to-Phy3D at specific iterations and compared with a representative car generated with the best baseline LLM-to-3D method. Note that the lower the drag, the better the physical performance of the car.
Figure 5: Ablation Studies on the effectiveness of different terms in the objective function on generating 3D artifacts (Top Row: LLM with Shap-E, Bottom Row: LLM with Trellis).
Figure 6: Example of geometric novelty between a reference car (in grey solids) and a car generated with LLM-to-Phy3D (in orange wireframe). The novelty heatmaps presented in the top row are the multi-view results of Fnovelty(.) , which highlight the regions contributing to the geometric novelty score of the generated car compared with the target one. Multiple views (bottom row) of the generated car overlay on top of the target car show regions corresponding to the contrasting body shapes attributed to the difference in aerodynamic performance.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Illustration of virtual wind tunnel for aerodynamics physics simulation. In a simplified controlled environment, the computational domain within the yellow wireframe box simulates the flow of particles from inlet to outlet, as shown in the directional black vector arrows. The physical interactions between the particles and the 3D object of interest are determined by the conditions at the boundary (skin material and surface topology) of the object. These interactions propagated throughout the computational domain form the aerodynamics performance of the 3D object, which can be used as a valid fitness score in a search.
Figure 8: Example of objects rendered under different camera projections and overlay projected objects under the same and different projections to demonstrate the geometric distortions. As the perspective projection does not preserve dimensions and ratios, geometric distortions occur, particularly at the far end of objects, which can cause a violation of the ordering of geometric novelty between the target object and the generated 3D object.
Generalize to Target Domain
Conform to Physics
Method
Diffusion-SDF Li et al. [2023b]
✓
✗
Shap-E Jun and Nichol [2023]
✓
✗
Trellis Xiang et al. [2024]
✓
✗
LLaMA-Mesh Wang et al. [2024]
✓
✗
Fun3D Xu et al. [2025]
✗
✓
Appendix
Table 2: Overview of Text-to-3D Generation Methods
Figure 9: Example of meta prompt designed for aerodynamic design optimization task. The instructions highlighted in green provide contextual information on the role, the search task the LLM needs to solve, and the operating constraints it must follow. On the other hand, the steps (in pink provide the intermediate sequential operations LLM needs to follow strictly for producing new prompt candidates. Moreover, LLM is instructed to embed the domain specification in the prompts it generates, following the prefix provided in blue . Every iteration after initialization will provide good exemplars of its previous attempt (as shown in gray ) to guide LLM in producing better prompts.
Provider
Model Name
# Params
Cut-Off Knowledge
Temperature
OpenAI
GPT 3.5 Turbo
undisclosed
Sep 2021
0.90 to 1.30
OpenAI
GPT 4o Mini
undisclosed
Oct 2023
0.90 to 1.30
Google
Gemini 2.0 Flash Lite
undisclosed
Jun 2024
0.15 to 1.00
Mistral
Mistral 3.1 Small
24B
∼ Oct 2023 ( ALLMO.ai [] )
0.15 to 1.00
Appendix
Table 3: Benchmark Large Language Models
Model Name/Variant
Reference
Type
No. Of Params
Shap-E
Jun and Nichol [2023]
Text-to-3D
300M
Trellis Text Large
Xiang et al. [2024]
Text-to-3D
1.1B
Appendix
Table 4: Benchmark 3D Generative Models
Figure 10: Similarity of 3D generated artifacts between and in various LLMs in the last iteration.
Figure 11: Examples of 3D cars generated with existing GPT-4o-Mini and Trellis text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 12: Examples of 3D cars generated with existing GPT-4o-Mini and Shap-E text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 13: Examples of 3D cars generated with existing GPT-3.5-Turbo and Trellis text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 14: Examples of 3D cars generated with existing GPT-3.5-Turbo and Shap-E text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 15: Examples of 3D cars generated with existing Gemini-2.0-Lite and Trellis text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 16: Examples of 3D cars generated with existing Gemini-2.0-Lite and Shap-E text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 17: Examples of 3D cars generated with existing Mistral-3.1-Small and Trellis text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Figure 18: Examples of 3D cars generated with existing Mistral-3.1-Small and Shap-E text-to-3D generative model (Left) and with LLM-to-Phy3D (Right). Note that the lower the aerodynamic drag, the better the physical performance of the generated car.
Recent advances in 3D content generation have amplified demand for dynamic models that are both visually realistic and physically consistent. However, state-of-the-art video diffusion models frequently produce implausible results such as momentum violations and object interpenetrations. Existing physics-aware approaches often rely on task-specific fine-tuning or supervised data, which limits their scalability and applicability. To address the challenge, we present PhyMAGIC, a training-free framework that generates physically consistent motion from a single image. PhyMAGIC integrates a pre-trained image-to-video diffusion model, confidence-guided reasoning via LLMs, and a differentiable physics simulator to produce 3D assets ready for downstream physical simulation without fine-tuning or manual supervision. By iteratively refining motion prompts using LLM-derived confidence scores and leveraging simulation feedback, PhyMAGIC steers generation toward physically consistent dynamics. Comprehensive experiments demonstrate that PhyMAGIC outperforms state-of-the-art video generators and physics-aware baselines, enhancing physical property inference and motion-text alignment while maintaining visual fidelity.
Siwei Meng, Yawei Luo, Ping Liu
Department of Computer Science & Engineering, University of Nevada, Reno, USA · School of Software Technology, Zhejiang University, China
Large Language Models (LLMs) can generate Computer-Aided Design (CAD), yet lack physical comprehension required for reliable engineering design. Instead of attempting to implicitly learn physical laws from data, we propose a Hybrid Agentic-Physical Architecture that embeds validated knowledge-based engineering tools directly into the decision making loop of autonomous AI agents. In this framework, engineering design is formulated as a closed-loop, sequential decision making process guided by explicit physical verification. Based on a load case, dedicated agents iteratively plan, generate, evaluate, and revise engineering designs using knowledge-based tools as a feedback signal. We introduce a benchmark dataset and metrics for assessing functional validity in generative CAD. Our system generates more complex and physically verified designs, with a 4.2 increase in structural complexity and improving compile rate by 3.5% compared to similar agentic methods. The codebase, prompts and dataset will be made publicly available to support reproducibility and future research.
Elias Berger, Muhammad Usama, Jan Mehlstäubl +2
Dresden University of Technology, Dresden, Germany · MAN Truck & Bus SE, Munich, Germany · German Research Center for Artificial Intelligence, Kaiserslautern, Germany +1
Creating high-quality 3D content has traditionally required specialist expertise and experience with complex tools. The growing ecosystem of AI-assisted creativity demands accessible and intuitive 3D generation systems. Recent advances in Large Language Models (LLMs) offer promising directions, yet extending their generative capabilities to 3D remains fundamentally challenging because existing 3D representations are not naturally compatible with language-based reasoning. Current approaches either depend on expensive fine-tuning or impose restrictive structural priors that sacrifice expressiveness and exclude organic shapes. We argue that the mismatch between LLM-native reasoning and conventional 3D parameterizations is a core bottleneck of LLM-driven 3D generation. To address this, we introduce the Hierarchical Patch Scaffold (HPS), a representation designed around semantically meaningful, interpretable part-level abstractions that are naturally understood by both humans and LLMs, enabling expressive, editable, and open-domain 3D generation without imposing restrictive geometric priors or expensive fine-tuning. Building upon HPS, we introduce SurfScaff3D, an interactive framework for text and image-to-3D generation. HPS inherits the flexibility of B-spline patches, while exposing an interface that leverages open-world capabilities of foundation models to reliably generate 3D. Its hierarchical part-level semantic labels enable targeted, natural-language edits at any level of granularity without re-running generation. Further, HPS offers explicit per-patch control over surface detail, enabling more expressive geometry capable of representing arbitrary objects. Extensive experiments against state-of-the-art LLM-based 3D generation methods demonstrate strong geometric fidelity, semantic alignment, and flexible editing across diverse object categories.