Part-Level 3D Gaussian Vehicle Generation with Joint and Hinge Axis Estimation
Authors: Shiyao Qian, Yuan Ren, Dongfeng Bai, Bingbing Liu
Organizations: The authors are with Huawei Noah’s Ark Lab. · Shiyao Qian is with the University of Toronto and contributed to this work during an internship at Huawei Noah’s Ark Lab, Canada.
Simulation is essential for autonomous driving, yet current frameworks often model vehicles as rigid assets and fail to capture part-level articulation. With perception algorithms increasingly leveraging dynamics such as wheel steering or door opening, realistic simulation requires animatable vehicle representations. Existing CAD-based pipelines are limited by library coverage and fixed templates, preventing faithful reconstruction of in-the-wild instances. We propose a generative framework that, from a single image or sparse multi-view input, synthesizes an animatable 3D Gaussian vehicle. Our method addresses two challenges: (i) large 3D asset generators are optimized for static quality but not articulation, leading to distortions at part boundaries when animated; and (ii) segmentation alone cannot provide the kinematic parameters required for motion. To overcome this, we introduce a part-edge refinement module that enforces exclusive Gaussian ownership and a kinematic reasoning head that predicts joint positions and hinge axes of movable parts. Together, these components enable faithful part-aware simulation, bridging the gap between static generation and animatable vehicle models.
Figures & tables
Fig. 1: Overview of the PointNet++ hierarchical feature learning architecture. For segmentation, learned features are interpolated back to the original point set to produce per-point scores. For classification, an additional abstraction layer aggregates features into a global representation, which is passed through fully connected layers to predict object class scores.
Fig. 2: Overview of our pipeline. TRELLIS generates a 3DGS asset from four multi-view images of the vehicle. The asset is converted into a point cloud, which is processed by the kinematic reasoning module for part-level segmentation and motion axes estimation. In parallel, SAGA segments the 3DGS representation directly, with additional movable part masks integrated alongside auto-segmented ones. The resulting clusters are aligned with the PointNet++ segmentation outputs to retain six target parts. Finally, part geometries are refined at the boundary in the part-edge refinement module, and all segmented parts with their predicted motion axes are combined to produce an animatable 3DGS vehicle.
Fig. 3: PointNet++ training data examples. Point cloud data are colored according to their part label, with joint and hinge positions marked as black balls.
Fig. 4: Rendering of all car parts before and after the boundary finetuning step
Fig. 5: Qualitative comparison of part-manipulated vehicle renderings across different ablation pipelines. From top to bottom: (1) SAGA + Trellis, (2) + Mask Insertion, (3) + Boundary Finetuning (ours), and (4) Ground Truth. Mask insertion leads to more complete part clustering, while the final boundary finetuning module produces cleaner edges and reduces floating Gaussians, yielding renderings that align more closely with the ground-truth vehicles in perceptual quality.
Components
CLIP (↑)
LPIPS (↓)
TRELLIS + SAGA
0.848
0.305
TRELLIS + SAGA + Mask insertion
0.855
0.304
TRELLIS + SAGA + Mask insertion + Edge ft (ours)
0.873
0.283
TABLE I: Ablation study of our pipeline. Performance with different components enabled are recorded. Adding mask insertion improves similarity, while the full pipeline with boundary finetuning produces the best result – the highest CLIP similarity (↑) and lowest LPIPS distance (↓) .
Creating realistic and simulation-ready 3D assets is crucial for autonomous driving research and virtual environment construction. However, existing 3D vehicle generation methods are often trained on synthetic data with significant domain gaps from real-world distributions. The generated models often exhibit arbitrary poses and undefined scales, resulting in poor visual consistency when integrated into driving scenes. In this paper, we present Unposed-to-3D, a novel framework that learns to reconstruct 3D vehicles from real-world driving images using image-only supervision. Our approach consists of two stages. In the first stage, we train an image-to-3D reconstruction network using posed images with known camera parameters. In the second stage, we remove camera supervision and use a camera prediction head that directly estimates the camera parameters from unposed images. The predicted pose is then used for differentiable rendering to provide self-supervised photometric feedback, enabling the model to learn 3D geometry purely from unposed images. To ensure simulation readiness, we further introduce a scale-aware module to predict real-world size information, and a harmonization module that adapts the generated vehicles to the target driving scene with consistent lighting and appearance. Extensive experiments demonstrate that Unposed-to-3D effectively reconstructs realistic, pose-consistent, and harmonized 3D vehicle models from real-world images, providing a scalable path toward creating high-quality assets for driving scene simulation and digital twin environments.
Hongyuan Liu, Bochao Zou, Qiankun Liu +10
University of Science and Technology Beijing · Li Auto Inc.
High-quality 3D vehicle assets are essential for autonomous driving simulation. Although multi-view diffusion-based paradigms enable controllable single-image reconstruction, they typically produce limited viewpoints and exhibit cross-view geometric inconsistencies, thereby reducing reconstruction fidelity in real-world scenarios. In this work, we introduce 3DCarGen, a scalable single-view 3D car generation framework designed for real-world images by synthesizing an arbitrary number of 3D-consistent multi-view images. Specifically, given a single image as input, we first synthesize a set of images from fixed viewpoints. These images are then fed into a feed-forward reconstruction model, resulting in a coarse 3D representation based on 3D Gaussian Splatting. Conditioned on this explicit 3D prior, our multi-view diffusion model generates 3D-consistent images from arbitrary camera viewpoints. We further extend a fast mesh reconstruction algorithm by incorporating color-normal joint optimization to recover detailed and coherent 3D vehicle models from the synthesized dense views. Extensive experiments on synthetic and real-world datasets demonstrate that our approach achieves robust geometric consistency and reconstruction fidelity compared to existing methods. Project page: https://honglixiao.github.io/3dcargen.github.io/.
Hongli Xiao, Youjian Zhang, Yaohui Jin +3
Shanghai Jiao Tong University, China · Academy of Military Science, China · The University of Sydney, Australia +1
High-quality 3D assets for traffic participants are critical for multi-sensor simulation, which is essential for the safe end-to-end development of autonomy. Building assets from in-the-wild data is key for diversity and realism, but existing neural-rendering based reconstruction methods are slow and generate assets that render well only from viewpoints close to the original observations, limiting their usefulness in simulation. Recent diffusion-based generative models build complete and diverse assets, but perform poorly on in-the-wild driving scenes, where observed actors are captured under sparse and limited fields of view, and are partially occluded. In this work, we propose a 3D latent diffusion model that learns on in-the-wild LiDAR and camera data captured by a sensor platform and generates high-quality 3D assets with complete geometry and appearance. Key to our method is a "reconstruct-then-generate" approach that first leverages occlusion-aware neural rendering trained over multiple scenes to build a high-quality latent space for objects, and then trains a diffusion model that operates on the latent space. We show our method outperforms existing reconstruction and generation based methods, unlocking diverse and scalable content creation for simulation.