Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Huawei Inc. · Huazhong University of Science and Technology
High-quality 3D assets are essential for VR/AR, industrial design, and entertainment, motivating growing interest in generative models that create 3D content from user prompts. Most existing 3D generators, however, rely on a single conditioning modality: image-conditioned models achieve high visual fidelity by exploiting pixel-aligned cues but suffer from viewpoint bias when the input view is limited or ambiguous, while text-conditioned models provide broad semantic guidance yet lack low-level visual detail. This limits how users can express intent and raises a natural question: can these two modalities be combined for more flexible and faithful 3D generation? Our diagnostic study shows that even simple late fusion of text- and image-conditioned predictions outperforms single-modality models, revealing strong cross-modal complementarity. We therefore formalize Text-Image Conditioned 3D Generation, which requires joint reasoning over a visual exemplar and a textual specification. To address this task, we introduce TIGON, a minimalist dual-branch baseline with separate image- and text-conditioned backbones and lightweight cross-modal fusion. Extensive experiments show that text-image conditioning consistently improves over single-modality methods, highlighting complementary vision-language guidance as a promising direction for future 3D generation research. Project page: https://jumpat.github.io/tigon-page
Figures & tables
Figure 1: Single-modality conditioning has limitations in satisfying user intent. Image-only conditioning captures local appearance but omits unobserved regions; text-only conveys semantics but lacks visual fidelity. In contrast, joint text–image conditioning produces 3D assets that are both semantically aligned with the description and faithful to the reference appearance.
Figure 2: Reference views used in our diagnostic study. Moving from View-0 to View-1 reduces observable cues and creates a lower-information setting. Under this shift, single-modality baselines exhibit a marked performance drop.
Model
Cond.
CLIP ↑
FD DINOv2↓
TripoSR
View-0
88.67
269.58
Step1X-3D †
View-0
89.99
152.69
Hunyuan3D-2.1 †
View-0
89.87
114.64
TRELLIS (GS)
View-0
92.88
56.08
UniLat3D (GS)
View-0
93.34
47.41
TripoSR
View-1
79.40
804.18
Table 1: Performance of existing methods on the Toys4K dataset under different conditioning signals. ‘GS’ denotes that the 3D representation is 3DGS.
Figure 3: TIGON employs a dual-branch architecture, with a text-conditioned DiT (left) and an image-conditioned DiT (right). Paired blocks exchange features via cross-modal bridges (“Zero Linears”). At each denoising step, two predictions are averaged to produce the velocity field v . T denotes the denoising timestep.
Model
Cond.
Rep.
Toys4K
UniLat1K
CLIP ↑
FD DINOv2↓
ULIP ↑
Uni3D ↑
CLIP ↑
FD DINOv2↓
ULIP ↑
Uni3D ↑
TripoSR [ 58 ]
I
M.
83.14
596.44
27.37
24.38
83.37
652.27
25.90
23.96
TRELLIS
I
M.
89.09
171.44
39.97
35.61
89.40
233.53
39.37
35.40
TRELLIS
I
GS
90.50
98.75
-
-
90.83
177.20
-
-
Step1X-3D † [ 31 ]
I
M.
84.77
361.44
34.15
30.04
85.36
402.25
33.62
30.38
Hunyuan3D-2.1 † [ 23 ]
I
M.
87.57
171.91
40.22
35.70
87.27
249.66
39.58
35.54
Table 2: Quantitative results on Toys4K (left) and UniLat1K (right). “Cond.” denotes conditioning modality (“I”: image, “T”: text), and “Rep.” denotes output representation (“M.”: mesh, “GS”: 3DGS).
Figure 4: Qualitative comparison on Toys4K and UniLat1K against image-only and text-only variants of TRELLIS and UniLat3D. Dashed boxes mark artifacts from prior methods. Image-only models respect the reference view but must hallucinate unseen regions, while text-only models lack pixel-aligned cues and often produce low-fidelity geometry and appearance. Full prompts are provided in the supplement.
Figure 5: Controllable generation under text and image conditions.
Figure 6: Generation with conflicting text-image conditions.
Bridges
Fusion Strategy
FT
CLIP ↑
FD DINOv2↓
Sim
AW
AT
✓
91.95
66.78
✓
✓
92.05
66.04
✓
✓
✓
92.33
61.59
✓
✓
✓
92.31
60.90
✓
✓
✓
92.26
62.00
Table 3: Ablations on Toys4K. “Bridges” denotes zero-initialized cross-modal bridges; “Sim”, “AW”, and “AT” denote three fusion strategies; “FT” denotes joint fine-tuning. The TIGON setting is highlighted in light gray.
Figure 7: Effect of early fusion. Without cross-modal bridges, the two branches diverge during denoising. Full text prompt is available in the supplement.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : The adaptive fusion module used in the ablation study. D denotes the intermediate feature dimension, and C denotes the latent output dimension.
Model
Cond.
CLIP
FD DINOv2
TRELLIS
I
90.50
98.75
TRELLIS
T
86.30
148.21
TRELLIS (w/o ss-bridge)
I+T
91.23
80.35
TRELLIS (w/ ss-bridge)
I+T
91.51
75.35
Appendix
Table A1 : Integrating TIGON with TRELLIS. Experiment is conducted on Toys4K. We use 3D-GS as the representation. ‘ss-bridge’ denotes the cross-modal bridge for the sparse-structure flow model.
Figure A2 : Meshes generated by Hunyuan3D-2.1, Step1X-3D, and TIGON. For the full text prompts, please refer to Appendix G .
Figure A3 : Additional visualization results of controllable generation with TIGON under joint text and image conditioning.
Recent progress in text-to-3D object generation enables the synthesis of detailed geometry from text input by leveraging 2D diffusion models and differentiable 3D representations. However, the approaches often suffer from limited controllability and texture ambiguity due to the limitation of the text modality. To address this, we present SIC3D, a controllable image-conditioned text-to-3D generation pipeline with 3D Gaussian Splatting (3DGS). There are two stages in SIC3D. The first stage generates the 3D object content from text with a text-to-3DGS generation model. The second stage transfers style from a reference image to the 3DGS. Within this stylization stage, we introduce a novel Variational Stylized Score Distillation (VSSD) loss to effectively capture both global and local texture patterns while mitigating conflicts between geometry and appearance. A scaling regularization is further applied to prevent the emergence of artifacts and preserve the pattern from the style image. Extensive experiments demonstrate that SIC3D enhances geometric fidelity and style adherence, outperforming prior approaches in both qualitative and quantitative evaluations.
Ming He, Zhixiang Chen, Steve Maddock
School of Computer Science, University of Sheffield
Recent multimodal large language models have achieved strong performance in unified text and image understanding and generation, yet extending such native capability to 3D remains challenging due to limited data. Compared to abundant 2D imagery, high-quality 3D assets are scarce, making 3D synthesis under-constrained. Existing methods often rely on indirect pipelines that edit in 2D and lift results into 3D via optimization, sacrificing geometric consistency. We present Omni123, a 3D-native foundation model that unifies text-to-2D and text-to-3D generation within a single autoregressive framework. Our key insight is that cross-modal consistency between images and 3D can serve as an implicit structural constraint. By representing text, images, and 3D as discrete tokens in a shared sequence space, the model leverages abundant 2D data as a geometric prior to improve 3D representations. We introduce an interleaved X-to-X training paradigm that coordinates diverse cross-modal tasks over heterogeneous paired datasets without requiring fully aligned text-image-3D triplets. By traversing semantic-visual-geometric cycles (e.g., text to image to 3D to image) within autoregressive sequences, the model jointly enforces semantic alignment, appearance fidelity, and multi-view geometric consistency. Experiments show that Omni123 significantly improves text-guided 3D generation and editing, demonstrating a scalable path toward multimodal 3D world models.
Continual learning enables models to acquire new knowledge over time while retaining previously learned capabilities. However, its application to text-to-3D generation remains unexplored. We present ReConText3D, the first framework for continual text-to-3D generation. We first demonstrate that existing text-to-3D models suffer from catastrophic forgetting under incremental training. ReConText3D enables generative models to incrementally learn new 3D categories from textual descriptions while preserving the ability to synthesize previously seen assets. Our method constructs a compact and diverse replay memory through text-embedding k-Center selection, allowing representative rehearsal of prior knowledge without modifying the underlying architecture. To systematically evaluate continual text-to-3D learning, we introduce Toys4K-CL, a benchmark derived from the Toys4K dataset that provides balanced and semantically diverse class-incremental splits. Extensive experiments on the Toys4K-CL benchmark show that ReConText3D consistently outperforms all baselines across different generative backbones, maintaining high-quality generation for both old and new classes. To the best of our knowledge, this work establishes the first continual learning framework and benchmark for text-to-3D generation, opening a new direction for incremental 3D generative modeling. Project page is available at: https://mauk95.github.io/ReConText3D/.
Muhammad Ahmed Ullah Khan, Muhammad Haris Bin Amir, Didier Stricker +1