Text-Image Conditioned 3D Generation

Jiazhong Cen; Jiemin Fang; Sikuang Li; Guanjun Wu; Chen Yang; Taoran Yi; Zanwei Zhou; Zhikuan Bao

cs.CVMar 22, 2026

Text-Image Conditioned 3D Generation

Authors: Jiazhong Cen, Jiemin Fang, Sikuang Li, Guanjun Wu, Chen Yang, Taoran Yi, Zanwei Zhou, Zhikuan Bao,

Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Huawei Inc. · Huazhong University of Science and Technology

Abstract

High-quality 3D assets are essential for VR/AR, industrial design, and entertainment, motivating growing interest in generative models that create 3D content from user prompts. Most existing 3D generators, however, rely on a single conditioning modality: image-conditioned models achieve high visual fidelity by exploiting pixel-aligned cues but suffer from viewpoint bias when the input view is limited or ambiguous, while text-conditioned models provide broad semantic guidance yet lack low-level visual detail. This limits how users can express intent and raises a natural question: can these two modalities be combined for more flexible and faithful 3D generation? Our diagnostic study shows that even simple late fusion of text- and image-conditioned predictions outperforms single-modality models, revealing strong cross-modal complementarity. We therefore formalize Text-Image Conditioned 3D Generation, which requires joint reasoning over a visual exemplar and a textual specification. To address this task, we introduce TIGON, a minimalist dual-branch baseline with separate image- and text-conditioned backbones and lightweight cross-modal fusion. Extensive experiments show that text-image conditioning consistently improves over single-modality methods, highlighting complementary vision-language guidance as a promising direction for future 3D generation research. Project page: https://jumpat.github.io/tigon-page

Figures & tables

Figure 1: Single-modality conditioning has limitations in satisfying user intent. Image-only conditioning captures local appearance but omits unobserved regions; text-only conveys semantics but lacks visual fidelity. In contrast, joint text–image conditioning produces 3D assets that are both semantically aligned with the description and faithful to the reference appearance.

Figure 2: Reference views used in our diagnostic study. Moving from View-0 to View-1 reduces observable cues and creates a lower-information setting. Under this shift, single-modality baselines exhibit a marked performance drop.

Model	Cond.	CLIP $\uparrow$	FD ${}_{\text{DINOv2}}\downarrow$
TripoSR	View-0	88.67	269.58
Step1X-3D †	View-0	89.99	152.69
Hunyuan3D-2.1 †	View-0	89.87	114.64
TRELLIS (GS)	View-0	92.88	56.08
UniLat3D (GS)	View-0	93.34	47.41
TripoSR	View-1	79.40	804.18

Table 1: Performance of existing methods on the Toys4K dataset under different conditioning signals. ‘GS’ denotes that the 3D representation is 3DGS.

Figure 3: TIGON employs a dual-branch architecture, with a text-conditioned DiT (left) and an image-conditioned DiT (right). Paired blocks exchange features via cross-modal bridges (“Zero Linears”). At each denoising step, two predictions are averaged to produce the velocity field $\mathbf{v}$ . $T$ denotes the denoising timestep.

Model	Cond.	Rep.	Toys4K				UniLat1K
Model	Cond.	Rep.	CLIP $\uparrow$	FD ${}_{\text{DINOv2}}\downarrow$	ULIP $\uparrow$	Uni3D $\uparrow$	CLIP $\uparrow$	FD ${}_{\text{DINOv2}}\downarrow$	ULIP $\uparrow$	Uni3D $\uparrow$
TripoSR [ 58 ]	I	M.	83.14	596.44	27.37	24.38	83.37	652.27	25.90	23.96
TRELLIS	I	M.	89.09	171.44	39.97	35.61	89.40	233.53	39.37	35.40
TRELLIS	I	GS	90.50	98.75	-	-	90.83	177.20	-	-
Step1X-3D † [ 31 ]	I	M.	84.77	361.44	34.15	30.04	85.36	402.25	33.62	30.38
Hunyuan3D-2.1 † [ 23 ]	I	M.	87.57	171.91	40.22	35.70	87.27	249.66	39.58	35.54

Table 2: Quantitative results on Toys4K (left) and UniLat1K (right). “Cond.” denotes conditioning modality (“I”: image, “T”: text), and “Rep.” denotes output representation (“M.”: mesh, “GS”: 3DGS).

Figure 4: Qualitative comparison on Toys4K and UniLat1K against image-only and text-only variants of TRELLIS and UniLat3D. Dashed boxes mark artifacts from prior methods. Image-only models respect the reference view but must hallucinate unseen regions, while text-only models lack pixel-aligned cues and often produce low-fidelity geometry and appearance. Full prompts are provided in the supplement.

Figure 5: Controllable generation under text and image conditions.

Figure 6: Generation with conflicting text-image conditions.

Bridges	Fusion Strategy			FT	CLIP $\uparrow$	FD ${}_{\text{DINOv2}}\downarrow$
Bridges	Sim	AW	AT	FT	CLIP $\uparrow$	FD ${}_{\text{DINOv2}}\downarrow$
	✓				91.95	66.78
	✓			✓	92.05	66.04
✓	✓			✓	92.33	61.59
✓		✓		✓	92.31	60.90
✓			✓	✓	92.26	62.00

Table 3: Ablations on Toys4K. “Bridges” denotes zero-initialized cross-modal bridges; “Sim”, “AW”, and “AT” denote three fusion strategies; “FT” denotes joint fine-tuning. The TIGON setting is highlighted in light gray.

Figure 7: Effect of early fusion. Without cross-modal bridges, the two branches diverge during denoising. Full text prompt is available in the supplement.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : The adaptive fusion module used in the ablation study. $D$ denotes the intermediate feature dimension, and $C$ denotes the latent output dimension.

Model	Cond.	CLIP	FD ${}_{\texttt{DINOv2}}$
TRELLIS	I	90.50	98.75
TRELLIS	T	86.30	148.21
TRELLIS (w/o ss-bridge)	I+T	91.23	80.35
TRELLIS (w/ ss-bridge)	I+T	91.51	75.35

Appendix

Table A1 : Integrating TIGON with TRELLIS. Experiment is conducted on Toys4K. We use 3D-GS as the representation. ‘ss-bridge’ denotes the cross-modal bridge for the sparse-structure flow model.

Figure A3 : Additional visualization results of controllable generation with TIGON under joint text and image conditioning.

3D Asset Generation

3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.

See all 163 papers

SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation
Apr 9, 2026Ming He, Zhixiang Chen, Steve MaddockText-to-3D Generation 3D Shape Generation
School of Computer Science, University of Sheffield
Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation
Apr 2, 2026Chongjie Ye, Cheng Cao, Chuanyu Pan +4Autoregressive 3D Scene Generation 3D Foundation Models
FNii-Shenzhen · SSE, CUHK(SZ) · Meshy AI
ReConText3D: Replay-based Continual Text-to-3D Generation
Apr 15, 2026Muhammad Ahmed Ullah Khan, Muhammad Haris Bin Amir, Didier Stricker +1Text-to-3D Generation 3D Asset Generation
DFKI · RPTU Kaiserslautern-Landau

Apr 9, 2026J/K move · Enter open · S save

SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation

Ming He, Zhixiang Chen, Steve Maddock

School of Computer Science, University of Sheffield

Text-to-3D Generation 3D Shape Generation 3D Gaussian Splatting

Open paper AskPDF arXiv

Recent progress in text-to-3D object generation enables the synthesis of detailed geometry from text input by leveraging 2D diffusion models and differentiable 3D representations. However, the approaches often suffer from limited controllability and texture ambiguity due to the limitation of the text modality. To address this, we present SIC3D, a controllable image-conditioned text-to-3D generation pipeline with 3D Gaussian Splatting (3DGS). There are two stages in SIC3D. The first stage generates the 3D object content from text with a text-to-3DGS generation model. The second stage transfers style from a reference image to the 3DGS. Within this stylization stage, we introduce a novel Variational Stylized Score Distillation (VSSD) loss to effectively capture both global and local texture patterns while mitigating conflicts between geometry and appearance. A scaling regularization is further applied to prevent the emergence of artifacts and preserve the pattern from the style image. Extensive experiments demonstrate that SIC3D enhances geometric fidelity and style adherence, outperforming prior approaches in both qualitative and quantitative evaluations.

Abstract

3D Asset Generation

Explore similar work

SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation

Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation

ReConText3D: Replay-based Continual Text-to-3D Generation