SciFusion: Physics-Guided Compositional Diffusion for Multimodal 3D Scientific Image Generation

Luoxi Wang; Alessio Quercia; Hanno Scharr; Arya Bangun

eess.IVOct 21, 2026

SciFusion: Physics-Guided Compositional Diffusion for Multimodal 3D Scientific Image Generation

Authors: Luoxi Wang, Alessio Quercia, Hanno Scharr, Arya Bangun

Organizations: Princeton University, Princeton, NJ, USA · Forschungszentrum Jülich, Jülich, Germany

Abstract

Existing 3D generative approaches are typically developed for individual imaging modalities, requiring substantial modality-specific data and limiting their ability to exploit shared structure across scientific imaging domains. We present SciFusion, a multimodal framework for unified 3D scientific image generation built on a latent diffusion backbone, with two key contributions. First, we introduce a compositional conditioning scheme that factorizes each modality into domain, object, and contrast representations. Sharing these factors across related modalities enables knowledge transfer across heterogeneous imaging domains and allows the factors to be recombined at inference time to synthesize unseen domain-object-contrast compositions in a zero-shot manner. Second, we incorporate domain-specific physics-informed objectives: Fourier spectral consistency for MRI, Radon projection consistency for CT, and multislice diffraction consistency for electron microscopy, to encourage generated samples to respect modality-dependent image formation processes. Extensive experiments across 15 medical and scientific imaging modalities demonstrate that SciFusion improves generation fidelity and cross-domain generalization over modality-specific and flattened multimodal conditioning strategies, while enabling zero-shot synthesis of unseen compositions without training on those target compositions. These results demonstrate that compositional, physics-guided conditioning provides a scalable framework for unified 3D scientific image generation, particularly for emerging modalities where large-scale training data are difficult to obtain.

Figures & tables

Figure 1: Overview of SciFusion. (a) Two-stage training with shared latent representation learning, compositional conditioning, and domain-specific physics guidance. (b) Inference via recombination of learned conditioning components, enabling both seen-modality generation and zero-shot synthesis of unseen modality combinations.

Backbone-specific baseline
Backbone	Conditioning	VMAE		I3D		JEDI $\downarrow$
		FVD $\downarrow$	KID $(\times 10^{3})$ $\downarrow$	FVD $\downarrow$	KID $(\times 10^{3})$ $\downarrow$
MedLoRD [ 29 ]	Flattened	28.63	106.80	11.80	21.08	2.356
Video Diffusion [ 16 ]	Original	56.87	184.97	24.00	49.28	4.63
3DMedDiff [ 37 ]	Original	91.27	373.75	33.40	78.48	8.341
Medical Diffusion [ 21 ]	Original	34.55	130.12	13.19	24.08	2.732

Table 1: Generalization of compositional conditioning across different diffusion architectures. Each backbone is evaluated using its original conditioning strategy and the proposed compositional conditioning. All results are macro-averaged over the 15 imaging modalities.

Figure 2: Selected generated 3D volumes across all selected modalities.

Target Modality	Backbone	Supervision	I3D		VMAE		JEDI $\downarrow$
			FVD $\downarrow$	KID $(\times 10^{3})$ $\downarrow$	FVD $\downarrow$	KID $(\times 10^{3})$ $\downarrow$
Knee T1	SciFusion (Ours)	Full	13.83	23.13 $\pm$ 2.00	14.56	58.71 $\pm$ 3.78	0.35
	SciFusion (Ours)	Zero-shot	26.14	50.23 $\pm$ 2.39	58.32	252.47 $\pm$ 7.07	2.88
	Medical Diffusion [ 21 ]	Zero-shot	28.46	54.50 $\pm$ 2.65	72.46	322.46 $\pm$ 7.36	3.59
	VDM + LoRA [ 16 , 17 ]	PEFT	15.99	23.06 $\pm$ 1.39	25.63	102.29 $\pm$ 6.65	1.05
Brain PD	SciFusion (Ours)	Full	2.90	4.30 $\pm$ 0.49	5.22	19.19 $\pm$ 1.25	0.46

Table 2: Zero-shot compositional generation on unseen target modalities compared with fully supervised target-modality training and parameter-efficient adaptation.

Figure 3: Qualitative comparison of generation on unseen target modalities (knee T1 and brain PD). (a) Real target-modality images, (b) our zero-shot generation without access to target-modality data or additional training, and (c) parameter-efficient fine-tuning (PEFT) using VDM with LoRA and target-modality samples.

Method	# Models	Comp.	Physics	I3D-FVD $\downarrow$	VMAE-FVD $\downarrow$	JEDI $\downarrow$
Single Modality	15	–	–	13.07	35.93	2.62
Flat Multimodal	1	–	–	11.80	28.63	2.36
Comp. Multimodal	1	✓	–	11.10	27.30	2.35
Ours	1	✓	✓	9.90	23.51	1.80

Table 3: Ablation study of SciFusion across four configurations: single-modality models, flattened multimodal conditioning, compositional conditioning, and the full model with physics guidance.

Domain	$\boldsymbol{\lambda_{\mathrm{phy}}}$	I3D		VMAE		JEDI $\downarrow$
		FVD $\downarrow$	KID $\downarrow$	FVD $\downarrow$	KID $\downarrow$
MRI	$5\cdot 10^{-4}$	10.18	18.04	24.70	95.35	1.46
	$4\cdot 10^{-4}$	11.59	22.26	28.69	114.18	1.64
	$2\cdot 10^{-4}$	11.46	21.40	28.95	115.29	1.64
EM	$5\cdot 10^{-4}$	5.42	6.28	16.85	46.68	1.50
	$4\cdot 10^{-4}$	5.04	5.59	15.15	38.46	2.13

Table 4: Effect of physical-loss regularization strength across different imaging domains. KID is multiplied by

10^{3}

for readability.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Anatomy	Contrast	Total
fastMRI	Knee	PD and PDFS	(480, 486)
fastMRI	Brain	T1,T2, and FLAIR	(328, 1367, 2547)
fastMRI	Prostate	T2 and DWI	(272, 261)
fastMRI	Breast	T1	609
IXI	Brain	PD	566
MRNet	Knee	T1	1130

Appendix

Table 5: Summary of MRI datasets

Dataset	Anatomy	Total
RSNA	Brain	339 exams (subset)
MSD	Liver	201 scans
MSD	Lung	96 scans
MSD	Pancreas	420 scans
MSD	Colon	190 scans
MSD	Liver	443 scans

Appendix

Table 6: Summary of the CT datasets

Parameter	Crystal data	Protein data
Pixel Dimension	$256\times 256$	$256\times 256$
Sampling / pixel size	$0.1~\text{\AA}$	$1.0~\text{\AA}$
Number of slices $(S)$	$16$	$16$
Slice thickness	$0.5~\text{\AA}$	$4.0~\text{\AA}$
Potential	Lobato [ 24 ]	Lobato [ 24 ]
Incident wave	Plane wave	Plane wave

Appendix

Table 7: abTEM simulation parameters used for dataset generation.

Figure 4: Representative generated 3D volumes across all imaging modalities. Each row corresponds to one modality, and the 16 columns visualize axial slices from a single generated volume.

Backbone-specific baseline conditioning
Backbone	Domain	VideoMAE-FD $\downarrow$	VideoMAE-KID $\downarrow$	I3D-FD $\downarrow$	I3D-KID $\downarrow$	JEDI $\downarrow$
			$\times 10^{3}$		$\times 10^{3}$
MedLord	MRI	27.33	109.08	11.21	20.10	1.668
	EM	17.38	43.82	3.93	3.13	2.522
	CT	47.13	184.68	23.85	47.63	3.969
Pixel Diffusion	MRI	45.84	141.80	29.28	63.99	2.042

Appendix

Table 8: Domain-level comparison between backbone-specific baseline conditioning and the proposed compositional conditioning across three imaging domains. Results are macro-averaged over modalities within each domain, including eight MRI modalities, four EM modalities, and three CT modalities. For MedLord, the backbone-specific baseline uses an independently learned embedding for each modality, whereas compositional conditioning uses the proposed domain–object–contrast representation. Other backbones retain their original conditioning mechanisms for the baseline setting.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

3D Diffusion Models for Medical Imaging
Oct 19, 2026Frank-Peter Schilling, Daniel Barco, Philipp Denzel +13D Diffusion Models Medical Image Synthesis
Centre for AI (CAI), Zurich University of Applied Sciences (ZHAW), Technikumstrasse 9, Winterthur, 8400, Zurich, Switzerland
AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation
Jun 30, 2026Eric Ji, Qiran Hu, Wufei Ma +4Synthetic Data Generation Diffusion-Based Image Generation
University of Illinois Urbana-Champaign · Johns Hopkins University · College of Engineering and Computer Science, VinUniversity
Generative Drifting for Conditional Medical Image Generation
Apr 21, 2026Zirong Li, Siyuan Mei, Weiwen Wu +33D Medical Imaging One-Step Generative Modeling
Department of Orthodontics and Orofacial Orthopedics, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany · Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany · Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany +1

Oct 19, 2026J/K move · Enter open · S save

3D Diffusion Models for Medical Imaging

Frank-Peter Schilling, Daniel Barco, Philipp Denzel, Marc Stadelmann

Centre for AI (CAI), Zurich University of Applied Sciences (ZHAW), Technikumstrasse 9, Winterthur, 8400, Zurich, Switzerland

3D Diffusion Models Medical Image Synthesis 3D Medical Imaging

Open paper AskPDF arXiv

Generative deep learning methods such as Variational Autoencoders, Generative Adversarial Networks or Diffusion Models have achieved remarkable results in image synthesis. In particular, Diffusion Models have demonstrated superior image fidelity and sample diversity compared to other approaches on standard benchmarks, while also offering more flexible training dynamics as evidenced both in natural image domains and in medical applications. In medical imaging including Computed Tomography, Magnetic Resonance Imaging or Positron Emission Tomography, such models have various important use-cases including data augmentation, domain transfer, image reconstruction, inpainting, image augmentation and artifact correction. Further, generated images can be conditioned on patient characteristics, medical reports, segmentation masks or other images. Specific challenges arise due to the 3D volumetric nature of medical images, which translate to substantial demands in terms of memory, compute, and training data. This article provides a comparative overview of 3D Diffusion Models which have been developed for medical images, with a focus on current performance, limitations, and an outlook for future work.

Abstract

Image Generation

Explore similar work

3D Diffusion Models for Medical Imaging

AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation

Generative Drifting for Conditional Medical Image Generation