OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

Hebaixu Wang; Jing Zhang; Haonan Guo; Di Wang; Jiayi Ma; Bo Du

cs.CVOct 11, 2026

OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

Authors: Hebaixu Wang, Jing Zhang, Haonan Guo, Di Wang, Jiayi Ma, Bo Du

Organizations: Wuhan University & Zhongguancun Academy

Abstract

Multimodal remote sensing image generation emerges as a promising paradigm for recovering complete and complementary observations from heterogeneous and partially observed remote sensing data. However, existing methods predominantly depend on sensor-specific fine-tuning of natural image generation models, limiting unified multimodal generation capacity due to the absence of shared representations and robust conditioning mechanisms. To address these challenges, we propose OmniRSI, a unified multimodal remote sensing image generation model. Specifically, we develop a remote sensing-specific variational autoencoder that captures multimodal priors and promotes a locally smooth latent space for unified representation and robust reconstruction across diverse observed data. Building upon that, we fully adapt Stable Diffusion 3.5 with a ControlNet branch to support unified multimodal generation and translation conditioned on textual and visual cues. Besides, we devise a training-free cyclic latent alignment mechanism that iteratively enriches incomplete visual conditions with generated multimodal information, progressively improving guidance reliability while preserving observed measurements. We also construct RSI-MM, a large-scale dataset comprising 11.5M multi-resolution images with text annotations and partially paired data for training and evaluation. Extensive experiments verify that OmniRSI achieves superior performance in multimodal reconstruction, generation and translation.

Figures & tables

Figure 1: Overview and limitations of latent diffusion-based remote sensing image generation.

		Modality
Datasets	Source&Platform	RGB	MSI	SAR	PAN	Text	Text Num.	Spat. Res.(m)	Band. Configs.	Image Num.	Image Size
Git-10M	GoogleEarth&Public datasets	✓	–	–	–	✓	10,452,649	0.5 $\sim$ 128	3	10,452,649	256 $\times$ 256
EuroSAT	Sentinel-2	✓	✓	–	–	❍	27,000	10/20/60	13	27,000	64 $\times$ 64
OSCD	Sentinel-2	✓	✓	–	–	❍	48	10/20/60	13	48	241 $\sim$ 1180
M3M-CR	PlanetScope&Sentinel-1	✓	✓	✓	–	❍	63,000	3/10	5	63,000	300/90
OpenEarthMap-SAR	UmbraSpace&NAIP&IGN&GSI	✓	–	✓	–	❍	5,033	0.15-0.5	4	5,033	1024 $\times$ 1024

Table 1: Details of RSI-MM datasets. ✓: available; –: unavailable; ❍: supplemented by us.

Figure 2: Overview of OmniRSI. Multimodal observations are slot-organized and encoded into a unified latent space. SD3.5 backbone is fine-tuned for generation, with a ControlNet branch injecting hierarchical condition guidance. The derived latents are decoded for multimodal reconstruction.

Method	PAN		OPT		SAR		NIR		MSI		Average		Complexity
Method	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	Params(M)	FLOPs(G)
Any2any	37.68	0.838	39.06	0.951	26.85	0.846	29.99	0.951	21.78	0.796	32.69	0.890	13.322 $\sim$ 83.610	329.120 $\sim$ 473.663
Ours	42.79	0.972	39.71	0.958	27.15	0.878	32.83	0.953	26.91	0.869	34.95	0.935	83.815	447.742

Table 2: Quantitative results of our RS-VAE and modality-specific VAEs in Any2any. FLOPS are measured at 256

\times

256 resolution. Bold ranks the best and underline ranks the second best.

Table 7

Figure 6: Qualitative results of image-to-image generation. Displayed in RGB composite.

Method			Step	NFE	PAN $\rightarrowtail$ MSI		MSI $\rightarrowtail$ PAN		SAR $\rightarrowtail$ MSI		MSI $\rightarrowtail$ SAR		Average
Solver	CLA	Text	Step	NFE	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
Euler	✕	✕	40	40	21.73	0.660	34.77	0.910	17.09	0.365	19.54	0.278	25.21	0.632
Euler	✓	✕	40	80	22.07	0.662	35.17	0.914	17.36	0.373	20.32	0.291	25.63	0.639
Euler	✕	✓	40	40	24.30	0.768	35.48	0.919	18.11	0.375	19.66	0.283	26.43	0.671
Euler	✓	✓	40	80	25.16	0.786	36.10	0.925	18.39	0.382	20.50	0.302	27.10	0.683
Euler	✕	✕	80	80	21.78	0.653	35.09	0.907	17.19	0.366	20.27	0.285	25.48	0.631

Table 8: Effect of CLA on different solvers for multimodal image translation tasks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Text annotation pipeline for RSI-MM. Representative visualizations are selected as inputs, followed by GPT-based caption generation with a fixed prompt template.

Slot	Name	Wavelength Range (nm)	Slot Wavelength Center (nm)	Sentinel-2A/B	Landsat 8/9	WorldView2/3	Landsat 7	GaoFen/GeoEye Series
0	PAN	Sensor-dependent	—	—	B8	PAN	B8	PAN
1	Coastal	400–454	427	B1	B1	Coastal	—	—
2	Blue	450–526	488	B2	B2	Blue	B1	Blue
3	Green	510–600	555	B3	B3	Green	B2	Green
4	Yellow	585-–625	605	—	—	Yellow	—	—
5	Red	630–690	660	B4	B4	Red	B3	Red

Appendix

Table A1: Canonical slots and their sensor-specific band mappings. Wavelength ranges and centers are slot reference values. Bands assigned to the same slot need not have identical spectral responses.

Configuration	Value
SD3.5 Medium backbone
MMDiT blocks	24
Latent input/output channels	16 / 16
Latent patch size	$2\times 2$
Hidden dimension	1,536
Attention heads / head dimension	24 / 64

Appendix

Table A2: Architecture configuration of the SD3.5 Medium backbone and ControlNet branch.

Figure A2: Qualitative results of text-to-image Generation.

Figure A3: Qualitative results of text-to-image Generation.

Figure A4: Zero-shot translation from PAN to SAR (top) and from SAR to PAN (bottom).

Valid Band	1	2	3	4	5	6	7	8	9	10	11	12	13
Self-Rec PSNR	32.66	33.45	34.61	35.61	35.86	37.95	38.95	38.97	39.02	41.44	43.01	43.95	45.58
Self-Rec SSIM	0.930	0.940	0.954	0.959	0.966	0.969	0.974	0.977	0.979	0.986	0.988	0.990	0.991
Full-Rec PSNR	29.75	32.59	33.42	33.45	33.58	36.04	36.82	37.03	37.20	41.15	42.68	43.78	45.58
Full-Rec SSIM	0.904	0.929	0.935	0.936	0.936	0.961	0.968	0.972	0.975	0.987	0.989	0.990	0.991

Appendix

Table A3: Reconstruction performance of RS-VAE with varying numbers of bands on EuroSAT (50 samples). Self-Rec evaluates the remaining observed bands, whereas Full-Rec evaluates all 13 original bands. Higher PSNR (dB) and SSIM indicate better reconstruction.

Valid Band	1	2	3	4	5	6	7	8	9	10
PSNR	26.22	30.60	33.32	33.76	34.06	37.36	37.63	39.18	40.92	41.67
SSIM	0.671	0.929	0.937	0.946	0.951	0.965	0.967	0.971	0.974	0.975

Appendix

Table A4: PAN translation performance with varying numbers of observed MSI bands on PSBench (50 samples, MSI

\rightarrow

PAN). Bands are progressively masked in a fixed order.

Resolution	256 $\times$ 256			512 $\times$ 512
Method	Mem.(G)	Time(s)	FPS	Mem.(G)	Time(s)	FPS
VAE Reconstruction
Any2any(MSI)	2.683	0.027	37.037	40.530	0.279	3.579
RS-VAE(Ours)	0.359	0.018	54.969	0.952	0.066	15.110
Text-to-image Generation
CC-Diff	7.961	2.974	0.336	9.832	5.805	0.172

Appendix

Table A5: Efficiency comparisons among different methods.

Explore similar work

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing
Oct 13, 2026Pan Yin. Shuai Yang, Yuanhao Liu, Datao Tang +6Image Generation Unified Multimodal Models
Xi’an Jiaotong University
Contrastive Parameter Disentanglement for Multi-modal Remote Sensing Image Generation
Jul 26, 2026Yu Zhang, Wenda Zhao, Haojun Tang +1T2I Generation Image Generation
Unit 92728 of PLA,Shanghai, 200436, China
MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling
May 19, 2026Zhiping Yu, Chenyang Liu, Jinqi Cao +4Cross-Modal Learning Unified Multimodal Models
Department of Aerospace Intelligent Science and Tech-nology, School of Astronautics, Beihang University, Beijing 100191, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China · Shenyuan Honors College, Beihang University, Beijing 100191, China

Oct 13, 2026J/K move · Enter open · S save

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

Pan Yin. Shuai Yang, Yuanhao Liu, Datao Tang, Jialin Li+5

Xi’an Jiaotong University

Image Generation Unified Multimodal Models Remote Sensing Image Understanding Multi-Task Learning Diffusion-Based Image Generation

Open paper AskPDF arXiv

Practical remote sensing applications often require both understanding and generation capabilities for the same geographic scene. However, most existing methods address either understanding or generation in isolation, while existing unified remote sensing models' generative ability is limited to text-to-image. To bridge this gap, we propose Omni-RS, a unified framework supporting four remote sensing understanding tasks and five generation tasks. Specifically, we introduce the "Scene-Guided Two-Stage Training" strategy, which first adapts a multimodal large language model to remote sensing understanding and then uses its hidden states to guide a diffusion transformer for image generation. To enable joint learning across diverse generation tasks, we construct UniGen, a unified multi-task generation dataset, by organizing heterogeneous remote sensing data into a unified format. Furthermore, we introduce "Detailed Visual Conditioning", which uses visual conditions encoded by a VAE to guide the generation process, thereby supporting additional generation tasks that require reference images. Extensive experiments demonstrate that Omni-RS performs exceptionally well in understanding tasks and has achieved outstanding results in a variety of generative tasks.

Abstract

Explore similar work

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

Contrastive Parameter Disentanglement for Multi-modal Remote Sensing Image Generation

MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling