Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

Pan Yin. Shuai Yang; Yuanhao Liu; Datao Tang; Jialin Li; Hao Wang; Pei Liu; Kaiyu Li; Cong Zhao

cs.CVOct 13, 2026

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

Authors: Pan Yin. Shuai Yang, Yuanhao Liu, Datao Tang, Jialin Li, Hao Wang, Pei Liu, Kaiyu Li, Cong Zhao, +1 more

Organizations: Xi’an Jiaotong University

Abstract

Practical remote sensing applications often require both understanding and generation capabilities for the same geographic scene. However, most existing methods address either understanding or generation in isolation, while existing unified remote sensing models' generative ability is limited to text-to-image. To bridge this gap, we propose Omni-RS, a unified framework supporting four remote sensing understanding tasks and five generation tasks. Specifically, we introduce the "Scene-Guided Two-Stage Training" strategy, which first adapts a multimodal large language model to remote sensing understanding and then uses its hidden states to guide a diffusion transformer for image generation. To enable joint learning across diverse generation tasks, we construct UniGen, a unified multi-task generation dataset, by organizing heterogeneous remote sensing data into a unified format. Furthermore, we introduce "Detailed Visual Conditioning", which uses visual conditions encoded by a VAE to guide the generation process, thereby supporting additional generation tasks that require reference images. Extensive experiments demonstrate that Omni-RS performs exceptionally well in understanding tasks and has achieved outstanding results in a variety of generative tasks.

Figures & tables

Figure 1: Comparison of task capabilities across different remote sensing model paradigms. (a) Understanding-based methods lack image generation capabilities. (b) Generation-based methods support individual generation tasks but lack understanding and multi-task generation. (c) Uni-RS bridges understanding and generation, but remains restricted to text-to-image synthesis. Right: Omni-RS supports four understanding tasks and five generation tasks within a unified framework.

Figure 2: Overview of UniGen’s construction pipeline. Heterogeneous remote sensing data are reorganized into five generation tasks, with Qwen-VL-Max providing captions and task instructions. The dashed arrow indicates the additional segmentation prompts supplied to SAM3 for inpainting.

Figure 3: Overview of Omni-RS. Stage I adapts the MLLM to remote sensing understanding through LoRA tuning with a cross-entropy loss. Stage II freezes the adapted MLLM and trains the diffusion transformer through LoRA tuning with a flow-matching loss. Semantic tokens from the MLLM and visual tokens from “Detailed Visual Conditioning” jointly guide image generation.

Figure 4: Qualitative results of Omni-RS on remote sensing understanding tasks.

Method	Presence $\uparrow$	Comparison $\uparrow$	Rural/Urban $\uparrow$	Ave. $\uparrow$
RSVQA ( Lobry et al., 2020 )	87.46	81.50	90.00	84.64
EasyToHard ( Yuan et al., 2022a )	90.66	87.49	91.67	88.89
VMGN ( Liu et al., 2026a )	91.30	92.52	94.00	92.03
RSUniVLM ( Liu and Lian, 2026 )	91.51	92.65	92.00	92.16
ScoreRS † ( Muhtar et al., 2026 )	89.79	89.68	91.36	89.75
UniRS † ( Li et al., 2024b )	91.64	92.68	90.00	92.21

Table 1: Comparison of Omni-RS with other methods on the RSVQA-LR dataset. The best results are shown in bold, and the second-best result is underlined.

\dagger

denotes zero-shot evaluation.

Table 6

Table 7

	RGB $\rightarrow$ NIR			RGB $\rightarrow$ SAR
Method	MAE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	PSNR $\uparrow$
pix2pix ( Isola et al., 2017 )	0.64	0.126	5.27	0.52	0.67	3.47
Text2Earth ( Liu et al., 2025 )	0.19	0.166	12.96	0.28	0.31	10.23
RSEdit ( Chen et al., 2026 )	0.30	0.097	9.24	0.29	0.36	8.99
Ours	0.09	0.655	18.68	0.17	0.23	12.89

Table 6: Comparison of cross-modal remote sensing image translation methods.

Figure 5: Qualitative results of Omni-RS on remote sensing generation tasks.

	Inpainting			Outpainting
Method	FID $\downarrow$	KID $\downarrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	LPIPS $\downarrow$
PowerPaint ( Zhuang et al., 2024 )	129.98	0.0614	0.46	67.84	0.0236	0.24
SD2 ( Rombach et al., 2022 )	47.98	0.0092	0.24	70.33	0.0224	0.32
RSEdit ( Chen et al., 2026 )	87.75	0.0304	0.53	81.22	0.0310	0.60
Text2Earth ( Liu et al., 2025 )	63.67	0.0192	0.25	66.65	0.0180	0.27
Ours	31.23	0.0025	0.12	35.33	0.0018	0.22

Table 7: Comparison of image inpainting and outpainting methods.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Generation task	Source dataset	Images / pairs	Train	Test
Text-to-image	Git-10M	2,669,400	2,668,400	1,000
Outpainting	Git-10M	2,669,400	2,668,400	1,000
Inpainting	Git-10M	199,794	198,795	999
Controllable generation	Git-10M, DIOR-R	223,257	210,520	12,737
RGB-to-NIR	WHU-OPT-SAR	144,528	144,028	500
RGB-to-SAR	WHU-OPT-SAR	127,600	127,100	500

Appendix

Table 8: Composition of UniGen by source data.

Figure 6: Inpainting results of Omni-RS . For each sample, distinct land-cover categories in the same geographic scene are independently masked and paired with specialized instructions.

Figure 7: Additional qualitative results of Omni-RS across five remote sensing generation tasks. We compare Omni-RS with the applicable baseline methods for text-to-image generation, outpainting, inpainting, and controllable generation. The input and ground-truth image are provided where applicable. Rows (l)–(o) present cross-modal translation results alongside their ground-truth targets.

Figure 8: Qualitative results of continuous large-scale outpainting by Omni-RS. Starting from an initial local satellite patch (top-left), the model iteratively expands the geographic canvas along both horizontal and vertical axes guided by outpainting instructions. Omni-RS seamlessly synthesizes an ultra-wide panoramic satellite image ( $2560\times 1024$ ).

Explore similar work

OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation
Oct 11, 2026Hebaixu Wang, Jing Zhang, Haonan Guo +3Multimodal Generation Unified Multimodal Models
Wuhan University & Zhongguancun Academy
MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling
May 19, 2026Zhiping Yu, Chenyang Liu, Jinqi Cao +4Cross-Modal Learning Unified Multimodal Models
Department of Aerospace Intelligent Science and Tech-nology, School of Astronautics, Beihang University, Beijing 100191, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China · Shenyuan Honors College, Beihang University, Beijing 100191, China
Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing
Jan 25, 2026Weiyu Zhang, Yuan Hu, Yong Li +1Unified Multimodal Models Multimodal Generation
Institute of Remote Sensing and Geographic Information System, School of Earth and Space Sciences, Peking University, Beijing, 100871, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong

Oct 11, 2026J/K move · Enter open · S save

OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

Hebaixu Wang, Jing Zhang, Haonan Guo, Di Wang+2

Wuhan University & Zhongguancun Academy

Multimodal Generation Unified Multimodal Models Image Reconstruction Latent Diffusion Models Image-to-Image Translation

Open paper AskPDF arXiv

Multimodal remote sensing image generation emerges as a promising paradigm for recovering complete and complementary observations from heterogeneous and partially observed remote sensing data. However, existing methods predominantly depend on sensor-specific fine-tuning of natural image generation models, limiting unified multimodal generation capacity due to the absence of shared representations and robust conditioning mechanisms. To address these challenges, we propose OmniRSI, a unified multimodal remote sensing image generation model. Specifically, we develop a remote sensing-specific variational autoencoder that captures multimodal priors and promotes a locally smooth latent space for unified representation and robust reconstruction across diverse observed data. Building upon that, we fully adapt Stable Diffusion 3.5 with a ControlNet branch to support unified multimodal generation and translation conditioned on textual and visual cues. Besides, we devise a training-free cyclic latent alignment mechanism that iteratively enriches incomplete visual conditions with generated multimodal information, progressively improving guidance reliability while preserving observed measurements. We also construct RSI-MM, a large-scale dataset comprising 11.5M multi-resolution images with text annotations and partially paired data for training and evaluation. Extensive experiments verify that OmniRSI achieves superior performance in multimodal reconstruction, generation and translation.

Abstract

Explore similar work

OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling

Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing