Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing

Weiyu Zhang; Yuan Hu; Yong Li; Yu Liu

cs.CVJan 25, 2026

Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing

Authors: Weiyu Zhang, Yuan Hu, Yong Li, Yu Liu

Organizations: Institute of Remote Sensing and Geographic Information System, School of Earth and Space Sciences, Peking University, Beijing, 100871, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong

Abstract

Unified remote sensing multimodal models exhibit a pronounced spatial reversal curse: Although they can accurately recognize and describe object locations in images, they often fail to faithfully execute the same spatial relations during text-to-image generation, where such relations constitute core semantic information in remote sensing. Motivated by this observation, we propose Uni-RS, the first unified multimodal model tailored for remote sensing, to explicitly address the spatial asymmetry between understanding and generation. Specifically, we first introduce explicit Spatial-Layout Planning to transform textual instructions into spatial layout plans, decoupling geometric planning from visual synthesis. We then impose Spatial-Aware Query Supervision to bias learnable queries toward spatial relations explicitly specified in the instruction. Finally, we develop Image-Caption Spatial Layout Variation to expose the model to systematic geometry-consistent spatial transformations. Extensive experiments across multiple benchmarks show that our approach substantially improves spatial faithfulness in text-to-image generation, while maintaining strong performance on multimodal understanding tasks like image captioning, visual grounding, and VQA tasks.

Figures & tables

Figure 1: Illustration of the Spatial Reversal Curse in unified remote sensing multimodal models. While the model correctly describes spatial relations from an input image (left), it often fails to faithfully execute the same relations during text-to-image generation (right).

Figure 2: Quantitative characterization of the Spatial Reversal Curse. The baseline MetaQuery* model exhibits a pronounced gap between spatial understanding and spatially faithful text-to-image generation. The * denotes that the model was trained by us under the same experimental settings as described in Section 5.1 due to the absence of public checkpoints. The Uni-RS largely reduces this gap after incorporating the proposed spatial mechanisms.

Figure 3: Overview of the Uni-RS framework. Uni-RS integrates a multimodal large language model (MLLM) with a diffusion-based image generator through learnable queries and a Transformer-based connector. During training, the framework is further enhanced by Spatial-Layout Planning, Spatial-Aware Query Supervision, and Image–Caption Spatial Layout Variation, which jointly reinforce instruction-level spatial constraints and improve spatial faithfulness during generation.

Method	FID ( $\downarrow$ )	Zero-Shot Cls-OA ( $\uparrow$ )	CLIP ( $\uparrow$ )
Attn-GAN	95.81	32.56%	20.19
DAE-GAN	93.15	29.74%	19.69
DF-GAN	109.41	51.99%	19.76
Lafite	74.11	49.37%	22.52
DALL-E	191.93	28.59%	20.13
Txt2Img-MHN ${}_{\text{VQVAE}}$	175.36	41.46%	21.35

Table 1: Comparison with existing text-to-image generation methods on RSICD.

\dagger

indicates methods evaluated without fine-tuning on RSICD. See supplementary materials for detailed descriptions of baselines.

	RSIEval			VRSBench
Method	FID ( $\downarrow$ )	SFR ( $\uparrow$ )	CLIP ( $\uparrow$ )	FID ( $\downarrow$ )	CLIP ( $\uparrow$ )
Text2Earth †	17.07	63.40	23.74	8.28	24.85
Text2Earth	16.22	65.10	23.79	7.42	24.93
Uni-RS (Ours)	13.04	80.16	24.01	4.51	25.46

Table 2: Comparison of spatial fidelity and generation quality on RSIEval and VRSBench.

\dagger

indicates methods evaluated without fine-tuning on the corresponding dataset.

RSIEval						VRSBench-val
	Captioning				VQA		Captioning				VQA	Grounding
Method	B-4	METEOR	R-L	CIDEr	Avg	Method	B-4	METEOR	R-L	CIDEr	Avg	[email protected]	[email protected]
BLIP2	0.0	3.4	11.6	0.1	45.6	Mini-Gemini	14.3	21.5	36.8	33.5	77.8	30.1	6.8
MiniGPT4	17.3	19.9	35.3	16.3	21.8	GeoChat	13.8	21.1	35.2	28.2	76.0	49.8	19.9
RSGPT	22.1	23.6	41.7	31.4	65.2	GeoPix	14.0	23.4	36.3	31.3	74.9	49.8	20.0
GeoPix †	2.8	9.4	16.1	2.9	65.5	LLaVA-1.5	14.7	21.9	36.9	33.9	76.4	41.6	13.6

Table 3: Unified comparison on RSIEval and VRSBench-val across multimodal understanding tasks. † indicates methods evaluated without task-specific fine-tuning.

Figure 4: Qualitative comparison of spatial faithfulness in text-to-image generation. BLIP3-o Chen et al. (2025a) is a general-domain unified multimodal model for understanding and generation, while CRS-Diff Tang et al. (2024) and Text2Earth Liu et al. (2025a) are remote-sensing-specific text-to-image generation models. The Text2Earth model is fine-tuned using RSICap and VRSBench datasets to produce $512\times 512$ images.

	VRSBench		RSIEval
Method	FID	CLIP	FID	CLIP	SFR
Base	4.91	25.33	13.47	23.70	47.62
Base + SLP	4.62	25.40	13.44	23.81	54.00
Base + SLP + SQS	4.64	25.42	13.33	23.79	63.49
Base + SLP + SQS + ISV	4.51	25.46	13.04	24.01	80.16

Table 4: Ablation study on VRSBench and RSIEval. Components are added cumulatively: (1) Spatial-Layout Planning (SLP), (2) Spatial-Aware Query Supervision (SQS), (3) Image–Caption Spatial Layout Variation (ISV).

Explore similar work

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing
Oct 13, 2026Pan Yin. Shuai Yang, Yuanhao Liu, Datao Tang +6Image Generation Unified Multimodal Models
Xi’an Jiaotong University
OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation
Oct 11, 2026Hebaixu Wang, Jing Zhang, Haonan Guo +3Multimodal Generation Unified Multimodal Models
Wuhan University & Zhongguancun Academy
RS-WorldModel: a Unified Model for Remote Sensing Understanding and Future Sense Forecasting
Mar 16, 2026Linrui Xu, Zhongan Wang, Fei Shen +4World Model Learning Spatiotemporal Forecasting
Central South University · Zhejiang University · National University of Singapore +2

Oct 13, 2026J/K move · Enter open · S save

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

Pan Yin. Shuai Yang, Yuanhao Liu, Datao Tang, Jialin Li+5

Xi’an Jiaotong University

Image Generation Unified Multimodal Models Remote Sensing Image Understanding Multi-Task Learning Diffusion-Based Image Generation

Open paper AskPDF arXiv

Practical remote sensing applications often require both understanding and generation capabilities for the same geographic scene. However, most existing methods address either understanding or generation in isolation, while existing unified remote sensing models' generative ability is limited to text-to-image. To bridge this gap, we propose Omni-RS, a unified framework supporting four remote sensing understanding tasks and five generation tasks. Specifically, we introduce the "Scene-Guided Two-Stage Training" strategy, which first adapts a multimodal large language model to remote sensing understanding and then uses its hidden states to guide a diffusion transformer for image generation. To enable joint learning across diverse generation tasks, we construct UniGen, a unified multi-task generation dataset, by organizing heterogeneous remote sensing data into a unified format. Furthermore, we introduce "Detailed Visual Conditioning", which uses visual conditions encoded by a VAE to guide the generation process, thereby supporting additional generation tasks that require reference images. Extensive experiments demonstrate that Omni-RS performs exceptionally well in understanding tasks and has achieved outstanding results in a variety of generative tasks.

Abstract

Explore similar work

Omni-RS: Unifying Multi-Task Understanding and Generation in Remote Sensing

OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation

RS-WorldModel: a Unified Model for Remote Sensing Understanding and Future Sense Forecasting