Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

Haoyang Chen; Jing Zhang; Hebaixu Wang; Shiqin Wang; Pohsun Huang; Jiayuan Li; Haonan Guo; Di Wang

cs.CVMar 4, 2026

Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

Authors: Haoyang Chen, Jing Zhang, Hebaixu Wang, Shiqin Wang, Pohsun Huang, Jiayuan Li, Haonan Guo, Di Wang,

Organizations: National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China · School of Automation, Beijing Institute of Technology ·

Abstract

Multi-modal remote sensing imagery provides complementary observations of the same geographic scene, yet such observations are frequently incomplete in practice. Existing cross-modal translation methods treat each modality pair as an independent task, resulting in quadratic complexity and limited generalization to unseen modality combinations. We formulate Any-to-Any translation as inference over a shared latent representation of the scene, where different modalities correspond to partial observations of the same underlying semantics. Based on this formulation, we propose Any2Any, a unified latent diffusion framework that projects heterogeneous inputs into a geometrically aligned latent space. Such structure performs anchored latent regression with a shared backbone, decoupling modality-specific representation learning from semantic mapping. Moreover, lightweight target-specific residual adapters are used to correct systematic latent mismatches without increasing inference complexity. To support learning under sparse but connected supervision, we introduce RST-1M, the first million-scale remote sensing dataset with paired observations across five sensing modalities, providing supervision anchors for any-to-any translation. Experiments across 14 translation tasks show that Any2Any consistently outperforms pairwise translation methods and exhibits strong zero-shot generalization to unseen modality pairs. Code and models are available at https://github.com/MiliLab/Any2Any.

Figures & tables

Figure 1: Comparison of PSNR across 14 modality translation tasks on our proposed RST-1M dataset between our method of different versions ( i.e., Any2Any-S, Any2Any-B & Any2Any-L) and representative image-to-image translation approaches (Pix2Pix ( Isola et al., 2017 ) , Pix2PixHD ( Wang et al., 2018 ) , BBDM ( Li et al., 2023 ) , ControlNet ( Zhang et al., 2023 ) and LBM ( Chadebec et al., 2025 ) ). Our Any2Any-L consistently outperforms the existing top-1 method on every modality pair, with per-modality performance gains highlighted in purple.

Dataset	#Imgs	#Imgs Pairs	Modality					#TDs	Translation Task
Dataset	#Imgs	#Imgs Pairs	RGB	SAR	NIR	MS	PAN	#TDs	Translation Task
SEN1-2 ( Schmitt et al., 2018 )	564,768	282,384	✓	✓				2	SAR $\rightarrow$ RGB
SEN12MS ( Schmitt et al., 2019 )	541,986	180,662	✓	✓				2	SAR $\rightarrow$ RGB
SpaceNet6 ( Shermeyer et al., 2020 )	6,802	3,401	✓	✓				2	SAR $\rightarrow$ RGB
QXS-SAROPT ( Huang et al., 2021 )	40,000	20,000	✓	✓				2	SAR $\rightarrow$ RGB
SAR2Opt ( Zhao et al., 2022 )	4,152	2,076	✓	✓				2	SAR $\rightarrow$ RGB

Table 1: Comparison of RST-1M with existing remote sensing image-to-image translation datasets. “#Imgs” denotes the total number of images involved in spatially aligned modality pairs, “#Img Pairs” denotes the number of such paired samples, and “#TDs” represents the number of supported modal Translation directions. “Translation Task” indicates the specific RS image-to-image translation task for which the dataset is constructed.

Figure 2: Statistics and example images of the RST-1M dataset. (a) Modality pair distribution of our RST-1M dataset, derived from five public datasets (SEN12MS, CACo, SEN1-2, Spacenet-5, and Spacenet-3). (b) Sample count for each of the seven modality pairs. (c) Statistics of the five modalities (PAN, MS, NIR, SAR, and RGB), including spatial resolution, representative examples, and image counts.

Figure 3: Overview of the Any2Any framework. The framework decouples modality-specific representation learning from shared semantic mapping: (i) independent VAEs $\{E_{i},D_{i}\}$ to establish a dimensionally unified latent manifold $\mathcal{Z}$ across heterogeneous sensors; (ii) a shared Diffusion Transformer (DiT) $f_{\theta}$ that executes $x_{0}$ -prediction steered by an MLP-based AdaLN mechanism; and (iii) target-indexed Residual Adapters $\{A_{j}\}$ for localized manifold calibration. During inference, the source latent $\mathbf{z}_{i}$ is concatenated with noise $\mathbf{z}_{t}$ and processed by the shared backbone; the predicted $\hat{\mathbf{z}}_{j}$ is then rectified by the corresponding adapter $A_{j}$ and reconstructed via $D_{j}$ in a single-pass feed-forward trajectory, ensuring efficient synthesis with constant computational overhead for any modality pair.

Method	SAR $\rightarrow$ RGB			NIR $\rightarrow$ RGB			NIR $\rightarrow$ MS			MS $\rightarrow$ RGB			SAR $\rightarrow$ MS			SAR $\rightarrow$ NIR			PAN $\rightarrow$ RGB
Method	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$
Pix2Pix ( Isola et al., 2017 )	11.84	0.08	68.57	13.14	0.08	60.35	12.65	0.38	62.62	13.58	0.12	55.81	12.46	0.32	64.29	12.57	0.08	64.94	8.57	0.00	96.26
Pix2PixHD ( Wang et al., 2018 )	18.48	0.50	35.81	18.08	0.46	36.66	11.18	0.51	73.39	18.02	0.52	39.72	10.39	0.27	80.67	10.86	0.15	78.10	19.07	0.64	33.69
BBDM ( Li et al., 2023 )	19.50	0.66	31.02	20.39	0.70	29.59	14.60	0.56	53.04	26.39	0.91	12.76	11.13	0.30	81.22	13.03	0.29	68.51	10.06	0.03	81.78
ControlNet ( Zhang et al., 2023 )	11.47	0.03	41.72	16.02	0.41	46.16	13.11	0.21	65.65	16.88	0.41	42.09	13.16	0.24	65.32	13.01	0.21	62.64	12.35	0.23	65.88
LBM ( Chadebec et al., 2025 )	14.64	0.07	48.86	13.76	0.49	58.42	18.88	0.66	34.15	11.89	0.45	71.91	13.10	0.09	58.62	14.04	0.08	51.85	27.17	0.82	13.75

Table 2: Quantitative comparison of different methods on the RST-1M test sets.

\uparrow

indicates that higher values are better, while

\downarrow

indicates that lower values are better. The best results are highlighted in bold , and the second-best results are underlined.

Figure 4: Qualitative comparison between our proposed Any2Any (A2A) and other compar methods on the test datasets. Across all modality translation tasks, our Any2Any produces results that are closer to the reference images and better preserve semantic consistency and structural integrity, demonstrating the effectiveness of the proposed method.

		Training Data		Network Architecture		Training Strategy		SAR $\rightarrow$ RGB		#Translation Directions
		SAR $\rightarrow$ RGB	RGB $\rightarrow$ SAR	w/o Adapter	w/ Adapter	Scratch	Incremental	PSNR $\uparrow$	RMSE $\downarrow$	#Translation Directions
Set 1	Setting 1	✓		✓		✓		20.68	28.51	1
Set 1	Setting 2	✓			✓	✓		20.88	27.89	1
Set 2	Setting 3	✓	✓		✓	✓		19.63	32.83	2
Set 2	Setting 4	✓	✓		✓		✓	21.44	25.87	2
		SAR $\rightarrow$ Other	Other $\rightarrow$ RGB	w/o Adapter	w/ Adapter	Scratch	Incremental	PSNR $\uparrow$	RMSE $\downarrow$

Table 3: Quantitative ablation results on RST-1M. “SAR

\rightarrow

Other” and “Other

\rightarrow

RGB” denote translations from SAR to all paired modalities and from all modalities paired with RGB to RGB, respectively. “Scratch” and “Incremental” indicate training from scratch and continued training from a pre-trained model. “#Translation Directions” denotes the number of translation directions used during training.

Figure 5: Qualitative results of our method on unseen remote sensing modality translation tasks with missing paired training data. The results demonstrate that our approach produces reasonable and semantically consistent translations under missing-modality settings, validating its any-to-any translation capability.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	DiT-S/4 (M)	DiT-B/4 (M)	DiT-L/4 (M)
Residual Adapter ( $\times$ 5)	0.006	0.006	0.006
VAE (SAR)	55.321	55.321	55.321
VAE (RGB)	55.326	55.326	55.326
VAE (MS)	13.335	13.335	13.335
VAE (NIR)	55.321	55.321	55.321
VAE (PAN)	83.649	83.649	83.649

Appendix

Table 4: Parameter breakdown of Any2Any under different DiT backbones. Modality-specific VAEs are frozen during Stage II training, while only the DiT backbone and lightweight residual adapters are optimized. All values are reported in millions (M).

Figure 6: Representative image pairs for the seven cross-modal combinations in our constructed RST-1M.

Figure 7: Visualization of the six-band Multi-Spectral (MS) modality, with individual channels shown separately.

Remote Sensing Image Understanding

Remote sensing image understanding uses AI to interpret satellite and aerial imagery for objects, land cover, change, hazards, and spatial relationships.

See all 207 papers

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Multimodal Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling
May 19, 2026Zhiping Yu, Chenyang Liu, Jinqi Cao +4Cross-Modal Learning Unified Multimodal Models
Department of Aerospace Intelligent Science and Tech-nology, School of Astronautics, Beihang University, Beijing 100191, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China · Shenyuan Honors College, Beihang University, Beijing 100191, China
One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception
May 18, 2026Yang Li, Weize Li, Quan Yuan +7Cooperative Perception Heterogeneous Collaborative Perception
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.

Jun 9, 2026J/K move · Enter open · S save

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

Miaoxin Cai, Guanqun Wang, Wei Zhang, Guangyao Zhou+5

National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China+5

Multimodal Understanding Multimodal Grounding Remote Sensing VQA Remote Sensing Image Understanding Multimodal Large Language Models Multimodal Reasoning+3

Open paper AskPDF arXiv

RS-MLLMs enable natural-language understanding and spatial reasoning over earth observation imagery. However, existing models support only a narrow range of sensor types and tasks, yielding a fragmented view of the earth and leaving cross-modal geoscientific knowledge largely unexploited. This work presents Earth-OneVision, a 2B RS-MLLM that unifies six sensor modalities (i.e., optical, SAR, infrared, multispectral, temporal, and video) and cross-sensor fusion across 9 task categories within a single autoregressive framework. Three dedicated mechanisms address three bottlenecks. Full-Granularity Vision-Language Alignment (FGVLA) aligns multi-level visual features with the multi-dimensional language space. Spatial-Linguistic Isomorphic Serialization (SLIS) unifies heterogeneous spatial outputs as autoregressive tokens. Progressive Cross-Modality Adaptation (PCMA) decomposes the compound domain gap into sequential stages, tackling the viewpoint and imaging physics gaps in turn. To support joint training, MMRS-OneVision is constructed with ~34M QA pairs spanning all six sensor modalities and cross-sensor fusion across 9 task categories, substantially exceeding existing RS multimodal instruction datasets. With only 2B parameters, Earth-OneVision achieves competitive or state-of-the-art results across extensive benchmarks, consistently matching or outperforming 4B-72B RS-MLLMs. It achieves 87.52% [email protected] on the OPT-RSVG testset for optical visual grounding and 80.68% on the SAR VQA benchmark SARLANG-Bench, exceeding 7B models by over 7%. It further achieves 75.74% recall on the BigEarthNet-MS testset for multispectral classification, and 81.94% MCQ accuracy on EarthMind-Bench for cross-modality reasoning.

Abstract

Remote Sensing Image Understanding

Explore similar work

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling

One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception