Learning the Target Priors Before Image Translation: A Decoupled Training Paradigm for Cross-Modal Image Translation in Remote Sensing

Keyan Hu; Mingtao Wang; Ziyu Zhou; Tiandong Shi; Haifeng Li; Ji Qi; Chao Tao

cs.CVAug 28, 2026

Learning the Target Priors Before Image Translation: A Decoupled Training Paradigm for Cross-Modal Image Translation in Remote Sensing

Authors: Keyan Hu, Mingtao Wang, Ziyu Zhou, Tiandong Shi, Haifeng Li, Ji Qi, Chao Tao

Organizations: Central South University, Changsha, China · Wuhan University, Wuhan, China · Guangzhou University, Guangzhou, China

Abstract

Cross-modal image translation in remote sensing must preserve source-observed content while matching the target-domain distribution. Existing methods jointly learn the target prior and cross-modal dependence from scarce paired data, overlooking a key asymmetry: only the latter intrinsically requires cross-modal correspondence. We formalize this distinction through conditional-score and denoising-risk analyses and propose Learning the Target Priors Before Image Translation (LTP-BIT), a prior-first paradigm that decouples the two learning tasks. LTP-BIT first learns a target-domain generative prior from large-scale unpaired imagery, then retains the pretrained backbone weights and learns source-conditioned control through P-DART, a parameter-efficient dual-stream architecture. Controlled experiments show that prior matching and scaling primarily improve target-domain realism, whereas instance fidelity relies more strongly on conditional adaptation. LTP-BIT achieves state-of-the-art performance across SAR-to-RGB and NIR-to-RGB benchmarks using only 9.81% task-specific parameters. On QXS-SAROPT, it retains near-full-data instance fidelity with only 25% of the paired samples.

Figures & tables

Figure 1: Overview of the prior-first learning paradigm in LTP-BIT for remote sensing cross-modal translation.

Figure 2: Target-domain prior initialization and scaling on QXS-SAROPT. Panels (a, b) compare random, ImageNet, and remote-sensing initialization under full-parameter fine-tuning; arrows report the epoch speedups achieved by remote-sensing initialization in reaching the respective final baseline levels. Under frozen-prior conditional adaptation, panel (c) varies the prior-backbone size with fixed DDT-head capacity, while panel (d) varies a strictly nested pretraining pool with the L prior and adaptation protocol fixed.

Prior Params CMMD $\downarrow$ PSNR $\uparrow$ S 10.0M 0.265 15.569 M 10.0M 0.254 15.422 L 10.0M 0.232 15.437 XL 10.0M 0.232 15.009	Prior Params CMMD $\downarrow$ PSNR $\uparrow$ S 10.6M 0.273 15.888 M 14.6M 0.258 15.965 L 22.0M 0.248 16.109 XL 45.4M 0.226 16.194	Data CMMD $\downarrow$ PSNR $\uparrow$ 50K 0.253 16.093 250K 0.231 16.009 500K 0.226 15.912 1M 0.224 15.845
(a) Prior-Model Scaling (Fixed DDT-Head Conditioning)	(b) Prior-Model Scaling (Full P-DART)	(c) Pretraining-Data Scaling

Table 1: Prior and data scaling. Panels (a) and (b) apply P-DART conditioning to the DDT head only and the full backbone, respectively; trainable parameters are therefore fixed in (a) and scale with the backbone in (b). Panel (c) scales nested pretraining data with the L prior and adaptation protocol fixed. Darker shading denotes better results.

Figure 3: Prior and Translation TFRR across nested pretraining-pool sizes. Panels (a) and (b) show Translation and Prior TFRR, respectively, versus normalized target-feature perturbation; panels (c)–(f) show their correspondence for 50K, 250K, 500K, and 1M pretraining samples. Color denotes perturbation magnitude; the dashed line marks equality.

Data scale	Manifold		Distribution-level		Instance-level
Data scale	Cov.@3 (%) $\uparrow$	Dens.@3 $\uparrow$	FID $\downarrow$	CMMD $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
50K	7.675	0.062	29.70	0.253	16.093	0.469
250K	7.750	0.055	22.23	0.231	16.009	0.460
500K	7.925	0.052	20.34	0.226	15.912	0.459
1M	8.000	0.053	18.98	0.224	15.845	0.461

Table 2: Target-manifold metrics and paired translation performance across nested pretraining-pool sizes.

QXS	Manifold		Distribution-level		Instance-level
QXS	Cov.@3 (%) $\uparrow$	Dens.@3 $\uparrow$	FID $\downarrow$	CMMD $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
0	7.075	0.0481	28.18	0.239	16.187	0.467
4K	21.275	0.1317	27.75	0.241	16.138	0.464
8K	29.075	0.2028	26.41	0.234	16.220	0.460
16K	36.900	0.2938	26.75	0.236	16.268	0.457

Table 3: Target-manifold metrics and paired translation performance under fixed-pool target-domain enrichment. All priors are initialized from the same model trained for 300 epochs on a 50K-sample pool and are then trained for 100 additional epochs; Coverage and Density are computed from 5K generated samples.

Figure 4: Dual-Stream Conditional Adaptation Architecture of P-DART. (a) Propagation path of the source-modality reference stream through the $\mathrm{DiT}^{\mathrm{DH}}$ architecture, comprising a DiT backbone and a DDT head; (b) bidirectional attention interactions between the reference and generation streams in adjacent DiT blocks, where the reference-stream attention is fully trainable, whereas the generation-stream attention is adapted using LoRA; and (c) architecture of the DDT head and the locations where LoRA is applied.

No pretrained generative prior
	Chesapeake
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CMMD $\downarrow$
CycleGAN	15.621	0.536	0.405	83.94	1.514
CFCA-SET	18.569	0.572 0.572	0.351	85.82	2.388
StegoGAN	16.099	0.559	0.390	48.91	0.625
BBDM	17.801	0.437	0.355	53.29	0.828

Table 5: Comparison on Chesapeake. Method variants and notation follow Table 4 .

Figure 5: Qualitative comparisons on the QXS-SAROPT dataset.

Figure 6: Effect of paired-sample scale under matched trainable parameters and 5,040 training steps. Concentric rings report CMMD (left; lower is better) and PSNR (right; higher is better) for RS-XL and IN-XL. From outer to inner, ring pairs correspond to 1K, 2K, 4K, 8K, and 16K paired samples. Annotations report the relative CMMD reduction and absolute PSNR gain of RS-XL over IN-XL.

Method	Base	Param	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CMMD $\downarrow$
Add.	FT	873M	16.349	0.380	0.459	55.03	0.358
Concat.	FT	877M	16.240	0.377	0.472	62.09	0.368
LoRA	Frozen	95M	15.936	0.361	0.455	20.55	0.222
DoRA	Frozen	95M	15.903	0.359	0.455	20.67	0.220
Ours-XL	Frozen	95M	16.194	0.362	0.445	20.68	0.226

Table 6: Full fine-tuning (FT) versus frozen-base adaptation. LoRA and DoRA use the Addition pathway.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Reconstruction errors across frequency components for RAE decoder variants. Following the frequency-domain analysis method in PRISM ( Fan et al. 2026 ) , we first average the RGB reconstruction residuals across the color channels, partition them into

8\times 8

non-overlapping patches, and compute the DCT patch by patch. The resulting 64 frequency coefficients are arranged in a zigzag order from the DC component to the high-frequency components. (a) Comparison among RAE-PT, RAE-RS, RAEv2-IN, RAEv2-Mix, and representative VAEs using 5K images randomly sampled from Git-10M. (b) Comparison of the four RAE decoders using 5K images from the ImageNet validation set.

Figure 2: Comparison of decoder reconstructions on remote sensing images. For two sets of Git-10M samples, the reconstruction results from left to right correspond to Ground Truth (GT), RAE-PT, RAE-RS, RAEv2-IN, RAEv2-Mix, SD1.5, SD3, SDXL, and Text2Earth VAE. The set-level FID, computed across 5K Git-10M evaluation images for each method, is annotated in the bottom-left corner of the first row (lower is better).

Figure 3: Comparison of RAE decoder reconstructions on ImageNet. From left to right, the results correspond to Ground Truth (GT), RAE-PT, RAE-RS, RAEv2-IN, and RAEv2-Mix. The set-level FID, computed across 5K ImageNet validation images for each method, is annotated in the bottom-left corner of the first row (lower is better).

Figure 4: Comprehensive evaluation of prior scaling and fixed-pool continued training controls. Blue and brownish-red represent the prior generation FID@5K and the translation FID@4K, respectively (lower is better). (a) Scaling up the prior model size while keeping the conditional capacity of the P-DART Decoder fixed; (b) Solid lines and solid markers indicate the continued training trajectory with progressively expanded data pools, whereas hollow markers denote the continued training controls initialized from the same parent exponential moving average (EMA) checkpoint while maintaining the original data pool. Dashed lines connect each hollow control point to the solid expansion point of the subsequent stage that shares an identical cumulative number of processed images. In the legend, “Equal compute” solely refers to the aforementioned matching of processed images, rather than implying strictly identical FLOPs or optimization trajectories; “Paired runs” indicates that a blue prior point and a brownish-red translation point with the same fill status originate from the same experimental group.

Prior backbone
Setting	S	M	L	XL
Parameters (M)	204.6	282.4	423.4	873.8
DiT backbone depth / width / heads	12 / 512 / 8	16 / 768 / 12	20 / 1024 / 16	28 / 1440 / 20
DDT head depth / width / heads	2 / 2048 / 16	2 / 2048 / 16	2 / 2048 / 16	2 / 2048 / 16
Auxiliary head attachment depth	4	5	6	8
MLP ratio	4.0	4.0	4.0	4.0

Appendix

Table 1: Prior model scales and their paired conditional adaptation configurations. The four prior tiers employ the same pretraining data and training schedule. The P-DART Decoder maintains a constant conditional capacity, whereas the encoder-side reference stream of the P-DART Codec scales synchronously with the backbone prior.

				Cumulative images	LR schedule (epochs)
Prior images	Training strategy	Initialization	Epochs	seen	Warm-up	Decay start	Decay end
50K	From scratch	Random	300	15M	100	–	200
250K	Expanded pool	50K checkpoint	100	40M	5	30	60
50K	Fixed-pool control	50K checkpoint	500	40M	5	30	60
500K	Expanded pool	250K checkpoint	100	90M	5	30	60
250K	Fixed-pool control	250K checkpoint	200	90M	5	30	60

Appendix

Table 2: Configurations for the Prior Data Scaling experiments. The expanded data pool branch and the fixed data pool branch are initialized from the same parent checkpoint and are matched by the cumulative number of processed images.

				Prior		Translation
Cumulative images	Init. from	Branch	Pool	FID $\downarrow$	CMMD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CMMD $\downarrow$
15M	Random	Initial	50K	22.35	0.856	16.093	0.373	0.469	29.70	0.253
40M	50K	Expanded	250K	14.81	0.794	16.009	0.361	0.460	22.23	0.231
40M	50K	Fixed	50K	20.25	0.833	16.057	0.369	0.466	27.21	0.241
90M	250K	Expanded	500K	13.89	0.773	15.912	0.356	0.459	20.34	0.226
90M	250K	Fixed	250K	14.37	0.782	15.946	0.359	0.460	20.86	0.222

Appendix

Table 3: Prior data scaling results for the expanded data trajectory and fixed pool continued training controls at matched cumulative training volumes. Each row reports the parent checkpoint, training route, current data pool, prior quality, and paired translation performance.

Resolution	Number of Images
$0.5\,\mathrm{m/pixel}$	344,669
$1\,\mathrm{m/pixel}$	206,801
$2\,\mathrm{m/pixel}$	124,081
$4\,\mathrm{m/pixel}$	74,449
$8$ – $128\,\mathrm{m/pixel}$	50,000 per resolution
Total	1,000,000

Appendix

Table 4: Spatial-resolution distribution of the constructed RS-1M dataset.

	Manifold metrics		Translation metrics
Pool	Cov.@3 (%) $\uparrow$	Dens.@3 $\uparrow$	FID $\downarrow$	CMMD $\downarrow$	PSNR $\uparrow$
50K	8.025	0.071	27.21	0.241	16.057
250K	8.200	0.058	20.86	0.222	15.946
500K	8.350	0.059	19.15	0.220	15.805

Appendix

Table 5: Target manifold structure and paired translation performance of the fixed-pool continued training controls. The cumulative numbers of processed images for the 50K, 250K, and 500K groups are 40M, 90M, and 190M, respectively.

	Manifold metrics		Translation metrics
QXS	Cov.@3 (%) $\uparrow$	Dens.@3 $\uparrow$	FID $\downarrow$	CMMD $\downarrow$	PSNR $\uparrow$
4K	21.275	0.132	24.33	0.241	14.748
8K	29.075	0.203	23.76	0.234	14.726
16K	36.900	0.294	23.81	0.228	14.677

Appendix

Table 6: Target-domain data enrichment controls with a frozen prior backbone. Coverage and Density are computed from the corresponding priors, while only the P-DART conditional side chain is trained during paired conditional adaptation.

Prior	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CMMD $\downarrow$
Random	14.566	0.425	0.829	453.73	6.270
ImageNet	16.1782	0.3658	0.4514	23.6509	0.2466
RS-1M	16.2187	0.3613	0.4390	19.9728	0.2273

Appendix

Table 7: Effect of backbone generative prior initialization on paired translation performance. We compare random initialization with generative pretraining on ImageNet and RS-1M.

Figure 5: Qualitative comparison of ImageNet and RS-1M prior initialization on QXS.

Dataset	End Epoch	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	CMMD $\downarrow$
Chesapeake	80	17.5835	0.3463	0.2935	18.5616	0.8395
Chesapeake	20	17.7171	0.3500	0.2904	18.1939	0.8247
SpaceNet6	80	18.9241	0.3511	0.2962	44.2808	0.8327
SpaceNet6	20	18.9237	0.3526	0.2926	43.7616	0.8084
QXS	80	16.0406	0.3518	0.4441	17.1970	0.2046
QXS	20	15.9555	0.3505	0.4457	16.5363	0.2009

Appendix

Table 8: Quantitative comparison of different curriculum endpoints on translation performance. The pure-noise replacement probability is kept at 100% for the first 10 epochs, followed by a linear decay to 25%.

Figure 6: Qualitative comparison of the fine-tuning strategies evaluated in the main text on the QXS dataset.

Figure 7: Qualitative comparisons on the QXS dataset.

Figure 8: Qualitative comparisons on the SpaceNet6 dataset.

Figure 9: Qualitative comparisons on the Chesapeake dataset.

Image-to-Image Translation

Image-to-image translation maps images between visual domains or modalities while preserving relevant content, structure, and semantics.

See all 115 papers

Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
Mar 4, 2026Haoyang Chen, Jing Zhang, Hebaixu Wang +7Remote Sensing Image Understanding Image-to-Image Translation
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China +2
Generative Translation Priors: Bayesian Imaging with Cross-Modality Image Translation
Aug 28, 2026Evan Bell, Jiaming Liu, Yifan Chen +1Image Reconstruction Bayesian Inverse Problems
Johns Hopkins University · Stanford University · University of California, Los Angeles
Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking
Aug 14, 2026Tomislav Dobrički, Byung-Woo HongImage-to-Image Translation Unpaired Image-to-Image Translation
Chung-Ang University, Seoul, South Korea

Mar 4, 2026J/K move · Enter open · S save

Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

Haoyang Chen, Jing Zhang, Hebaixu Wang, Shiqin Wang+6

National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China+2

Remote Sensing Image Understanding Image-to-Image Translation Cross-Modal Representation Transfer Multimodal Diffusion Models Cross-Modal Alignment

Open paper AskPDF arXiv

Multi-modal remote sensing imagery provides complementary observations of the same geographic scene, yet such observations are frequently incomplete in practice. Existing cross-modal translation methods treat each modality pair as an independent task, resulting in quadratic complexity and limited generalization to unseen modality combinations. We formulate Any-to-Any translation as inference over a shared latent representation of the scene, where different modalities correspond to partial observations of the same underlying semantics. Based on this formulation, we propose Any2Any, a unified latent diffusion framework that projects heterogeneous inputs into a geometrically aligned latent space. Such structure performs anchored latent regression with a shared backbone, decoupling modality-specific representation learning from semantic mapping. Moreover, lightweight target-specific residual adapters are used to correct systematic latent mismatches without increasing inference complexity. To support learning under sparse but connected supervision, we introduce RST-1M, the first million-scale remote sensing dataset with paired observations across five sensing modalities, providing supervision anchors for any-to-any translation. Experiments across 14 translation tasks show that Any2Any consistently outperforms pairwise translation methods and exhibits strong zero-shot generalization to unseen modality pairs. Code and models are available at https://github.com/MiliLab/Any2Any.

Abstract

Image-to-Image Translation

Explore similar work

Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

Generative Translation Priors: Bayesian Imaging with Cross-Modality Image Translation

Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking