Sat-JEPA-Diff: Bridging Self-Supervised Learning and Generative Diffusion for Remote Sensing
Authors: Kursat Komurcu, Linas Petkevicius
Organizations: Vilnius University, Faculty of Mathematics and Informatics, Institute of Computer Science, Artificial Intelligence Methods Lab, Vilnius, LT-03225, Lithuania
Predicting satellite imagery requires a balance between structural accuracy and textural detail. Standard deterministic methods like PredRNN or SimVP minimize pixel-based errors but suffer from the "regression to the mean" problem, producing blurry outputs that obscure subtle geographic-spatial features. Generative models provide realistic textures but often misleadingly reveal structural anomalies. To bridge this gap, we introduce Sat-JEPA-Diff, which combines Self-Supervised Learning (SSL) with Hidden Diffusion Models (LDM). An IJEPA module predicts stable semantic representations, which then route a frozen Stable Diffusion backbone via a lightweight cross-attention adapter. This ensures that the synthesized high-accuracy textures are based on absolutely accurate structural predictions. Evaluated on a global Sentinel-2 dataset, Sat-JEPA-Diff excels at resolving sharp boundaries. It achieves leading perceptual scores (GSSIM: 0.8984, FID: 0.1475) and significantly outperforms deterministic baselines, despite standard autoregressive stability limits. The code and dataset are publicly available on https://github.com/VU-AIML/SAT-JEPA-DIFF.
Figures & tables
Figure 1: Overview of Sat-JEPA-Diff. The IJEPA module (left) predicts future semantic embeddings z^t+1 from input It . These embeddings, combined with coarse spatial structure, condition a frozen SD3.5 backbone via a learned adapter to generate I^t+1 .
Model
L1 ↓
MSE ↓
PSNR ↑
SSIM ↑
GSSIM ↑
LPIPS ↓
FID ↓
Deterministic Baselines
Default
0.0131
0.0008
37.52
0.9361
0.7858
0.0708
0.6959
PredRNN Wang et al. (2022)
0.0117
0.0005
38.38
0.9476
0.7836
0.0726
9.9720
SimVP v2 Gao et al. (2022)
0.0131
0.0006
37.63
0.9391
0.7719
0.0928
18.7208
Generative Models
SD 3.5 Esser et al. (2024)
0.0175
0.0005
32.98
0.8398
0.8711
0.4528
0.1533
Table 1: Quantitative comparison on the test set. Arrows indicate whether lower ( ↓ ) or higher ( ↑ ) values are better. Best results are highlighted in bold .
Figure 2: Qualitative comparison of next-frame predictions ( t→t+1 ). While deterministic baselines (PredRNN, SimVP) suffer from spectral blurring, Sat-JEPA-Diff preserves high-frequency details and geospatial boundaries.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Geographical distribution of the 100 selected Regions of Interest (RoIs).
Figure 4: Systematic IJEPA Loss Ablation. Validation metrics over 100 epochs demonstrate that our full objective function (E curve) uniquely avoids representation collapse and maintains high embedding variance compared to near-zero variance in the reduced underlying models (A-D curves). Despite higher total loss, the full model maintains high cosine similarity and achieves superior spatial variance.
Figure 5: Long-horizon autoregressive rollout comparison ( 2018→2024 ) on the Rio de Janeiro Coast. Top Row: Ground Truth. Rows 2-3: Deterministic baselines rapidly degrade into spectral blurring (spatial collapse) after 2-3 steps. Bottom Row: Sat-JEPA-Diff maintains high contrast and structural sharpness throughout the 7-year horizon.
We introduce TerraDiT, a diffusion transformer designed for text-to-satellite image generation with point-based control. Existing controlled satellite image generative models often require pixel-level maps that are time-consuming to acquire, yet semantically limited. To address this limitation, we introduce a novel point-based conditioning framework that controls the generation process through the spatial location of the points and the textual description associated with each point, providing semantically rich control signals. This approach enables flexible, annotation-friendly, and computationally simple inference for satellite image generation. To this end, we introduce an adaptive local attention mechanism that effectively regularizes the attention scores based on the input point queries. We systematically evaluate various domain-specific design choices for training TerraDiT, including the selection of satellite image representation for alignment and geolocation representation for conditioning. Our experiments demonstrate that TerraDiT achieves impressive generation performance, surpassing the state-of-the-art remote sensing generative models. Our models, dataset, and code are available at https://github.com/mvrl/TerraDiT.
Generative diffusion priors have recently achieved state-of-the-art performance in natural image super-resolution, demonstrating a powerful capability to synthesize photorealistic details. However, their direct application to remote sensing image super-resolution (RSISR) reveals significant shortcomings. Unlike natural images, remote sensing images exhibit a unique texture distribution where ground objects are globally stochastic yet locally clustered, leading to highly imbalanced textures. This imbalance severely hinders the model's spatial perception. To address this, we propose TexADiff, a novel framework that begins by estimating a Relative Texture Density Map (RTDM) to represent the texture distribution. TexADiff then leverages this RTDM in three synergistic ways: as an explicit spatial conditioning to guide the diffusion process, as a loss modulation term to prioritize texture-rich regions, and as a dynamic adapter for the sampling schedule. These modifications are designed to endow the model with explicit texture-aware capabilities. Experiments demonstrate that TexADiff achieves superior or competitive quantitative metrics. Furthermore, qualitative results show that our model generates faithful high-frequency details while effectively suppressing texture hallucinations. This improved reconstruction quality also results in significant gains in downstream task performance. The source code of our method can be found at https://github.com/ZezFuture/TexAdiff.
Diffusion models have emerged as powerful tools for a wide range of vision tasks, including text-guided image generation and editing. In this work, we explore their potential for object grounding in remote sensing imagery. We propose a hybrid pipeline that integrates diffusion-based localization cues with state-of-the-art segmentation models such as RemoteSAM and SAM3 to obtain more accurate bounding boxes. By leveraging the complementary strengths of generative diffusion models and foundational segmentation models, our approach enables robust and adaptive object localization across complex scenes. Experiments demonstrate that our pipeline significantly improves localization performance, achieving over a 14% increase in [email protected] compared to existing state-of-the-art methods.