Deep Learning for Atmospheric Turbulence Mitigation: A Comparison Between GANs and Transformers

Juan S. Carrillo; Jérôme Gilles

eess.IVOct 13, 2026

Deep Learning for Atmospheric Turbulence Mitigation: A Comparison Between GANs and Transformers

Authors: Juan S. Carrillo, Jérôme Gilles

Organizations: San Diego State University, Computational Science Research Center, Department of Mathematics and Statistics, 5500 Campanile Drive, San Diego, CA 92182, USA · San Diego State University, Department of Mathematics and Statistics, 5500 Campanile Drive, San Diego, CA 92182, USA

Abstract

Atmospheric turbulence creates complex spatial and temporal distortions that significantly reduce image quality. This work compares two deep learning methods for turbulence reduction: Generative Adversarial Networks (GANs) and Transformer-based models. Both methods are trained and tested on the SOTIS dataset, which includes multi-frame sequences under different turbulence conditions with ground truth references. Our results show an important finding: GAN-based models perform better in severely degraded, low-quality situations, while Transformer-based models achieve better results in moderate to high-quality settings, especially in correcting geometric distortions. We also enhance the Turbulence Mitigation Transformer (TMT) by introducing a refined training strategy, an augmented input design that includes a temporally averaged frame as prior structural information, and a modified loss function that incorporates the Structural Similarity Index (SSIM). These improvements lead to consistent performance gains across various turbulence conditions. Overall, this study provides an in-depth evaluation of deep learning methods for turbulence reduction and offers practical guidance for selecting models in different imaging environments.

Figures & tables

Panel of Figure 1 — Figure 1: Distribution of images in the SOTIS dataset across turbulence conditions.

Panel of Figure 2 — Figure 2: Overview of the experimental pipeline used in this study. Starting from the SOTIS dataset, three main branches are evaluated: GAN based models, Transformer based TMT models and the MAMAT model included as an external reference method.

Panel of Figure 3 — Figure 3: Overview of the GAN based restoration pipeline. Turbulence degraded image sequences are first stabilized and provided as input to the generator, which produces a reconstructed image.

Panel of Figure 4 — Figure 4: Generator architecture used in the GAN model. It includes reflected edge padding, transposed convolutions, and residual blocks for stable training.

Panel of Figure 5 — Figure 5: Discriminator architecture composed of convolutional layers and a fully connected classifier that distinguishes real (clean) from generated images.

Figure 6: Internal structure of the Transformer module. It integrates normalization, depth-wise 3D convolutions, GELU activation, residual connections, and the CTJA module shown in Fig. 7 .

Figure 7: Overview of the CTJA module. Queries, keys, and values are extracted after temporal-channel feature mixing and subsequently processed through multi-head attention.

Panel of Figure 8 — Figure 8: Rearrangement of temporal-channel features into multiple attention heads prior to multi-head attention.

Figure 9: Overview of the TMT architecture, which processes a tensor (b, c, t, h, w) through downsampling, transformer-based encoding, and upsampling to reconstruct the input resolution.

Panel of Figure 10 — Figure 10: Temporal sequence of $n$ frames capturing the same scene at different times.

Panel of Figure 11 — Figure 11: Effect of temporal averaging in reducing frame tilt and geometric distortion across multiple captures of the same scene.

Figure 12: Performance of the GAN model as a function of turbulence strength

C_{n}^{2}

Panel of Figure 13 — Figure 13: Performance of the GAN model with respect to sensor–target distance $L$ .

SSIM Bin	Turbulence (SSIM)	Reconstructed (SSIM)	SSIM Gain
(0–0.2]	0.1823	0.2284	0.0461
(0.2–0.3]	0.2561	0.3023	0.0462
(0.3–0.4]	0.3520	0.4058	0.0538
(0.4–0.5]	0.4506	0.5236	0.0730
(0.5–0.6]	0.5534	0.6863	0.1329
(0.6–0.7]	0.6544	0.8334	0.1790

Table 1: GAN model performance across SSIM bins. The best improvement occurs in the [0.6–0.7] range.

SSIM B	Turb	NLTV–LK	TV–LK	TV–TVL1	T.A.	NLTV–TVL1	all
(0–0.2]	0.1748	0.1606	0.1608	0.1616	0.3321	0.3326	0.2084
(0.2–0.3]	0.2577	0.2537	0.2540	0.2481	0.4142	0.4114	0.2989
(0.3–0.4]	0.3511	0.3895	0.3898	0.3585	0.5022	0.5021	0.4204
(0.4–0.5]	0.4506	0.5627	0.5630	0.5014	0.6306	0.6299	0.5795
(0.5–0.6]	0.5554	0.7611	0.7613	0.7079	0.7849	0.7848	0.7587
(0.6–0.7]	0.6525	0.8647	0.8648	0.8436	0.8694	0.8694	0.8570

Table 2: The performance of the GEN-only model across SSIM bins for different stabilization-based training configurations is shown. Each column corresponds to a GEN model trained using inputs generated by a specific stabilization method, as well as a mixed training configuration that uses all stabilization methods jointly.

Panel of Figure 14 — Figure 14: Performance of the GEN-only model as a function of turbulence strength $C_{n}^{2}$ using all stabilization methods.

Panel of Figure 15 — Figure 15: Performance of the GEN-only model with respect to sensor–target distance $L$ using all stabilization methods.

SSIM Bin	Turbulence (SSIM)	Reconstructed (SSIM)	SSIM Gain
(0–0.2]	0.1654	0.2274	0.0618
(0.2–0.3]	0.2535	0.3470	0.0934
(0.3–0.4]	0.3518	0.4681	0.1164
(0.4–0.5]	0.4529	0.5707	0.1178
(0.5–0.6]	0.5533	0.6562	0.1029
(0.6–0.7]	0.6465	0.7288	0.0824

Table 3: Original transformer model performance across SSIM bins. The strongest improvements occur in the [0.5–0.6] and [0.6–0.7] ranges.

Figure 16: Performance of the Transformer model with respect to turbulence strength

C_{n}^{2}

Panel of Figure 17 — Figure 17: Performance of the Transformer model as a function of sensor–target distance $L$ .

SSIM Bin	Turbulence (SSIM)	Reconstructed (SSIM)	SSIM Gain
(0–0.2]	0.1654	0.2472	0.0816
(0.2–0.3]	0.2535	0.3810	0.1274
(0.3–0.4]	0.3518	0.5323	0.1805
(0.4–0.5]	0.4529	0.7055	0.2526
(0.5–0.6]	0.5533	0.8423	0.2890
(0.6–0.7]	0.6465	0.9013	0.2548

Table 4: Modified TMT model performance across SSIM bins. The strongest improvements occur in the [0.5–0.6] and [0.6–0.7] ranges.

SSIM Bin	Turbulence (SSIM)	Reconstructed (SSIM)	SSIM Gain
(0–0.2]	0.1659	0.2485	0.0826
(0.2–0.3]	0.2539	0.3945	0.1405
(0.3–0.4]	0.3518	0.5504	0.1986
(0.4–0.5]	0.4529	0.7275	0.2746
(0.5–0.6]	0.5532	0.8670	0.3138
(0.6–0.7]	0.6464	0.9210	0.2746

Table 5: Performance of the modified TMT model with the inclusion of a temporally averaged frame (TMT-TA) across SSIM bins. The strongest improvements are observed in the mid-range SSIM bins.

Panel of Figure 18 — Figure 18: Comparison of restoration results for the original TMT, the improved TMT, and the improved TMT with temporal average frame.

Panel of Figure 19 — Figure 19: Generalization performance of the proposed Transformer model on an external atmospheric turbulence dataset. (a) Direct evaluation using the model trained exclusively on the SOTIS dataset. (b) Performance after transfer learning on the external dataset.

SSIM Bin	Turbulence	Direct Evaluation	Transfer Learning	SSIM Gain
(0–0.2]	0.1582	0.2612	0.3467	0.1030 / 0.1885
(0.2–0.3]	0.2577	0.3773	0.4886	0.1196 / 0.2309
(0.3–0.4]	0.3539	0.4665	0.5927	0.1125 / 0.2388
(0.4–0.5]	0.4514	0.5435	0.6791	0.0922 / 0.2277
(0.5–0.6]	0.5515	0.6197	0.7609	0.0682 / 0.2094
(0.6–0.7]	0.6508	0.6878	0.8269	0.0370 / 0.1761

Table 6: Performance of the proposed Transformer model on the external dataset.

SSIM Bin	Turbulence (SSIM)	Reconstructed (SSIM)	SSIM Gain
(0–0.2]	0.1654	0.2255	0.0433
(0.2–0.3]	0.2535	0.3133	0.0523
(0.3–0.4]	0.3518	0.4128	0.0606
(0.4–0.5]	0.4529	0.5165	0.0647
(0.5–0.6]	0.5533	0.6116	0.0565
(0.6–0.7]	0.6465	0.6945	0.0463

Table 7: MAMAT performance across SSIM bins on the SOTIS dataset.

Model	Trainable Parameters	Model Size (MB)
GAN (training)	812,434	3.10
GEN	34,929	0.13
Transformer (TMT)	22,936,370	87.50

Table 8: Architectural complexity of the evaluated restoration models.

Model	Input	Inference (ms/image)	Throughput (img/s)
GAN (training)	$1\times 256\times 256$	2.10	475.17
GEN	$1\times 256\times 256$	2.04	489.90
TMT	$13\times 120\times 120$	753.60	1.33

Table 9: Inference performance of the evaluated restoration models.

Panel of Figure 20 — Figure 20: Comparison of SSIM values between the modified TMT model and the modified TMT model incorporating a temporally averaged frame across different SSIM bins. Blue bars represent the modified TMT model, while orange bars correspond to the modified TMT model incorporating the temporally averaged frame. Annotated values indicate the absolute SSIM gain introduced by including the averaged frame.

Panel of Figure 21 — Figure 21: GEN reconstructed outputs for different stabilization methods starting from a low SSIM input, illustrating the strong dependence of reconstruction quality on stabilization choice in severe turbulence conditions.

Panel of Figure 22 — Figure 22: Comparison of reconstructed outputs from GEN, TMT variants, and MAMAT, illustrating the superior global contrast and edge definition achieved by the Transformer based models despite occasional high SSIM values in GEN reconstructions.

Panel of Figure 23 — Figure 23: Comparison of SSIM gains for all algorithms across different SSIM regimes

Explore similar work

GSTurb: Gaussian Splatting for Atmospheric Turbulence Mitigation
Feb 26, 2026Hanliang Du, Zhangji Lu, Zewei Cai +3Image Restoration Image Deblurring
Faculty of Science and Technology, University of Macau, Macau 999078, China · College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen 518000, China · College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen, 518060, China +4
D $^2$ Turb: Depth-Aware Simulation and Decoupled Learning for Single-Frame Atmospheric Turbulence Mitigation
May 25, 2026Zixiao Hu, Tianyu Li, Guoqing Wang +4Image Deblurring
University of Electronic Science and Technology of China · Beijing Institute of Space Mechanics and Electricity · School of Physics, Northwest University Xi’an
ReMATF: Recurrent Motion-Adaptive Multi-scale Turbulence Mitigation for Dynamic Scenes
May 20, 2026Zhiming Liu, Zhicheng Zou, Nantheera AnantrasirichaiVideo Restoration
Visual Information Laboratory, School of Computer Science, University of Bristol

Feb 26, 2026J/K move · Enter open · S save

GSTurb: Gaussian Splatting for Atmospheric Turbulence Mitigation

Hanliang Du, Zhangji Lu, Zewei Cai, Qijian Tang+2

Faculty of Science and Technology, University of Macau, Macau 999078, China · College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen 518000, China · College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen, 518060, China+4

Image Restoration Image Deblurring Gaussian Splatting

Open paper AskPDF arXiv

Atmospheric turbulence causes significant image degradation due to pixel displacement (tilt) and blur, particularly in long-range imaging applications. In this paper, we propose a novel framework for atmospheric turbulence mitigation, GSTurb, which integrates optical flow-guided tilt correction and Gaussian splatting for modeling non-isoplanatic blur. The framework employs Gaussian parameters to represent tilt and blur, and optimizes them across multiple frames to enhance restoration. Experimental results on the ATSyn-static dataset demonstrate the effectiveness of our method, achieving a peak PSNR of 27.67 dB and SSIM of 0.8735. Compared to the state-of-the-art method, GSTurb improves PSNR by 1.3 dB (a 4.5% increase) and SSIM by 0.048 (a 5.8% increase). Additionally, on real datasets, including the TSRWGAN Real-World and CLEAR datasets, GSTurb outperforms existing methods, showing significant improvements in both qualitative and quantitative performance. These results highlight that combining optical flow-guided tilt correction with Gaussian splatting effectively enhances image restoration under both synthetic and real-world turbulence conditions. The code for this method will be available at https://github.com/DuhlLiamz/3DGS_turbulence/tree/main.

Abstract

Explore similar work

GSTurb: Gaussian Splatting for Atmospheric Turbulence Mitigation

D2^22Turb: Depth-Aware Simulation and Decoupled Learning for Single-Frame Atmospheric Turbulence Mitigation

ReMATF: Recurrent Motion-Adaptive Multi-scale Turbulence Mitigation for Dynamic Scenes

D $^2$ Turb: Depth-Aware Simulation and Decoupled Learning for Single-Frame Atmospheric Turbulence Mitigation