Breaking Noise Shortcuts in Self-Supervised Learning Via Noise-Aligned View Generation
Organizations: Department of Electrical & Computer Engineering, Georgia Institute of Technology, Atlanta, GA, 30332 · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, 30332
Abstract
Self-supervised learning (SSL) is increasingly applied to scientific imaging domains, such as microscopy, where labels are scarce but noisy data is abundant. These domains exhibit significant signal-dependent noise, which can create a representational shortcut: since both augmented views derive from the same noisy image, the encoder can boost view agreement by also encoding noise patterns rather than semantic content. To enable noise-robust representation learning without knowledge of underlying noise models, we introduce Self-Aligned Noise Augmentation (SEANA), a drop-in module that generates noise-aware SSL views. SEANA learns an invertible variance-stabilizing transform (VST) from noisy data, then estimates the clean signal and resamples independent noise in learned VST space, requiring no clean targets, no dataset-specific denoiser training or use at inference, and no changes to the SSL objective or encoder. On CIFAR-10 and ImageNet-100 under synthetic Gaussian, Poisson-Gaussian, and multiplicative noise, SEANA improves clean-test linear-probe accuracy across contrastive/non-contrastive SSL methods, indicating higher-quality representations. On real fluorescence imaging, SEANA improves Jurkat cell-cycle stage classification by up to 28.6pp over standard SSL and 27.5pp over denoiser-preprocessed baselines. In all settings, SEANA outperforms the denoiser-preprocessed SSL pipelines, demonstrating that learned VST-space noise-aligned resampling yields more robust representations than denoising alone. Code is available at: https://github.com/opatil31/SEANA
Figures & tables
| Noise condition | Baseline | SEANA | Reduction |
|---|---|---|---|
| Gaussian (low) | 0.021 | 0.013 | 40.3% |
| Gaussian (high) | 0.115 | 0.063 | 45.1% |
| PG (low) | 0.026 | 0.019 | 25.5% |
| PG (high) | 0.079 | 0.051 | 35.8% |
| Speckle (low) | 0.020 | 0.016 | 23.3% |
| Speckle (high) | 0.051 | 0.036 | 29.9% |
| Noise | SSL | Base | N2N | SEANA | Noise | SSL | Base | N2N | SEANA | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| (Low) Gauss | BYOL | 52.6 1.1 | 53.1 0.7 | 60.8 0.2 | +8.2 | (High) PG | BYOL | 30.0 0.9 | 26.9 1.8 | 47.8 0.6 | +17.8 |
| SimCLR | 59.4 0.7 | 59.6 0.7 | 66.0 0.2 | +6.6 | SimCLR | 31.8 2.1 | 24.5 0.5 | 50.5 0.4 | +18.7 | ||
| VICReg | 63.0 0.4 | 64.2 0.7 | 69.4 0.1 | +6.4 | VICReg | 34.2 2.1 | 24.9 2.4 | 54.7 0.3 | +20.5 | ||
| (High) Gauss | BYOL | 28.4 0.2 | 25.3 0.7 | 41.9 0.4 | +13.5 | (Low) Speckle | BYOL | 51.3 1.2 | 50.6 0.2 | 61.5 0.6 | +10.3 |
| SimCLR | 27.5 0.9 | 26.8 0.4 | 42.2 0.5 | +14.7 | SimCLR | 58.4 0.2 | 58.2 0.5 | 67.0 0.1 | +8.6 | ||
| VICReg | 29.8 1.1 | 30.1 1.9 | 47.4 0.6 | +17.6 | VICReg | 63.6 0.5 | 63.4 0.1 | 71.4 0.4 | +7.8 |
| Noise | SSL | Base | N2N | SEANA | Noise | SSL | Base | N2N | SEANA | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| (Low) Gauss | BYOL | 55.9 0.9 | 57.5 0.5 | 60.5 0.2 | +4.6 | (High) Shot | BYOL | 42.6 0.3 | 43.5 1.0 | 50.9 0.3 | +8.4 |
| SimCLR | 51.9 0.3 | 53.7 0.2 | 58.0 0.1 | +6.1 | SimCLR | 40.8 0.9 | 41.1 0.0 | 47.1 0.1 | +6.3 | ||
| VICReg | 54.1 0.2 | 55.1 0.3 | 59.3 0.6 | +5.3 | VICReg | 42.5 0.4 | 41.0 0.5 | 49.1 0.2 | +6.6 | ||
| (High) Gauss | BYOL | 39.2 0.7 | 38.0 0.3 | 46.5 0.4 | +7.2 | (Low) Speckle | BYOL | 54.8 0.4 | 57.3 0.8 | 59.8 0.3 | +4.9 |
| SimCLR | 38.7 0.8 | 36.9 0.4 | 44.5 0.4 | +5.7 | SimCLR | 51.0 0.5 | 53.9 0.3 | 56.5 0.6 | +5.5 | ||
| VICReg | 39.6 0.7 | 35.5 0.5 | 45.4 0.2 | +5.8 | VICReg | 51.6 0.8 | 55.1 0.9 | 58.3 0.9 | +6.7 |
| Noise | SSL | Baseline | N2N (Denoiser) + SSL | NSP (Denoiser) + SSL | SEANA (Ours) | |
|---|---|---|---|---|---|---|
| Real noise | BYOL | 63.4 3.4 | 67.3 2.0 | 67.9 2.3 | 75.7 0.1 | +12.3 |
| SimCLR | 46.9 0.3 | 48.2 0.6 | 46.6 0.6 | 75.5 0.1 | +28.6 | |
| VICReg | 46.8 0.2 | 47.4 0.8 | 48.6 0.3 | 74.4 0.2 | +27.5 | |
| LeJEPA | 47.5 0.4 | 49.4 0.3 | 49.3 0.4 | 74.4 0.3 | +26.9 | |
| MoCoV2 | 71.5 1.9 | 73.3 0.1 | 73.9 0.1 | 75.1 0.6 | +3.6 |
| SSL Method | Baseline | VST Only | Resampling Only | SEANA (Full) |
|---|---|---|---|---|
| BYOL | 63.4 3.4 | 59.6 0.1 | 46.5 0.2 | 75.7 0.1 |
| SimCLR | 46.9 0.3 | 47.3 0.5 | 74.2 0.0 | 75.5 0.1 |
| VICReg | 46.8 0.2 | 46.1 1.0 | 73.3 0.1 | 74.4 0.2 |
| LeJEPA | 47.5 0.4 | 47.6 0.3 | 73.2 0.1 | 74.4 0.3 |
| MoCoV2 | 71.5 1.9 | 66.8 0.1 | 68.1 1.1 | 75.1 0.6 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| SSL method | Baseline | NSP + SSL | SEANA |
|---|---|---|---|
| SimCLR | |||
| VICReg | |||
| LeJEPA | |||
| BYOL | |||
| MoCoV2 |
| SSL Method | Approach | Brain | Thorax | Cervix | Abdomen | Femur | Other | Macro-F1 |
|---|---|---|---|---|---|---|---|---|
| SimCLR | Baseline | 78.7 | 65.5 | 92.5 | 44.0 | 43.0 | 69.7 | 65.6 |
| SEANA | 84.3 | 70.5 | 91.5 | 55.7 | 51.8 | 72.2 | 71.0 | |
| VICReg | Baseline | 77.3 | 66.5 | 92.1 | 42.1 | 44.2 | 70.2 | 65.4 |
| SEANA | 82.3 | 70.1 | 90.4 | 51.4 | 47.8 | 70.8 | 68.8 | |
| LeJEPA | Baseline | 78.3 | 69.5 | 91.6 | 43.3 | 51.2 | 70.6 | 67.4 |
| SEANA | 82.8 | 70.2 | 90.7 | 51.8 | 48.9 | 70.9 | 69.2 |
| SSL method | E6 + Aloka | Voluson S10 |
|---|---|---|
| SimCLR | ||
| VICReg | ||
| LeJEPA | ||
| BYOL | ||
| MoCoV2 |
| DRUNet | DnCNN | SCUNet | SwinIR | ||
|---|---|---|---|---|---|
| SSL | Base | U-Net+Res | Plain CNN | Swin-UNet | Swin Trans. |
| BYOL | 63.4 3.4 | 75.7 0.1 | 75.3 0.1 | 75.8 0.1 | 75.3 0.1 |
| SimCLR | 46.9 0.3 | 75.5 0.1 | 75.1 0.1 | 75.3 0.1 | 75.2 0.1 |
| VICReg | 46.8 0.2 | 74.4 0.2 | 74.7 0.1 | 75.2 0.2 | 75.1 0.2 |
| Mean | 52.4 | 75.2 | 75.0 | 75.4 | 75.2 |
| Rank | Approach | Privileged information | Mean accuracy (%) | Mean gain (pp) | Median gain (pp) | Gain range (pp) |
|---|---|---|---|---|---|---|
| – | Oracle | Clean images and exact noise model | 59.37 | |||
| 1 | SEANA | None | 57.68 | |||
| 2 | Analytic VST + R | Noise family and relevant parameters | 54.86 | |||
| 3 | NSP + R | Exact noise model | 53.24 | |||
| 4 | NSP | None | 46.35 | |||
| 5 | N2N + R | Exact noise model | 45.68 |
| Noise condition | SSL | N2N | NSP | N2N + R | NSP + R | Analytic VST + R | SEANA | Oracle | Baseline |
|---|---|---|---|---|---|---|---|---|---|
| Gaussian (low) | BYOL | 51.8 | 54.4 | 57.6 | 57.4 | 60.7 | 60.8 | 61.4 | 52.3 |
| SimCLR | 59.8 | 58.6 | 62.3 | 58.1 | 61.9 | 66.0 | 67.2 | 60.1 | |
| VICReg | 64.8 | 61.7 | 67.3 | 64.6 | 69.3 | 69.4 | 69.5 | 62.5 | |
| Gaussian (high) | BYOL | 22.9 | 25.8 | 26.9 | 40.3 | 41.0 | 41.9 | 49.3 | 27.5 |
| SimCLR | 28.1 | 29.2 | 36.1 | 40.5 | 42.5 | 42.2 | 49.1 | 22.1 | |
| VICReg | 30.1 | 28.9 | 38.3 | 43.8 | 43.4 | 47.4 | 50.1 | 30.8 |
| Noise condition | N2N | NSP | DRUNet | DRUNet + SEANA VST |
|---|---|---|---|---|
| Gaussian (high) | 13.46 | 17.55 | 16.90 | 17.96 |
| Poisson–Gaussian (high) | 15.00 | 18.44 | 15.97 | 19.96 |
| Speckle (high) | 16.00 | 18.76 | 13.56 | 20.13 |
| Number of knots | |||||
|---|---|---|---|---|---|
| SSL | 8 | 16 | 32 | 64 | 128 |
| BYOL | 74.4 | 75.8 | 75.7 | 75.5 | 75.5 |
| SimCLR | 75.4 | 75.5 | 75.8 | 75.7 | 75.7 |
| VICReg | 74.0 | 74.1 | 73.7 | 74.3 | 74.4 |
| Mean | 74.6 | 75.1 | 75.1 | 75.2 | 75.2 |
| SEANA | Dataset-Specific Denoiser (N2N) + Resampling | |
|---|---|---|
| What is optimized | Monotonic spline (VST) | U-Net (denoiser) |
| Trainable parameters | 130 | 1,259,475 ( ) |
| Optimization steps | 5,000 | 20,000 ( ) |
| Requires clean/paired data | No | No |
| Requires noise model knowledge | No | Yes |
Self-Supervised Visual Representation Learning
Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.
See all 257 papers