Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality
Organizations: CSAIL, Department of EECS, Massachusetts Institute of Technology · School of CIT, MCML, MDSI, Technical University of Munich
Abstract
Synthetic data can scale visual pretraining, but current selection criteria only partially explain which datasets train strong representations. We propose using the energy profile of a dataset--the distribution of log-probability levels it occupies under a normalised real-image density model--as an audit and curation principle for synthetic data. To operationalise this view, we introduce Energy-Level Distribution Distance (ELD), which compares real and synthetic energy distributions. Real images span sparse high-likelihood scenes and dense low-likelihood textures, so useful synthetic data should cover this energy range rather than collapse to likely but low-variation samples. Across diverse synthetic datasets, ELD exposes mismatches missed by FID and predicts pretraining performance beyond FID, LPIPS variation, and feature-space recall, including across CNN and ViT self-supervised settings. We then use ELD to resample and mix synthetic sources, matching the real-image energy histogram while preserving diversity and improving downstream transfer.
Figures & tables
| Data metrics | DINO/ViT-S downstream | |||||||
| Dataset | Eld | FID | LPIPS | Recall | IN-1K Lin.% | IN-1K FT% | IN-C% | IN-A% |
| Noise-like | ||||||||
| FractalDB | .426 | 1254 | .260 | .018 | 23.91 | 56.66 | 18.70 | 10.30 |
| Perlin noise | .427 | 1922 | .208 | .000 | 21.53 | 55.38 | 16.79 | 9.11 |
| Gaussian noise | .500 | 2145 | .237 | .000 | 20.26 | 52.26 | 15.53 | 5.01 |
| Over-concentrated | ||||||||
| DINO linear evaluation | ELD | FID | LPIPS | Recall |
| All sources ( ) | 0.82 | 0.91 | 1.67 | 4.53 |
| Without noise-like sources ( ) | 0.99 | 1.66 | 1.47 | 3.27 |
| Data metrics | Learning-to-See downstream | |||||||
| Dataset | Eld | FID | LPIPS | Recall | IN Lin.% | IN FT% | IN-C% | IN-A% |
| Dead leaves | ||||||||
| dead-leaves squares | .393 | 1480 | .316 | .005 | 38.60 | 67.52 | 27.23 | 3.24 |
| dead-leaves oriented | .319 | 1390 | .299 | .035 | 40.90 | 68.48 | 25.12 | 3.97 |
| dead-leaves mixed | .310 | 1310 | .301 | .074 | 40.84 | 68.86 | 25.00 | 3.82 |
| dead-leaves textures | .261 | 1340 | .241 | .084 | 41.00 | 68.20 | 27.93 | 3.83 |
| Scorer | Agreement with Guth | Linear | Fine-tuning |
| Guth dual-score model | 1.00 | -0.87 | -0.78 |
| DenseFlow | 0.89 | -0.85 | -0.78 |
| Diffusion VLB | 0.91 | -0.77 | -0.72 |
| Selection | NN mean | NN p95 | Eff. classes | Eff. rank |
| Uniform | 0.634 | 0.784 | 116.445 | 59.175 |
| Top likelihood | 0.660 | 0.788 | 82.940 | 34.132 |
| ELD-guided | 0.621 | 0.780 | 128.001 | 71.828 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Regime | # datasets | Mean shift | Mean spread | Mean pEld |
| Over-concentrated | 3 | -4.43 | 0.52 | .281 |
| Broad / shifted | 7 | -5.41 | 1.47 | .284 |
| Noise-like / high-energy | 4 | 2.92 | 0.22 | .300 |
| Surrogate energy model | Description | Spearman with pELD |
| ResNet-feature Gaussian | Diagonal Gaussian on frozen features | |
| DCT Gaussian | Diagonal Gaussian on low-frequency DCT coefficients | |
| DCT-GMM | GMM on PCA-compressed DCT coefficients |
| DINO subset | ELD | FID | LPIPS | Recall | |
| All sources | 9 | 0.82 | 0.91 | 1.67 | 4.53 |
| Without noise-like | 6 | 0.99 | 1.66 | 1.47 | 3.27 |
| Without GANs | 7 | 1.06 | 1.17 | 1.63 | 1.34 |
| Without diffusion | 6 | 1.07 | 0.46 | 2.26 | 4.08 |
| Without uniform mixture | 8 | 0.76 | 0.96 | 1.75 | 4.95 |
Synthetic Data Pretraining
Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.
See all 130 papers