Semantic Synthesis over Brute-Force Search: LLM-Guided Transformation Design for Efficient Learning
Organizations: Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Abstract
Data transformation pipelines shape the training distribution in computer vision, but designing them is a computationally expensive problem. We investigate whether large language models can enable efficient discovery by synthesizing executable transformation programs outside predefined augmentation spaces. We compare nine strategies under a unified training protocol, including standard augmentation policies, brute-force search, direct code generation with two LLMs, and performance-guided generation using LLMs fine-tuned on accuracy-ordered transformation pairs. When the best candidates are retrained for 20 epochs, LLM-generated transformation programs lead on every dataset, reaching 0.8877 accuracy on CIFAR-10, 0.5825 on CIFAR-100, and 0.9588 on SVHN, surpassing brute-force search over 6,000 candidates. The best transform on CIFAR-10 and SVHN comes from a fine-tuned generator using 270 candidates, while the best on CIFAR-100 comes from an off-the-shelf generator using only 10. This needs 22x fewer candidate evaluations in the generation step and 600x fewer for off-the-shelf generations, relative to brute-force search. Beyond transformation discovery, we analyze the reliability of the reduced-budget evaluation commonly used for augmentation search. Correlating one-epoch selection scores with 20-epoch outcomes reveals strong, weak, and absent predictive relationships depending on the dataset and architecture, with correlations ranging from 0.756 to near zero. A positive control indicates that range restriction alone is unlikely to explain the low correlations. Our results indicate that semantic synthesis with code LLMs provides an efficient alternative to search-based transformation discovery and highlight the importance of validating evaluation fidelity when optimizing data transformations.
Figures & tables
| Dataset | Architecture | Mean | Best | 95% CI (mean) | ||
|---|---|---|---|---|---|---|
| CIFAR-10 | ResNet | 1 | 2000 | 0.5256 | 0.6124 | [0.5234, 0.5279] |
| 2 | 2000 | 0.4832 | 0.6071 | [0.4801, 0.4863] | ||
| 3 | 2000 | 0.4401 | 0.5983 | [0.4363, 0.4439] | ||
| MobileNetV2 | 1 | 2000 | 0.3895 | 0.4604 | [0.3877, 0.3914] | |
| 2 | 2000 | 0.3569 | 0.4655 | [0.3545, 0.3593] | ||
| 3 | 2000 | 0.3261 | 0.4517 | [0.3231, 0.3290] |
| Dataset | Strategy | Candidate | Acc@1ep | Acc@20ep | rank |
|---|---|---|---|---|---|
| CIFAR-10 | No augmentation | 1 | 0.5832 | 0.7656 | |
| AutoAugment | 1 | 0.4083 | 0.7537 | ||
| RandAugment | 1 | 0.3554 | 0.7149 | ||
| TrivialAugment | 1 | 0.4673 | 0.8037 | ||
| Brute force | 6000 | 0.6124 | 0.8115 | ||
| OlympicCoder-7B | 10 | 0.4708 | 0.8728 |
| Dataset | Architecture | Score span | CI | |||
| Panel A: one-epoch selection score 20-epoch accuracy | ||||||
| CIFAR-10 | ResNet | 43 | 0.350–0.612 | 0.88 | ||
| CIFAR-10 | MobileNetV2 | 43 | 0.200–0.527 | 0.002 | ||
| CIFAR-100 | ResNet | 42 | 0.035–0.260 | 0.88 | ||
| SVHN | ResNet | 41 | 0.524–0.906 | 0.001 | ||
| Panel B: control, ResNet one-epoch MobileNetV2 one-epoch | ||||||
Data Augmentation
Data augmentation expands or transforms training examples to improve learning, data efficiency, and robustness to distribution shifts.
See all 273 papers