Procedural Adversarial Learning
Organizations: Department of Computer Science, Princeton University
Abstract
For many computer vision tasks, neural networks can be trained to produce a low average error, yet they still fail on seemingly simple scenes. Reducing these failures requires first finding them, but they may not be in a fixed dataset. We study procedural adversarial learning (PAL), a framework of minimax training and adversarial evaluation which searches the input space of a procedural generator for difficult instances. The procedural generator is an interpretable program mapping a low-dimensional vector of control parameters (e.g. layout, lighting, pose) to a rendered image and its label. We introduce the Simple-PAT algorithm, which uses zeroth-order optimization in the procedural generator's parameter space to propose adversarial training samples that are generated on demand. We also create PAL-Dojo, a collection of efficient procedural generators for surface normal estimation, image segmentation, and human pose estimation. Against a cross-model black-box adversary with 50M aggregate queries, Simple-PAT reduces worst-case error by a geometric mean of 3.15x compared to standard training and 2.63x compared to the strongest robust training baseline. Code is available at https://github.com/princeton-vl/procedural-adversarial-learning.
Figures & tables
| Task | Dimension | Resolution | Scenes/second |
|---|---|---|---|
| Normals | 25 | 8700 | |
| Chair-Simple | 13 | 870 | |
| Chair-Full | 69 | 350 | |
| Pose | 49 | 320 |
| Cross-model, queries | Test set, 65,536 scenes | |||||||
|---|---|---|---|---|---|---|---|---|
| Task | Method | Worst case | vs. Simple-PAT | Max | Median | |||
| Normals | Uniform (Adam) | 49.7 | 1.1 | 0.0079 | 10.63 | 0.32 | 1.636 | 0.022 |
| angle ( ∘ ) | Uniform (Muon) | 43.4 | 3.0 | 0.0079 | 5.53 | 0.47 | 0.8904 | 0.0041 |
| Tilted | 37.1 | 1.4 | 0.0079 | 8.06 | 0.64 | 3.732 | 0.020 | |
| CVaR | 38.4 | 2.2 | 0.0079 | 5.186 | 0.092 | 1.8097 | 0.0081 | |
| Simple-PAT | 16.0 | 2.3 | — | 3.51 | 0.22 | 1.713 | 0.018 | |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Single-model, queries | Test set, 65,536 scenes | |||||||
|---|---|---|---|---|---|---|---|---|
| Task | Method | Worst case | vs. Simple-PAT | Max | Median | |||
| Normals | Uniform (Adam) | 49.7 | 1.1 | 0.0079 | 10.63 | 0.32 | 1.636 | 0.022 |
| angle ( ∘ ) | Uniform (Muon) | 43.4 | 3.0 | 0.0079 | 5.53 | 0.47 | 0.8904 | 0.0041 |
| Tilted | 37.1 | 1.4 | 0.0079 | 8.06 | 0.64 | 3.732 | 0.020 | |
| CVaR | 38.4 | 2.2 | 0.0079 | 5.186 | 0.092 | 1.8097 | 0.0081 | |
| Simple-PAT | 15.4 | 2.8 | — | 3.51 | 0.22 | 1.713 | 0.018 | |
| Search | Test set, 65,536 scenes | ||||||
| Task | Model | Worst case | Max | Median | |||
| Normals | MoGe-2 (64 px) | 82.8 | 54.5 | 22.2 | |||
| angle ( ∘ ) | MoGe-2 (512 px) | 69.8 | 52.0 | 18.9 | |||
| Simple-PAT (single) | 15.4 | 2.8 | 3.51 | 0.22 | 1.713 | 0.018 | |
| Simple-PAT (cross) | 16.0 | 2.3 | |||||
| Chair-Simple | SAM 3 | 2.98 | 3.29 | 0.00172 | |||
Adversarial Robustness
Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.
See all 509 papers