Architectural Insights for Post-Tornado Damage Recognition
Organizations: Lisa · The University of Alabama · Colorado State University · Argonne National Laboratory · Old Dominion University · University of South Alabama
Abstract
Rapid and accurate building damage assessment in the immediate aftermath of tornadoes is critical for coordinating life-saving search and rescue operations, optimizing emergency resource allocation, and accelerating community recovery. However, current automated methods struggle with the unique visual complexity of tornado-induced wreckage, primarily due to severe domain shift from standard pre-training datasets and extreme class imbalance in real-world disaster data. To address these challenges, we introduce a systematic experimental framework evaluating 79 open-source deep learning models, encompassing both Convolutional Neural Networks (CNNs) and Vision Transformers, across over 2,300 controlled experiments on our newly curated Quad-State Tornado Damage (QSTD) benchmark dataset. Our findings reveal that achieving operational-grade performance hinges on a complex interaction between architecture and optimization, rather than architectural selection alone. Most strikingly, we demonstrate that optimizer choice can be more consequential than architecture: switching from Adam to SGD provided dramatic F1 gains of +25 to +38 points for Vision Transformer and Swin Transformer families, fundamentally reversing their ranking from bottom-tier to competitive with top-performing CNNs. Furthermore, a low learning rate of 1x10^(-4) proved universally critical, boosting average F1 performance by +10.2 points across all architectures. Our champion model, ConvNeXt-Base trained with these optimized settings, demonstrated strong cross-event generalization on the held-out Tuscaloosa-Moore Tornado Damage (TMTD) dataset, achieving 46.4% Macro F1 (+34.6 points over baseline) and retaining 85.5% Ordinal Top-1 Accuracy despite temporal and sensor domain shifts.
Figures & tables
| Element | DS1 - Slight | DS2 - Moderate | DS3 - Extensive | DS4 - Complete |
|---|---|---|---|---|
| Roof Covering | 2–15% of Roof Covering Damaged | 15–50% of Roof Covering Damage | More than 50% of Roof Covering Damaged | More than 50% of Roof Covering Damaged (typically) |
| AND/OR | ||||
| Window/Door | 1 window or door failure | 2 or 3 windows/doors failed | More than 3 windows/doors failed | More than 3 windows/doors failed (typically) |
| AND/OR | ||||
| Roof Sheathing | No Roof Sheathing Failure | 1–3 sections of roof sheathing failed | More than 3 sections AND less than 35% of roof sheathing failed | More than 35% of roof sheathing failed |
| AND/OR | ||||
| Architectural | Pre-training | Representative Models | Stage 0 | Stage 1 | Stage 2 | Stage 3 |
| Family | Sources | (Stage 0) | (Zero-Shot) | (Base Case) | (Optimization) § | (TTD) |
| AlexNet | †, | AlexNet [ 32 ] | 2 | 2 | 58 | ✗ |
| ConvNeXt | † | ConvNext-Small [ 44 ] | 1 | 4 | 116 | ✓ |
| DenseNet | †, | DenseNet161 [ 37 ] | 2 | 5 | 145 | ✗ |
| EfficientNet | † | EfficientNet-B3 [ 38 ] | 1 | 12 | 348 | ✗ |
| GoogLeNet | † | GoogLeNet [ 42 ] | 1 | 1 | 29 | ✗ |
| Hyperparameter | Base Case | Search Space | Variations ‡ | Domain-Specific Goal |
| Activation function | Model Default* | {ReLU, LeakyReLU, GELU, SiLU, ELU} | 5 | Investigate impact of non-linearity on learning complex damage textures and preventing dying neurons |
| Batch Size | 64 | {64, 32, 16, 8, 4} | 4 | Balance gradient stability with noise-induced regularization; assess impact on rare class representation |
| Optimizer | Adam | {Adam, SGD (momentum=0.9)} | 1 | Compare adaptive vs. momentum-based optimization for disaster imagery |
| Regularization | None | {None, Dropout with } | 3 | Combat overfitting to dataset-specific debris patterns and lighting artifacts |
| Input resolution | { , , , , , , } † | 6 | Enable multi-scale feature recognition from fine-grained (windows) to gross (structural) damage |
| Rank ‡ | Model | Pre-training | Macro-F1 | Ordinal Top-1 | Standard ¶ Top-1 | Param ‖ | Inference |
|---|---|---|---|---|---|---|---|
| (%) | Acc. (%) | Acc. (%) | (M) | Time (ms) | |||
| 1 | EfficientNet-B3 | † | 14.3 | 50.5 | 26.1 | 10.7 | 13.3 |
| 2 | DenseNet161 | 12.6 | 75.9 | 61.0 | 26.5 | 21.5 | |
| 3 | AlexNet | 11.3 | 33.5 | 23.2 | 57.0 | 0.6 | |
| 4 | GoogLeNet | † | 8.4 | 14.8 | 9.4 | 5.6 | 7.1 |
| 5 | AlexNet | † | 6.9 | 39.5 | 8.3 | 57.0 | 0.6 |
| Model Configuration | Macro F1 | Ordinal | Std. | DS0 | DS1 | DS2 | DS3 | DS4 | DS5 |
|---|---|---|---|---|---|---|---|---|---|
| (%) | Top-1 Acc (%) | Top-1 Acc (%) | (F1) | (F1) | (F1) | (F1) | (F1) | (F1) | |
| Naive Baseline | 11.8 | 43.9 | 14.0 | 9.3 | 0.7 | 19.4 | 17.8 | 7.9 | 15.9 |
| (ImageNet TMTD) | |||||||||
| QSTD-Optimized | 46.4 | 85.5 | 58.3 | 63.5 | 20.6 | 22.6 | 36.2 | 75.5 | 59.1 |
| (ImageNet QSTD TMTD) |
| Component | Recommendation | Impact | Notes |
|---|---|---|---|
| Optimizer | SGD with momentum 0.9 for Vision Transformers; Adam acceptable for CNNs | +25 to +38 F1 points for ViT/Swin families | Most impactful optimization; elevates Transformers from bottom-tier to competitive performance |
| Learning Rate | (universal across all families) | +10.2 F1 points on average | Rates consistently resulted in poorer performance |
| Input Resolution | to | +14.2% F1 for DS1–DS3 | Higher resolutions show diminishing returns with increased cost |
| Data Augmentation | Standard (moderate geometric transforms) | +2.4 F1 vs. none; F1 with heavy augmentation | Aggressive augmentation degrades fine-grained damage discrimination |
| Loss Function | Architecture-dependent tuning | Mixed effects (48% improved; 49% degraded with Focal Loss) | Cross-entropy remains the most robust baseline on the QSTD benchmark |
| Pre-training Source | ImageNet (default); Places365 for AlexNet, DenseNet161, ResNet18 | Up to +142% F1 (zero-shot) | Scene-centric pre-training benefits certain architectures |
| Deployment | Recommended | Optimizer | Learning | Input | Aug- | Performance | Key |
| Scenario | Model | Rate | Resolution | mentation | Metrics | Advantages | |
| Resource-Constrained (Edge devices, mobile platforms) | MobileNetV3-Large or ShuffleNetV2-x1.5 | SGD (momentum=0.9) | Standard | 62.0% F1, 4.2M params, 5ms inference | Optimal accuracy-efficiency trade-off for limited compute | ||
| Maximum Accuracy (GPU-enabled backend) | ConvNeXt-Base | SGD (momentum=0.9) | Standard | 68.0% F1 (QSTD), 46.4% F1 (TMTD), 85.5% Ordinal Acc (TMTD) | Highest accuracy with strong cross-event generalization |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Class | Initial | Final | Net Change | Primary Adjustment |
| Annotation | Dataset | |||
| DS0 - Undamaged | 3,744 | 3,375 | ( ) | Reclassified to DS1 upon detection of subtle roof/window damage |
| DS1 - Slight | 480 | 606 | ( ) | Received reclassifications from DS0 and DS2 |
| DS2 - Moderate | 512 | 432 | ( ) | Many reclassified to DS3 based on roof sheathing criteria |
| DS3 - Extensive | 127 | 368 | ( ) | Significant increase due to consistent application of sheathing failure thresholds |
| DS4 - Complete | 440 | 455 | ( ) | Minor adjustments; clearest damage state with least ambiguity |
| Damage Class | Average Width (px) | Average Height (px) |
|---|---|---|
| DS0 - Undamaged | 347.02 | 225.70 |
| DS1 - Slight | 371.34 | 238.19 |
| DS2 - Moderate | 367.34 | 229.52 |
| DS3 - Extensive | 366.38 | 235.88 |
| DS4 - Complete | 373.15 | 248.80 |
| DS5 - Debris | 1048.47 | 632.23 |
| Strategy | Transformations | Key Parameters | Context |
|---|---|---|---|
| None | – | – | Provides a non-augmented baseline; used for validation and test sets. |
| Basic | Horizontal flip | Basic invariance to viewpoint. | |
| (Base Case) | Resize | Target input resolution | |
| Standard | Horizontal flip | Simulates moderate changes in camera angle and framing | |
| Random rotation | |||
| Random crop | From padded input |
| # | Architecture | Number of | Models |
| Variants | |||
| 1 | AlexNet | 2 | AlexNet, AlexNet-Places365 |
| 2 | ConvNeXt | 4 | ConvNeXt-Tiny, ConvNeXt-Small, ConvNeXt-Base, ConvNeXt-Large |
| 3 | DenseNet | 5 | DenseNet121, DenseNet161, DenseNet161-Places365, DenseNet169, DenseNet201 |
| 4 | EfficientNet | 12 | EfficientNet-B0, EfficientNet-B1, EfficientNet-B1v2*, EfficientNet-B2, EfficientNet-B3, EfficientNet-B4, EfficientNet-B5, EfficientNet-B6, EfficientNet-B7, EfficientNet-v2-s, EfficientNet-v2-m, EfficientNet-v2-l |
| 5 | GoogLeNet | 1 | GoogLeNet |
| Rank | Model | Pre-training | Macro F1 | Ordinal Top-1 | Standard Top-1 | Params | Inference |
|---|---|---|---|---|---|---|---|
| (Val) (%) | Acc (Val) (%) | Acc (Val) (%) | (M) | Time (ms) | |||
| 1 | EfficientNet-B6 | † | 66.0 | 91.5 | 77.7 | 40.75 | 22.89 |
| 2 | EfficientNet-B3 | † | 64.4 | 88.7 | 77.0 | 10.71 | 13.29 |
| 3 | EfficientNet-B4 | † | 63.9 | 89.8 | 77.4 | 17.56 | 16.65 |
| 4 | EfficientNet-B1v2 | † | 63.4 | 89.1 | 75.5 | 6.52 | 11.70 |
| 5 | ConvNeXt-Large | † | 62.8 | 90.4 | 75.2 | 196.21 | 9.34 |
| Metrics (Average) | CCE* | Focal Loss | |
| Macro F1 | 52.9 | 52.4 | |
| Standard Top-1 | 71.7 | 71.8 | 0.0 |
| Ordinal Top-1 | 86.2 | 86.1 | |
| Params (M) | 58.77 | 61.54 | |
| GFLOPs | 8.98 | 9.85 | |
| Inference Time (ms) | 11.16 | 11.42 |
| Architectural Family | Variants | Variants | Avg. F1 Change | Top F1 | Top F1 |
|---|---|---|---|---|---|
| Tested | Improved (%) | (Points) | Gain | Loss | |
| AlexNet | 2 | 0 (0) | — | (AlexNet-Places365) | |
| ConvNeXt | 4 | 0 (0) | — | (ConvNeXt-Base) | |
| DenseNet | 5 | 2 (40) | (DenseNet161) | (DenseNet121) | |
| EfficientNet | 12 | 6 (50) | (EfficientNet-B2) | (EfficientNet-B6) | |
| GoogLeNet | 1 | 0 (0) | — | (GoogLeNet) |
| Metrics (Average) | 224* | 160 | 192 | 256 | 384 | 448 | 512 |
|---|---|---|---|---|---|---|---|
| Macro F1 | 52.9% | 53.0% | 52.7% | 52.8% | 53.5% | 53.9% | 52.7% |
| Standard Top-1 | 71.7% | 72.1% | 71.8% | 72.4% | 72.4% | 72.3% | 72.1% |
| Ordinal Top-1 | 86.2% | 86.4% | 86.4% | 86.5% | 86.5% | 86.9% | 86.1% |
| Params (M) | 58.77 | 52.57 | 52.57 | 52.57 | 52.57 | 52.57 | 52.57 |
| GFLOPs | 8.98 | 4.17 | 6.00 | 10.66 | 23.96 | 32.61 | 42.59 |
| Inference Time (ms) | 11.16 | 11.11 | 11.19 | 11.27 | 11.46 | 11.66 | 11.88 |
| Augmentation Strategy | None | Basic* | Standard | Advanced | Heavy |
|---|---|---|---|---|---|
| Avg. Macro F1 | 51.6% | 52.9% | 54.0% | 50.2% | 46.8% |
| Avg. Standard Top-1 | 71.3% | 71.7% | 72.5% | 71.9% | 70.7% |
| Avg. Ordinal Top-1 | 85.7% | 86.2% | 86.4% | 85.8% | 84.6% |
| Avg. Params (M) | 58.78 | 58.77 | 58.77 | 58.78 | 61.54 |
| Avg. GFLOPs | 8.98 | 8.98 | 8.98 | 8.98 | 9.85 |
| Avg. Inference Time (ms) | 11.23 | 11.16 | 11.23 | 11.21 | 11.43 |
| Metrics (Average) | Adam* | SGD | |
| Macro F1 | 52.9% | 59.8% | |
| Standard Top-1 | 71.7% | 74.6% | |
| Ordinal Top-1 | 86.2% | 89.3% | |
| Params (M) | 58.77 | 61.28 | |
| GFLOPs | 8.98 | 9.85 | |
| Inference Time (ms) | 11.16 | 11.42 |
| Metrics (Average) | * | |||
|---|---|---|---|---|
| Macro F1 | 52.9% | 63.1% | 43.6% | 40.9% |
| Standard Top-1 | 71.7% | 76.2% | 69.1% | 68.3% |
| Ordinal Top-1 | 86.2% | 90.3% | 83.8% | 82.5% |
| Params (M) | 58.77 | 61.28 | 61.28 | 61.28 |
| GFLOPs | 8.98 | 9.85 | 9.85 | 9.85 |
| Inference Time (ms) | 11.16 | 11.43 | 11.40 | 11.41 |
| Metrics (Average) | None* | Cosine | Step |
|---|---|---|---|
| Macro F1 | 52.9% | 52.8% | 53.8% |
| Standard Top-1 | 71.7% | 71.8% | 73.2% |
| Ordinal Top-1 | 86.2% | 86.0% | 86.9% |
| Params (M) | 58.77 | 61.53 | 61.53 |
| GFLOPs | 8.98 | 9.85 | 9.85 |
| Inference Time (ms) | 11.16 | 11.46 | 11.39 |
| Metrics (Average) | 64* | 32 | 16 | 8 | 4 |
|---|---|---|---|---|---|
| Macro F1 | 52.9% | 51.0% | 48.5% | 45.7% | 43.3% |
| Standard Top-1 | 71.7% | 71.4% | 70.8% | 70.1% | 69.2% |
| Ordinal Top-1 | 86.2% | 85.4% | 84.8% | 84.1% | 82.8% |
| Params (M) | 58.77 | 61.28 | 61.28 | 61.28 | 61.28 |
| GFLOPs | 8.98 | 9.85 | 9.85 | 9.85 | 9.85 |
| Inference Time (ms) | 11.16 | 11.40 | 11.37 | 11.41 | 11.37 |
| Metrics (Average) | 0.0* | 0.2 | 0.3 | 0.5 |
|---|---|---|---|---|
| Macro F1 | 52.9% | 52.5% | 52.7% | 52.1% |
| Standard Top-1 | 71.7% | 71.8% | 71.8% | 71.4% |
| Ordinal Top-1 | 86.2% | 86.2% | 86.2% | 85.8% |
| Params (M) | 58.77 | 61.28 | 61.28 | 61.28 |
| GFLOPs | 8.98 | 9.85 | 9.85 | 9.85 |
| Inference Time (ms) | 11.16 | 11.71 | 11.69 | 11.63 |
| Metrics (Average) | Default* | ReLU | Leaky ReLU | ELU | SiLU | GELU |
|---|---|---|---|---|---|---|
| Macro F1 | 52.9% | 50.5% | 49.6% | 44.5% | 46.5% | 48.7% |
| Standard Top-1 | 71.7% | 70.5% | 70.3% | 69.2% | 69.8% | 70.5% |
| Ordinal Top-1 | 86.2% | 85.2% | 84.5% | 83.2% | 83.8% | 84.4% |
| Params (M) | 58.77 | 61.28 | 61.28 | 61.28 | 61.28 | 61.28 |
| GFLOPs | 8.98 | 9.85 | 9.85 | 9.85 | 9.85 | 9.85 |
| Inference Time (ms) | 11.16 | 11.60 | 11.60 | 11.61 | 11.52 | 11.57 |