When Does Blockwise Additivity Break? Diagnosing and Repairing the Additive Assumption in Distillation-Based Architecture Search
Organizations: Niigata University
Abstract
Blockwise distillation methods for architecture search, such as DNA, DONNA, LANA and Puzzle, score each candidate replacement block in isolation and combine the scores additively. We ask at which budgets this additivity assumption fails, whether fine-tuning repairs the failure, what diagnoses it, and how it can be repaired cheaply. On DeiT-S, DeiT-B, Swin-T and Qwen2.5-0.5B with seven latency budgets each, an additive integer program trails a pairwise interaction-aware program after fine-tuning by at most 1.6 top-1 points on ImageNet-1k at the mildest budgets, but by up to 17.5 points, and a factor of 1.9 in perplexity, at aggressive ones. After 10,000 fine-tuning steps 3.3-9.9 points remain on DeiT, whereas on Swin-T the gap shrinks to at most about 1.8 points within 500 steps. In a learning-rate sweep with the standard 2,000 steps, 6.0-10.7 points remain on DeiT when each method's rate is chosen by its evaluation-set accuracy. An intervention that keeps the block library and only forbids more than two removals in any three adjacent layers shrinks the DeiT gap from 5.8-17.5 to 0.7-5.2 points, mostly by improving the additive solution: adjacent removals are a major failure pattern. Splitting the additivity error into a metric and a functional part, relative to additive logit changes, indicates where a cheap logit-sum predictor can be trusted. A hybrid search that ranks moves with this predictor and measures only the best needs no pair measurements and, after Holm adjustment, is significantly better than the pairwise program in 10 of 14 ImageNet-1k settings, with no statistically significant losses. An ablation shows that its ranking helps except at the most aggressive DeiT-B budget, where the predictor misleads and random ranking or ten-times-costlier evolution does better.
Figures & tables
| model | Kendall (sum) | (+ pair terms) | super-additive | median act./add | MAE | MAE |
|---|---|---|---|---|---|---|
| DeiT-S | 0.894 | 0.952 | 90% | 1.16 | 0.027 | 0.228 |
| DeiT-B | 0.894 | 0.956 | 79% | 1.09 | 0.003 | 0.246 |
| budget | DeiT-S val100 | DeiT-B val100 | DeiT-S val1k | DeiT-B val1k |
|---|---|---|---|---|
| 70% | [ , ] ∗ | [ , ] | ∗ | ∗ |
| 65% | [ , ] ∗ | [ , ] ∗ | ∗ | ∗ |
| 60% | [ , ] | [ , ] ∗ | ∗ | ∗ |
| 55% | [ , ] ∗ | [ , ] ∗ | ∗ | ∗ |
| 50% | [ , ] ∗ | [ , ] ∗ | ∗ | ∗ |
| 45% | [ , ] ∗ | [ , ] ∗ | ∗ | ∗ |
| model | budget | ILP | lin | IQP | hybrid | hybrid IQP |
|---|---|---|---|---|---|---|
| DeiT-S | 70% | 93.59 | 94.47 | 94.36 | 94.27 | [ , ] |
| 65% | 93.11 | 93.11 | 93.80 | 93.93 | [ , ] | |
| 60% | 92.27 | 93.16 | 92.61 | 93.11 | [ , ] ∗ | |
| 55% | 88.99 | 92.85 | 92.56 | 92.75 | [ , ] | |
| 50% | 87.42 | 90.62 | 90.75 | 92.49 | [ , ] ∗ | |
| 45% | 85.32 | 90.27 | 88.77 | 90.05 | [ , ] ∗ |
| model | budget | ILP | IQP | lin | random | evolution | hybrid | IQP ILP [95% CI] |
|---|---|---|---|---|---|---|---|---|
| DeiT-S | 70% | 74.28 | 75.45 | 74.93 | 74.89 | 74.89 | 75.77 | [ , ] |
| 65% | 73.34 | 74.23 | 73.29 | 73.05 | 73.50 | 74.53 | [ , ] | |
| 60% | 72.83 | 72.04 | 72.88 | 72.83 | 72.83 | 73.21 | [ , ] | |
| 55% | 63.21 | 70.24 | 71.41 | 71.41 | 71.41 | 71.30 | [ , ] | |
| 50% | 58.19 | 67.85 | 69.26 | 69.26 | 69.26 | 69.34 | [ , ] | |
| 45% | 56.52 | 63.06 | 65.85 | 65.85 | 65.85 | 66.27 | [ , ] |
| before fine-tuning | after fine-tuning | |||||||
|---|---|---|---|---|---|---|---|---|
| budget | ILP | IQP | ILP | IQP | lin | random / evol. | hybrid | IQP ILP [95% CI] |
| 70% | 78.64 | 79.13 | 80.36 | 80.25 | 80.26 | 80.26 / 80.26 | 80.32 | [ , ] |
| 65% | 78.08 | 77.99 | 80.32 | 79.79 | 80.30 | 80.30 / 80.30 | 80.44 | [ , ] ∗ |
| 60% | 76.88 | 77.54 | 79.98 | 80.04 | 79.98 | 79.98 / 79.98 | 80.09 | [ , ] |
| 55% | 75.32 | 75.88 | 79.45 | 79.48 | 79.45 | 79.45 / 79.35 | 79.14 | [ , ] |
| 50% | 68.99 | 74.25 | 77.92 | 78.45 | 78.25 | 78.25 / 78.25 | 78.46 | [ , ] ∗ |
| zero-shot accuracy | perplexity | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| budget | ILP | IQP | lin | hybrid | IQP ILP [95% CI] | ILP | IQP | lin | hybrid |
| 90% | 52.96 | 52.81 | 53.49 | 53.83 | [ , ] | 18.8 | 18.7 | 19.2 | 19.1 |
| 85% | 52.17 | 52.95 | 53.21 | 53.23 | [ , ] ∗ | 19.8 | 19.5 | 20.6 | 20.0 |
| 80% | 51.65 | 51.67 | 52.49 | 51.72 | [ , ] | 21.7 | 20.9 | 22.1 | 21.6 |
| 75% | 50.07 | 50.33 | 50.78 | 50.93 | [ , ] | 23.4 | 23.4 | 24.2 | 23.0 |
| 70% | 49.01 | 48.67 | 48.17 | 49.74 | [ , ] | 27.6 | 24.2 | 29.4 | 25.1 |
| unconstrained | same option | removal class | removal class, | ||||||
|---|---|---|---|---|---|---|---|---|---|
| model | budget | gap | run | ILP | IQP | ||||
| DeiT-S | 55% | 4 / 3 | 70.21 | 70.95 | |||||
| 50% | 5 / 3 | 65.48 | 66.61 | ||||||
| 45% | 3 / 4 | 59.87 | 61.94 | ||||||
| 40% | 4 / 5 | 54.91 | 58.94 | ||||||
| DeiT-B | 55% | 5 / 4 | 73.69 | 75.51 | |||||
| budget | step | DeiT-S | DeiT-B | Swin-T |
|---|---|---|---|---|
| 50% | 0 | ∗ | ∗ | ∗ |
| 500 | ∗ | ∗ | ∗ | |
| 2,000 | ∗ | ∗ | ∗ | |
| 10,000 | [ , ] ∗ | [ , ] ∗ | [ , ] ∗ | |
| top-1 | 66.64 / 71.28 / 72.34 | 71.88 / 76.55 / 76.34 | 78.77 / 79.06 | |
| hyb IQP | ∗ | – |
| budget | ILP | IQP | hybrid | IQP ILP | hyb IQP | main | |
|---|---|---|---|---|---|---|---|
| S | 50% | 62.40 (2) | 68.60 (2) | 69.85 (1) | ∗ | ∗ | |
| 40% | 51.72 (4) | 62.41 (2) | 63.92 (2) | ∗ | ∗ | ||
| B | 50% | 68.72 (2) | 74.68 (1) | 74.39 (1) | ∗ | ∗ | |
| 40% | 56.43 (2) | 65.91 (2) | 67.49 (2) | ∗ | ∗ |
| model | set | add | lin | act add | metric | share |
|---|---|---|---|---|---|---|
| DeiT-S | pairs, KL | 0.924 | 0.931 | 0.0171 | 0.0102 | 59% |
| struct., KL | 0.894 | 0.911 | 0.0694 | 0.0454 | 65% | |
| pairs, acc. | 0.811 | 0.875 | 0.255 | 0.094 | 37% | |
| struct., acc. | 0.825 | 0.897 | 1.256 | 0.446 | 36% | |
| DeiT-B | pairs, KL | 0.934 | 0.952 | 0.0143 | 0.0038 | 27% |
| struct., KL | 0.894 | 0.932 | 0.1095 | 0.0267 | 24% |
| model | search | whole block | attention | longest run |
|---|---|---|---|---|
| DeiT-S | ILP | 49–75% | 6–33% | 2–3 |
| IQP | 55–66% | 7–13% | 1–2 | |
| DeiT-B | ILP | 82–92% | 0% | 5–6 |
| IQP | 50–69% | 8–14% | 1–3 | |
| Swin-T | ILP | 0–18% | 78–98% | 0–1 |
| IQP | 0–35% | 58–93% | 0–1 |
| DeiT-S | DeiT-B | ||||||
| set | budget | add | pair | lin | add | pair | lin |
| band | 70% | 0.66 | 0.80 | 0.73 | 0.74 | 0.83 | 0.77 |
| 55% | 0.66 | 0.78 | 0.68 | 0.60 | 0.63 | 0.61 | |
| 40% | 0.63 | 0.68 | 0.67 | 0.53 | 0.63 | 0.57 | |
| near | 70% | 0.64 | 0.81 | 0.49 | 0.69 | 0.85 | 0.74 |
| 65% | 0.53 | 0.81 | 0.46 | 0.29 | 0.59 | 0.41 | |
| 55% | 50% | 45% | 40% | |
| DeiT-S | ||||
| hybrid | 71.30 | 69.34 | 66.27 | 62.27 |
| lin, -lin | ||||
| add | ||||
| add, -lin | ||||
| random | ||||
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| DeiT-S | DeiT-B | |||||
|---|---|---|---|---|---|---|
| budget | ILP | IQP | hybrid | ILP | IQP | hybrid |
| 70% | 73.5% | 73.0% | 73.0% | 71.1% | 72.0% | 71.9% |
| 65% | 68.2% | 67.9% | 68.8% | 66.6% | 67.2% | 66.6% |
| 60% | 64.1% | 64.0% | 64.2% | 61.8% | 62.8% | 62.0% |
| 55% | 59.0% | 59.4% | 59.5% | 56.9% | 57.3% | 57.3% |
| 50% | 54.4% | 54.6% | 54.8% | 51.6% | 52.2% | 52.8% |
| model | IQP pair stage | hybrid | ratio |
|---|---|---|---|
| DeiT-S | 6.95 M | 0.93 M | 13.4% |
| DeiT-B | 6.95 M | 1.84 M | 26.5% |
| DeiT-S val100 | DeiT-B val100 | |||
|---|---|---|---|---|
| budget | random | evolution | random | evolution |
| 70% | ∗ | ∗ | ||
| 65% | ||||
| 60% | ||||
| 55% | ||||
| 50% | ∗ | ∗ | ∗ | |
| budget | ILP | IQP | IQP ILP [95% CI] |
|---|---|---|---|
| 70% | 95.66 | 95.99 | [ , ] ∗ |
| 65% | 95.03 | 95.62 | [ , ] ∗ |
| 60% | 94.65 | 95.33 | [ , ] ∗ |
| vs. ILP | vs. IQP | vs. hybrid | vs. evolution | vs. random | |
|---|---|---|---|---|---|
| val100 | 10/4/0 | 6/8/0 | 1/13/0 | 1/13/0 | 3/11/0 |
| val1k | 13/0/1 | 4/4/6 | 1/4/9 | 10/0/4 | 8/0/6 |
| DeiT-S | DeiT-B | |||
| score | select hyb | select hyb | ||
| calib KL, 0 steps (used so far) | ( ) | ( ) | ||
| calib KL, 50–500 steps | to | to | to | to |
| calib top-1, 0 steps | ( ) | ( ) | ||
| calib top-1, 50–500 steps | to | to | to | to |
| calib top-1, 200 steps (fixed) | ( ) | ( ) | ||
| vs. ILP | vs. IQP | vs. lin | vs. random | vs. evolution | |
|---|---|---|---|---|---|
| DeiT-S | 7/0/0 | 6/1/0 | 4/3/0 | 5/2/0 | 4/3/0 |
| DeiT-B | 7/0/0 | 4/3/0 | 5/1/1 | 6/1/0 | 5/1/1 |
| total (14) | 14/0/0 | 10/4/0 | 9/4/1 | 11/3/0 | 9/4/1 |
| unadjusted | 14/0/0 | 13/1/0 | 10/3/1 | 11/3/0 | 10/3/1 |
| orig. | without block_skip | |||||
|---|---|---|---|---|---|---|
| model | budget | gap | ILP | IQP | gap [95% CI] | run |
| S | 55% | 62.76 | 69.80 | [ , ] ∗ | 3 / 2 | |
| 50% | 63.48 | 65.90 | [ , ] ∗ | 4 / 2 | ||
| 45% | 53.47 | 59.30 | [ , ] ∗ | 9 / 6 | ||
| B | 55% | 69.22 | 74.57 | [ , ] ∗ | 7 / 3 | |
| 50% | 65.06 | 70.11 | [ , ] ∗ | 7 / 5 | ||
| budget | IQP | hyb. | seed 0 | seed 1 | seed 2 | evo. hyb. [95% CI] |
|---|---|---|---|---|---|---|
| 50% | 74.45 | 74.29 | 74.10 | 75.16 | 74.43 | [ , ] ∗ |
| 45% | 71.05 | 72.13 | 71.83 | 72.90 | 72.17 | [ , ] |
| 40% | 63.83 | 65.54 | 69.40 | 69.66 | 67.79 | [ , ] ∗ |
| model | budget | IQP | hybrid | random | evolution | random | evolution |
|---|---|---|---|---|---|---|---|
| DeiT-S | 70% | 73.40 | 73.73 | ||||
| 65% | 70.92 | 71.57 | |||||
| 60% | 66.23 | 69.10 | |||||
| 55% | 62.45 | 66.56 | |||||
| 50% | 58.15 | 62.35 | |||||
| 45% | 47.67 | 56.26 |
Neural Architecture Search
Neural Architecture Search automates neural network design for target tasks, resource budgets, and hardware constraints.
See all 138 papers