Size-Matched Nulls Reveal What Block Ablation Measures in MambaVision
Organizations: Department of Computer Science, COMSATS University Islamabad, Islamabad, Pakistan
Abstract
Hybrid vision backbones interleave Mamba state-space mixers with self-attention, and a block's role is usually probed by removing it and measuring the damage. We test what that measurement means in MambaVision. On all 50,000 ImageNet validation images and three model sizes, every block ablation is paired with a size-matched null, the same perturbation with its channels shuffled or their signs flipped. In top-1 accuracy and in true-class rank, real and null damage are nearly indistinguishable in absolute terms: pooled over the blocks of MambaVision-T, the content effect lies within [-0.044,+0.016] percentage points (90% interval). Two attention-only controls show that the test detects content, with real ablations worse than their nulls by up to 4.2 points in DeiT-Small and 1.0 point in Swin-T. Part of MambaVision's near-equivalence reflects how little damage its blocks do (0.11--0.34 points on average, against 1.7 and 2.1). Scaled by damage, its content share is comparable to Swin-T's in accuracy, lower in rank, and far below DeiT-Small's: single-block damage in MambaVision is small and mostly positional. Ablation also hides what blocks carry together: removing all blocks of one type compounds damage to 3.7--8.4 times the sum of single effects, far more than size-matched nulls, including nulls that keep the alignment between the blocks' perturbations; for adjacent pairs, aligned nulls reproduce most of it. Freezing input-dependent selection in all Mamba mixers shows that it accounts for 20--32% of the Mamba contribution, mainly through the B and C projections. We turn these results into a protocol for causal attribution in hybrid networks.
Figures & tables
| accuracy (pp) | log-prob. (nats) | rank ( ) | |||||
|---|---|---|---|---|---|---|---|
| Block | type | shuffled | sign-flip | shuffled | sign-flip | shuffled | sign-flip |
| L2B0 | M | 0.081 [ 0.024, 0.186] | 0.037 [ 0.075, 0.154] | 0.041 ∗ | 0.045 ∗ | 1 | 1 |
| L2B1 | M | 0.023 [ 0.111, 0.068] | 0.053 [ 0.149, 0.041] | 0.017 ∗ | 0.021 ∗ | 0 | 1 |
| L2B2 | M | 0.007 [ 0.090, 0.071] | 0.039 [ 0.051, 0.128] | 0.019 ∗ | 0.022 ∗ | 0 | 1 |
| L2B3 | M | 0.041 [ 0.122, 0.041] | 0.027 [ 0.119, 0.067] | 0.012 ∗ | 0.009 ∗ | 0 | 0 |
| L2B4 | A | 0.163 [ 0.305, 0.003] | 0.213 [ 0.375, 0.039] | 0.030 ∗ | 0.037 ∗ | 0 | 1 |
| Frozen | mode | acc. (pp) [95% CI] | share (%) | log-prob. drop |
|---|---|---|---|---|
| mean | 1.136 [ 1.328, 0.954] | 29.1 [24.8, 33.5] | 0.114 | |
| position | 0.790 [ 0.958, 0.628] | 20.3 [16.4, 24.3] | 0.116 | |
| mean | 0.540 [ 0.692, 0.390] | 13.8 [10.2, 17.5] | 0.039 | |
| position | 0.152 [ 0.274, 0.034] | 3.9 [0.9, 6.9] | 0.054 | |
| mean | 0.274 [ 0.396, 0.146] | 7.0 [3.8, 10.1] | 0.079 | |
| position | 0.250 [ 0.370, 0.132] | 6.4 [3.5, 9.4] | 0.054 |
| Frozen | mode | copy acc. % | loss 1st | loss 2nd | benefit |
|---|---|---|---|---|---|
| none | – | 19.06 | 12.75 | 6.94 | 5.81 |
| mean | 0.00 | 12.81 | 12.42 | 0.39 | |
| position | 0.00 | 12.71 | 12.47 | 0.23 | |
| mean | 0.00 | 12.60 | 12.14 | 0.46 | |
| position | 0.00 | 12.59 | 12.22 | 0.37 | |
| mean | 0.00 | 13.21 | 12.71 | 0.50 |
| Step | Worked example: L2B4 | |
|---|---|---|
| 1 | Build the reference set from the input distribution under test and record how it was built. Recompute it when the resolution or input distribution changes, and re-score effects on images outside it. | Means over the first 256 images of the fixed order. Excluding all reference images changes no effect by more than 0.024 pp. At 384 px, reusing 224 px means would have hidden a pooled content effect of pp. |
| 2 | Ablate, and test the effect against the unmodified model with a paired test on the same images. | Mean ablation costs 1.13 pp (McNemar ), the largest single-block drop in the model. |
| 3 | Run a size-matched null with several permutations, and a second kind of null, on the same images. Report with a paired bootstrap interval and correct over blocks. | Shuffled null (ten permutations): pp, so pp . Sign-flip null: pp, pp . Neither survives Holm correction. The null damages L2B4 at least as much as its real ablation. |
| 4 | Relate damage to perturbation size to see whether a position is fragile. | Across the attention positions, null damage grows by 5.4 pp per unit of relative perturbation size ( ). L2B4 receives the largest perturbation of all blocks (0.30 of the residual norm) and shows the largest damage. Its ranking reflects position, not content. |
| 5 | Pool over blocks and test equivalence against a margin fixed in advance and anchored to the null’s own spread, and scale content effects by damage , so that small damage is not mistaken for absent content. | Pooled over the 12 blocks, the 90% interval pp lies inside pp; the null of L2B4 alone varies by 0.20 pp across permutations. Relative to damage, the content share of the model is , comparable to Swin-T’s. |
| 6 | Report accuracy, log-probability and true-class rank , and read disagreement between them as a change of confidence. | Log-probability: the null lowers more ( and nats). Rank: real 0.0238, nulls 0.0235 and 0.0229. The ordering of classes changes equally under real and null. |
Neural Network Interpretability
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
See all 364 papers