Connectome-constrained neural networks make biological components directly accessible to intervention, but their mechanistic conclusions can still depend on the comparison model, intervention, circuit representation and trained solution. We stress-test these dependencies in flyvis, a Drosophila motion-vision model, using its 50-model ensemble, task-matched causal CNN controls and replicated interventions. At the level of model behaviour, three-level reverse-phi reversal in flyvis is strongly period- and velocity-dependent and also appears in task-matched CNNs; neither family reproduces the fly's stationary-sawtooth response. Adversarial direction reversal requires a perturbation bound of 0.21--0.27 times stimulus contrast in flyvis versus 0.13--0.25 in CNNs, but the gap over a parameter-matched CNN largely closes for larger CNNs. At the level of mechanism, matching attribution to the intervention matters: for held-out-mean replacement, first-order edge attribution patching reaches pooled AUROC 0.970 and Spearman 0.872, versus 0.933 and 0.126 when its zero-aligned score is evaluated against the same ground truth. Circuit conclusions also vary with representation and ensemble member. Greedy cell-type sets meet a model-relative T4/T5 tuning criterion in 25 of 50 models, while T4/T5 output silencing reduces the tested optic-flow behaviour to at most two of eight clips in 40. In six models where the cell-type search first succeeds, the corresponding connection sets can be pruned to 13--31 added classes; under an additional two-sided response-amplitude check, only two of those six first-success cell-type sets still meet the tuning threshold. These results support a narrower use of connectome-constrained models: they make mechanistic hypotheses directly testable, but the resulting claims must be scoped to the model family, intervention, representation and trained solution.
Figures & tables
model
parameters
n
common-eval. ℓ2 , mean (range)
clean passes
flyvis 000–009
734 + 7,427
10
1132.7 (1103.5–1159.8)
5 each
CNN small
8,161
5
1098.7 (1078.4–1114.7)
5, 4, 5, 3, 4
CNN medium
50,002
5
1040.6 (1023.9–1053.9)
5 each
CNN large
988,418
5
1028.8 (1013.0–1045.7)
5 each
Table 1: Models used in the attack and detailed circuit experiments. Common-evaluation ℓ2 norm is the equal-weight mean over the original 16-subsequence Sintel evaluation set with augmentation disabled; full preprocessing and reduction details are in Appendix C . The set is scene-held-out relative to CNN training but is not demonstrably held out for released flyvis, so this is a descriptive cross-family comparison rather than a fair shared held-out generalization test. The zero-flow predictor is 1209.6. Clean passes count the five grating contrasts (1 to 0.05) decoded within 45∘ of truth.
Figure 1: Three-level reverse-phi evaluation with a fixed 3.944∘ displacement per update. Solid: fraction of flyvis models meeting the reversal criterion; dashed: fraction passing the matched standard control (legend: “standard sanity”).
ϵ∗/c
attack advantage
contrast c
0.5
0.25
0.1
0.05
0.5
0.25
0.1
0.05
flyvis
0.274 (10)
0.231 (10)
0.234 (10)
0.215 (10)
1.99
2.17
2.17
2.37
CNN small
0.137 (5)
0.126 (5)
0.131 (2)
0.245 (1)
3.66
3.66
4.25
1.83
CNN medium
0.194 (5)
0.212 (5)
0.236 (2)
0.154 (2)
2.18
2.08
1.93
2.40
CNN large
0.194 (5)
0.212 (5)
0.252 (2)
0.245 (2)
2.10
2.18
2.08
1.93
Table 2: Minimal reversing perturbation as a fraction of contrast (median over models and seeds; n in parentheses) and attack advantage ϵnaive/ϵ∗ (median, first two CNN seeds). Contrasts 0.5 and 0.25 pool five CNN seeds; Contrasts 0.1 and 0.05 use seeds 0 and 1 of each size, subject to the same clean-decoding exclusion. Identical entries arise because ϵ∗ lies on the bisection grid.
Figure 2: Adversarial margins. (a) Minimal reversing ℓ∞ perturbation divided by contrast (median and interquartile range). (b) Attack advantage over a naive counter-moving grating; the panel labels ϵ∗ as ϵPGD .
Spearman
method
score semantic → ground truth
B0
B1
AUROC
P@20
IG
zero → zero
0.777
0.614
0.936
0.473
IG
zero → held-out
0.122
0.007
0.918
0.440
IG
held-out → held-out
0.565
0.385
0.947
0.505
IG
zero → counterfactual
0.218
−0.061
0.927
0.453
IG
counterfactual → counterfactual
0.638
0.118
0.948
0.490
Table 3: Attribution against exhaustive single-connection ground truth. Zero → held-out and zero → counterfactual apply the zero-aligned score to a different ground truth; zero → zero and the remaining rows match attribution and intervention semantics. Spearman is the mean full-vector correlation across models 000–009 for each behaviour; AUROC (ground-truth top 20) and precision@20 are means over the 20 model–behaviour pairs.
Figure 3: Attribution agreement averaged over the 20 model–behaviour pairs. (a) Full-vector Spearman and (b) AUROC for the ground-truth top 20, for zero-aligned and intervention-matched IG and first-order EAP. The zero → zero bars show the matched zero-ablation case for reference.
outcome
models
rate (95% CI)
T4/T5 jointly silenced, F≤2
40/50
80.0% (67.0–88.8%)
T4/T5 jointly silenced, F≥6
2/50
4.0% (1.1–13.5%)
anatomical set, Ftun≥6
4/50
8.0% (3.2–18.8%)
anatomy exceeds all 20 matched random draws
35/50
70.0% (56.2–80.9%)
greedy set reaches Ftun≥6 within 15 additions
25/50
50.0% (36.6–63.4%)
Table 4: Cell-type results over all 50 released models, with Wilson 95% intervals.
model
added types
source types retained
added connections
after pruning
total connections
001
4
4
101
13
197
003
4
4
71
14
198
004
9
9
153
26
210
005
6
6
120
18
202
007
9
9
117
31
215
009
9
6
164
15
199
Table 5: Matched T4/T5-criterion results for the six models in which the cell-type search reaches Ftun≥6 . “Source types retained” counts added source types with at least one outgoing connection left after pruning.
49/50; phase-equivalent to same-speed backward motion
not diagnostic ( Appendix B )
Two-level contrast-inverting grating, 17-pixel step: ≥35/50 reverse at one of the three slowest velocities
42–48/50 for the 30∘ period
met; phase-equivalent to translation ( Appendix B )
Timestep 1/50 s changes no decoder verdict
0/10 changed
met
IG/EAP AUROC ≤0.75 on B1 in ≥6/10 models
mean AUROC 0.94–0.95
failed
Misses enriched for inhibitory connections
held-out-matched misses 44.1%/50.0%; held-out top-20 pools about 43–46% inhibitory
not supported
Appendix
Table 6: Directional predictions and analysis gates recorded in scripts before execution. Results over all 50 models use the same criteria as the ten-model results. Three entries (population-vector readout, timestep check, anatomical set without lamina) are reported only here; predictions about analyses outside this paper are omitted.
family
period
197.2
98.6
65.7
39.4
19.7
7.89
flyvis (50)
30∘
34 → 14
48 → 46
48 → 25
48 → 32
48 → 37
42 → 20
flyvis (50)
90∘
7 → 3
48 → 32
48 → 37
39 → 19
26 → 1
18 → 3
CNN small (5)
30∘
2 → 0
4 → 0
5 → 2
4 → 3
4 → 1
1 → 1
CNN small (5)
90∘
0 → 0
5 → 4
5 → 5
5 → 4
3 → 0
0 → 0
CNN medium (5)
30∘
0 → 0
5 → 2
5 → 3
5 → 5
5 → 3
2 → 3
CNN medium (5)
90∘
0 → 0
5 → 5
5 → 5
5 → 0
5 → 0
5 → 2
Appendix
Table 7: Reversal counts for the two-level contrast-inverting grating → the three-level reverse-phi stimulus, by velocity ( ∘ /s). CNN denominators are five checkpoints per architecture.
fly → fly
CNN → CNN (same size)
CNN → CNN (other size)
fly → CNN
CNN → fly
transfer, minimal ϵ
0.148
0.292
0.271
0.109
0.107
transfer, fixed ϵ
0.612
0.500
0.427
0.445
0.249
gradient cosine
0.137
−0.076
0.027
−0.005
−0.005
Appendix
Table 8: Transfer rates and input-gradient cosine similarity, averaged over four conditions.
Figure 4: (a) Transfer and input-gradient similarity by source and target family. (b) Input-gradient cosine similarity between all 56 models, averaged over conditions; rows and columns are ordered flyvis 000–049, then CNNs.
Sawtooth gradient, 45∘ period, 5 s
family
criterion met
met for both polarities
median response ratio
flyvis (50)
3 / 3
0
0.023 / 0.026
CNN small (5)
0 / 0
0
0.014 / 0.019
CNN medium (5)
0 / 0
0
0.022 / 0.016
CNN large (5)
0 / 0
0
0.055 / 0.031
Four-level peripheral-drift sequence
Appendix
Table 9: Stationary patterns. Top: sawtooth gradients ( Agrochao et al., 2020 ) , number of models meeting the direction and magnitude criterion (left-to-right / right-to-left polarity) and median response relative to a moving grating. Bottom: four-level peripheral-drift sequence (black, dark grey, white, light grey; Conway et al., 2005 ), number of models reporting the ordering-predicted direction (forward / reverse ordering) in the sustained window, by spatial period in ommatidia.
Figure 5: Decoder-level connection sufficiency. Mean faithful clips F (of 8) when only k connections are kept, over 10 models and two ranking stimuli. (a) Complement zero-ablated. (b) Complement replaced by same-stimulus clean mean currents; the apparently small sets are leakage-prone and mostly lack an input-to-output path. Legend: “true single-edge ranking” is the ranking by true single-connection effect; “greedy (true ablation)” is greedy forward selection evaluated by direct intervention.
Figure 6: Forward search under the matched T4/T5 tuning criterion in models 000–009. (a) Tuning score after 15 additions for cell-type and single-connection searches. (b) Connection-search scores at 15, 30 and 60 additions; each colored trajectory is one model and overlapping trajectories coincide. The horizontal line marks Ftun≥6 .
transform of canonical layout
gratings
moving edges
identity
190
190
rotation 90∘
53
61
rotation 180∘
92
101
rotation 270∘
65
48
mirror
126
122
mirror, rotation 90∘
57
49
Appendix
Table 10: Subtype–model cases (of 400) within 45∘ of the canonical T4/T5 layout after each frame transform. Mirroring is θ→−θ , applied before the rotation.