Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA
Organizations: Independent researcher, Zug, Switzerland
Abstract
Untrained convolutional networks have been reported to rival trained networks as models of visual cortex, and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference -0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r. In exploratory analyses, across 36 combinations of unit normalisation, distance measure and voxel normalisation (28 distinct analyses), the sign of the V1 contrast depended on the analysis (intervals above 0 in 5 and below 0 in 9 distinct analyses), and in our networks the resolution dependence of the original convention appeared in only one cell. Raw correlation-distance representational similarity analysis (RSA) covaried with the alignment of model and voxel patterns with the unit-mean profile (Spearman 0.97 across our networks, 0.79 across six ImageNet architectures); the advantage was absent from the component orthogonal to the mean profile, and the parallel component alone did not resolve it. On the unweighted angular component of centred patterns, trained networks were ahead at V1 (our network and 5 of 6 ImageNet models) and at LOC (all seven); the norm-weighted angular term was not resolved. A luminance-and-contrast bank scored above every network under ridge (point estimates), and adding network features under a shared penalty did not improve prediction. Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.
Figures & tables
| Statement | Status | Uncertainty reported | Main caveat |
|---|---|---|---|
| Untrained below Backprop under ridge at V1, also in MSE skill and per-fold (Section 3.1 ) | Pre-specified; verdict “effect disappears”; robustness checks exploratory (F-R1 to F-R3 met) | 5 seeds; one-sided sign-flip test; no interval over stimuli, voxels or subjects | One layer; sub-03 near zero; low absolute scores (median 0.086; MSE skill 0.0111); not blind |
| V1 ranking depends on the analysis (Section 3.3 ) | Exploratory; GEN2 met; cell counts descriptive | Stimulus-bootstrap 95% intervals per cell, uncorrected; cells dependent | 36 chosen cells, 28 distinct analyses; three subjects |
| Advantage absent from the component orthogonal to the mean profile (Section 3.5 ) | Exploratory; M1, M2, M4 met; M3 not met | Stimulus-bootstrap intervals | Parallel component alone unresolved; wide interval; paired contrast of the one-sided ablations not tested; paired full-minus-orthogonal contrast resolved (X1) |
| Raw RSA covaries with mean-profile alignment (Section 3.4 ) | Exploratory; G1, GEN1 met; G2 not met | Spearman over 70 and 64 cells, no interval; cells not independent | Correlational; the ablation tests only the orthogonal remainder |
| Trained ahead on the unweighted angular component at V1 and LOC (Section 3.6 ) | Exploratory; SEP3, SEP4 met; SEP2 not met | Stimulus-bootstrap intervals; model counts descriptive | Weighted term not resolved; one weight set per ImageNet model; convnext_tiny exception |
| Resolution slope of own CNN vanishes under centring or standardisation (Sections 3.2 , 3.3 ) | Exploratory; R3, O2, P_ridge met | Stimulus-bootstrap intervals; seed SD for ridge | Rests on three rules together; ImageNet slope remains (GEN4 not met) |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Analysis | Hits | Plateau | Excluded | Max. gain |
|---|---|---|---|---|
| Pilot diagnosis (sub-01, ) | 4 | 4 | 0 | |
| RSA versus ridge: collapse, random folds (sub-01) | 3 | 3 | 0 | |
| RSA versus ridge: run-grouped folds (sub-01) | 4 | 4 | 0 | |
| RSA versus ridge: encoding-RSA (sub-01; repeats the hits above) | 7 | 7 | 0 | |
| Other subjects, (sub-02, sub-03) | 1 | 1 | 0 | |
| Diagnosis of sub-03 (controls, ) | 3 | 3 | 0 |
| Pooled , random folds | Pooled , run folds | Per-fold , run folds | |
|---|---|---|---|
| 1 | (0/5) | (0/5) | (0/5) |
| 4 | (0/5) | (0/5) | (0/5) |
| 8 | (0/5) | (0/5) | (0/5) |
| Folds ( ) | Centred | Uncentred |
|---|---|---|
| Random folds | (2/5) | (3/5) |
| Run-grouped folds | (0/5) | (2/5) |
| Only 0–14 | Notch 0–14 | |
|---|---|---|
| V1, raw | ||
| V1, z | ||
| LOC, raw | ||
| LOC, z |
| Input | , untrained | , Backprop |
|---|---|---|
| Native 224 px | 0.707 | 0.638 |
| Only 0–14 | 0.619 | 0.484 |
| Notch 0–14 | 0.832 | 0.830 |
| Prediction | Rule and threshold | Result | Section |
|---|---|---|---|
| P_mech2 | in 5 of 5 seeds, and Spearman over 50 cells | 5 of 5 seeds; Spearman 0.27 | 3.2 |
| G2 | partial V1 gap at NATIVE 224 px with upper CI bound , and | gap [ , ]; [ , ] | 3.4 |
| GEN3 | LOC trained untrained at 224 px with CI above 0 in all 4 bootstrap cells, for of 6 models | 0 models; the raw/euclidean cell includes 0 for every model | 3.3 |
| GEN4 | for every model whose (raw, correlation) has a CI excluding 0: | 5 qualifying models, none within the bound | 3.3 |
| SEP2 | V1 norm contrast ( = center) with CI above 0 in the own CNN and in of 6 ImageNet models | own CNN [ , ]; ImageNet 4 of 6 | 3.6 |
| M3 | parallel component: V1 gap at NATIVE 224 px with CI above 0, and resolution slope with CI above 0 | gap [ , ]; slope [ , ] | 3.5 |
| sub-01 | sub-02 | sub-03 | |
|---|---|---|---|
| 1 | (0/5) | (0/5) | (0/5) |
| 2 | (0/5) | (0/5) | (0/5) |
| 4 | (0/5) | (0/5) | (0/5) |
| 8 | (0/5) | (0/5) | (0/5) |
| sub-01 | sub-02 | sub-03 | |
|---|---|---|---|
| 1 | 0.118/ 0.094 | 0.102/ 0.096 | 0.023/ 0.002 |
| 2 | 0.129/ 0.104 | 0.106/ 0.104 | 0.027/ 0.002 |
| 4 | 0.155/ 0.123 | 0.132/ 0.122 | 0.030/ 0.013 |
| 8 | 0.161/ 0.126 | 0.135/ 0.122 | 0.035/ 0.014 |
| MSE skill | per-fold | |
|---|---|---|
| 1 | (0/5) | (0/5) |
| 2 | (0/5) | (0/5) |
| 4 | (0/5) | (0/5) |
| 8 | (0/5) | (0/5) |
| Subject | Condition | Null mean [2.5%, 97.5%] | Observed | |
|---|---|---|---|---|
| sub-01 | 1 | untrained | [ , ] | |
| sub-01 | 1 | Backprop | [ , ] | |
| sub-01 | 4 | untrained | [ , ] | |
| sub-01 | 4 | Backprop | [ , ] | |
| sub-02 | 1 | untrained | [ , ] | |
| sub-02 | 1 | Backprop | [ , ] |
| Network | Bank alone | Network alone | Network and bank | |
|---|---|---|---|---|
| 1 | untrained | |||
| 1 | Backprop | |||
| 4 | untrained | |||
| 4 | Backprop |
| Cell ( , , ) | Value | Fixed (1000) | Re-estimated (200) | Interval |
|---|---|---|---|---|
| , correlation, paper | [ , ] | [ , ] | below 0 | |
| center, euclidean, paper | [ , ] | [ , ] | above 0 | |
| center, correlation, center720 | [ , ] | [ , ] | includes 0 |
| Model | Model RDM | Brain RDM | V1 contrast [95% CI] |
| own CNN | full | full | [ , ] |
| own CNN | full | orthogonal | [ , ] |
| own CNN | orthogonal | full | [ , ] |
| own CNN | orthogonal | orthogonal | [ , ] |
| alexnet | orthogonal | full | [ , ] |
| vgg11_bn | orthogonal | full | [ , ] |
Representational Similarity Analysis
Representational Similarity Analysis (RSA) compares the structure of neural representations across models, brain recordings, tasks, and modalities.
See all 42 papers