While functional magnetic resonance imaging (fMRI) decoders reconstruct images and 3D objects, recovering the metric geometry of rooms from a moving viewpoint remains an open challenge. We introduce, to our knowledge for the first time, the task of reconstructing 3D layouts of held-out scenes using only the known camera trajectories and the fMRI signals of participants watching indoor walkthroughs, without any visual frames. The task is hard because of the temporal mismatch between rapid visual changes and blurred fMRI signals. Our method, DRAFT (Decoding and Reconstructing 3D room lAyouts from fMRI and Trajectories), extracts a compact layout representation from the early visual cortex and fuses this neural code with the geometric free space derived from camera motion. To evaluate it, we propose layout r, the Pearson correlation between predicted and ray-cast log depths. On ten held-out scenes (20 participants), DRAFT reaches a layout r of 0.480, outperforming the camera path alone (0.446) and seven fMRI decoders (at most 0.429). DRAFT also works as a plug-in that improves all seven decoders without retraining, raising their layout r from 0.171-0.429 to 0.487-0.493. Brain activity and camera motion can thus recover the metric layout of unseen rooms, a first step toward reconstructing the 3D spaces people see.
Figures & tables
Figure 1: DRAFT recovers where walls and floor are on a held-out route. (a) Four views of scene 4: frame (not an input), furnished (F) and empty-room (E) ground truth, DRAFT (D), Brain-Diffuser (BD) detail with the DRAFT layout (B), and layout r of D and B against F and E. (b) Merged surface of D from above, blue within 25 cm of F. Hand-picked route, ranked 60th in layout r and 94th in F-score of 324 routes. Rule-chosen routes: Figure 7 and Appendix K .
Figure 2: Overview of DRAFT: the coarse layout is fixed before the detail. (a) The readout maps EVC and camera motion to the layout code, which (b) a renderer and generator turn into eight depth maps. (c) Trajectory and fusion heads give the DRAFT layout, which (d) sets the broad structure of any depth map before range calibration and lifting to 3D. Real data of the Figure 1 route, except the noise z0 .
Layout
Pixel
Surface
Layout from
Inputs
r↑
Row ↑
L/R ↑
MAE (m) ↓
F@25cm ↑
Chamfer (m) ↓
Generator alone
none
0.303
0.626
0.602
1.507
0.478
0.449
Readout, 3×3 target
C, M
0.440
0.656
0.621
1.176
0.485
0.417
Trajectory head
M
0.390
0.665
0.632
1.264
0.447
0.443
Trajectory head
M, T
0.446
0.688
0.658
1.252
0.465
0.427
Fusion head (DRAFT)
C, M, T
0.480
0.696
0.668
1.153
0.484
0.405
Table 1: Main results: fusion gives the best layout at no measurable cost in surface accuracy. Each layout replaces the broad structure of one generator’s depth. C: EVC, M: camera motion (12 readout regressors, 72 trajectory-head features), T: free space. Row: row order, L/R: left/right, bold: best, underline: second. Layout, MAE: 324-route means, others: ten-scene means. Last rows: DRAFT minus generator alone ( ∗ : 95% paired scene-bootstrap CI excludes zero).
Layout r↑
F@25cm ↑
Chamfer (m) ↓
Method
alone
+ path
+ DRAFT
alone
+ path
+ DRAFT
alone
+ path
+ DRAFT
Linear depth
0.429
0.454
0.490 ∗
0.393
0.440
0.455 ∗
0.508
0.454
0.436 ∗
CLIP retrieval
0.261
0.451
0.488 ∗
0.455
0.473
0.493 ∗
0.421
0.406
0.383 ∗
Image decoding
0.171
0.450
0.488 ∗
0.442
0.456
0.474 ∗
0.446
0.426
0.403 ∗
Depth decoding
0.322
0.455
0.491 ∗
0.317
0.435
0.449 ∗
0.727
0.462
0.444 ∗
DREAM depth branch †
0.428
0.450
0.488 ∗
0.368
0.435
0.449 ∗
0.561
0.464
0.445 ∗
Table 2: Comparison with fMRI decoders: every decoder scores best with the DRAFT layout, and none matches it given only the camera path. Alone: unmodified. + path, + DRAFT: with the trajectory layout (camera path only) or the DRAFT layout, post hoc except for linear depth and Brain-Diffuser. Averaging as in Table 1 . Bold: row best. ∗ : 95% CI of the gain over alone excludes zero ( Table 5 ). Unmodified MindEye and Brain-Diffuser keep slightly higher F-scores than DRAFT. Decoders: 418 recordings, DRAFT: 416. ‡ DA V2 on the frames. † Our adaptation ( Section C.1 ).
Comparison
Metric
Gain [95% interval]
Scenes
DRAFT vs generator
r
+0.172 [0.135, 0.210]
10 a
Row
+0.066 [0.040, 0.094]
10
L/R
+0.065 [0.028, 0.101]
8
MAE
+0.356 [0.304, 0.408]
10
vs fixed template
r
+0.144 [0.084, 0.204]
9 b
vs per-second median
r
+0.225 [0.149, 0.298]
10 a
Table 3: Layout r gains of DRAFT hold in nine or ten scenes, and surface gains are smaller. Ten-scene means of paired differences (positive: first better), 95% paired scene-bootstrap CI, scenes favoring the first. r : layout r , BD: Brain-Diffuser, M, T: as in Table 1 , cosine: similarity of centered codes. MAE, Chamfer in m. Sign-flip p : a 0.002, b 0.004, c 0.012, d 0.025, others untested.
Figure 3: EVC predicts the layout code best of the tested regions. (a) Template masks, not individual localizers. (b) Held-out r per layout-code entry (ten-scene mean), from motion alone or with 1,000 voxels of one mask. (c) Per-scene differences in cosine similarity to the true code.
Figure 4: Surface scores separate methods less than layout scores do. (a to d) Display route (scene 6), one of the top five for DRAFT among routes where DRAFT and BD + DRAFT layout beat unmodified Brain-Diffuser (BD) in F-score. b to d: true points below 2.4 m by distance to the nearest prediction (F: F-score, R: recall, Figure 6 ). On rule-chosen routes, DRAFT has the lowest F-score of the generator, DRAFT and BD + DRAFT layout in seven of ten ( Table 9 ). (e) Layout r gain over the fixed template. (f) Post hoc F-score gain over a wrong scene.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Scene
1
2
3
4
5
6
7
8
9
10
Total
Archive ID suffix
01
02
03
04
05
06
07
09
10
12
Styles
4
4
4
4
4
4
4
3
2
3
36
Recordings
24
24
24
24
74
74
74
38
24
36
416
Participants
6
6
6
6
19
19
19
13
12
12
20
Routes
36
36
36
36
36
36
36
27
18
27
324
Appendix
Table 4: The ten scenes differ in styles and participants. A recording is one participant watching one style video, and each video gives nine routes of eight consecutive one-second views. The participant total counts distinct people. Counts are those of DRAFT, and the decoders use two more recordings (scenes 9 and 10, Appendix C ).
(a) DRAFT minus each method
Layout r↑
F@25cm ↑
DRAFT minus
Mean [95% CI] (scenes)
p
Mean [95% CI] (scenes)
p
References
Video frames + DA V2 ‡
− 0.393
[ − 0.445, − 0.340]
–
− 0.258
[ − 0.295, − 0.221]
–
Fixed template
+ 0.144
[0.084, 0.204]
–
+ 0.093
[0.080, 0.108]
–
Decoder alone (unmodified output)
Appendix
Table 5: Given the same camera path, DRAFT has a higher layout r than every decoder, and the DRAFT layout improves every decoder. (a) DRAFT minus each method of Table 2 . (b) The markers of Table 2 : decoder with the DRAFT layout and range calibration ( a=2 ) minus unmodified decoder ( a=1 ), ∗ if the interval excludes zero. The surface gains of Brain-Diffuser and MindEye are not reliable. Ten-scene means of per-scene differences, 95% paired scene-bootstrap intervals and scenes of ten favoring DRAFT or the layout. Post hoc exact sign-flip p ( Appendix B ) for (b) and the trajectory-layout rows of (a). Layout rows are post hoc except for linear depth and Brain-Diffuser ( Section C.2 ). ‡ DA V2 on the frames. † Our adaptation ( Section C.1 ).
Figure 5: EVC’s gain over camera motion peaks at a 3 to 4 s lag and exceeds all 20 circular shifts. Gain: cosine similarity of the EVC readout minus that of motion alone. (a) Single-second BOLD lags. Dots: scenes (hollow and numbered at or below zero). Diamonds and bars: mean and 95% paired scene-bootstrap interval. Shading and dashed line: the 2 to 5 s window of the main readout, which averages four lags and can gain more than any single one. (b) Circular shifts of 10 to 70 s (gray) and the unshifted window (teal), on the scale of (a).
Input
Cos. ↑
Entry r↑
MSE ↓ (std.)
MAE ↓ (std.)
Δ Cos. ↑
Scenes >0
EVC (V1 to hV4)
0.334
0.394
0.848
0.678
+ 0.219 [ + 0.150, + 0.290]
10
Scene-selective
0.174
0.221
0.971
0.740
+ 0.059 [ + 0.019, + 0.088]
9
Union (both)
0.334
0.394
0.847
0.678
+ 0.219 [ + 0.151, + 0.287]
10
Motion alone (reference)
0.115
0.185
0.987
0.751
–
–
Appendix
Table 6: With 1,000 voxels each, EVC predicts the layout code best, and the scene-selective regions beat motion alone. Ten-scene means. Cos.: cosine similarity of centered predicted and true codes. Entry r : mean per-entry correlation (neither is layout r ). MSE, MAE: mean squared and absolute error (std.: standardized units). Δ Cos.: minus motion alone, with 95% paired scene-bootstrap interval. Scenes >0 : of 10, not applicable to the reference row. Bold: within 0.002 of best. Contrasts between regions: Section D.2 and Table 3 .
Layout r↑
Pixel MAE (cm) ↓
F@25cm (pp) ↑
Chamfer (cm) ↓
G
D
G 1
G 2
D 1
D 2
G 1
G 2
D 1
D 2
G 1
G 2
D 1
D 2
Feature replacement: intact minus centered EVC features
Mean
+ 0.064
+ 0.037
− 5.7
− 3.5
− 7.7
− 4.2
+ 2.1
+ 1.1
+ 4.0
+ 3.6
− 2.3
+ 0.3
− 4.7
− 2.5
95% low
+ 0.046
+ 0.025
− 7.4
− 5.8
− 10.6
− 6.6
+ 1.4
+ 0.7
+ 2.2
+ 2.4
− 3.0
− 0.4
− 6.4
− 3.6
95% high
+ 0.082
+ 0.051
− 4.4
− 1.4
− 5.3
− 2.0
+ 2.8
+ 1.5
+ 5.9
+ 5.0
− 1.7
+ 0.9
− 3.1
− 1.1
Scenes better
10
10
10
7
10
9
10
10
10
10
10
4
10
9
Appendix
Table 7: Replacing or shifting the brain input worsens layout and surfaces in most scenes. Intact minus control, for feature replacement (readout kept) and circular shift (readout refitted). Ten-scene means with 95% paired scene-bootstrap intervals (low, high). Scenes better: of ten, scenes in which the intact input scores better (per-scene values omitted; Appendix K ). Pixel MAE and Chamfer in cm (m in the other tables), F in percentage points (pp). G: generator, D: DRAFT, subscript: a . Only one mean favors the control, G 2 Chamfer under feature replacement, within noise.
Held-out scene
D
1
2
3
4
5
6
7
8
9
10
Mean
D − G
95% interval
better
Layout r↑
G
0.307
0.336
0.314
0.283
0.343
0.277
0.285
0.275
0.262
0.324
0.301
D
0.551
0.513
0.578
0.532
0.492
0.359
0.479
0.391
0.408
0.427
0.473
+ 0.172
[ + 0.135, + 0.210]
10/10
Pixel MAE (m) ↓
G
1.419
1.493
1.577
1.434
1.529
1.559
1.410
1.512
1.668
1.572
1.517
D
1.225
1.131
1.049
1.024
1.242
1.153
1.120
1.172
1.289
1.211
1.162
− 0.356
[ − 0.408, − 0.304]
10/10
F@25cm ↑
G
0.455
0.478
0.465
0.480
0.534
0.506
0.472
0.462
0.466
0.464
0.478
Appendix
Table 8: DRAFT raises layout r in every held-out scene, and its surface comparison with the generator depends on a . At a=2 it lowers pixel MAE in every scene and Chamfer distance in eight, but its F-score is higher in only four. Scene means (equal weights, 324 routes) of the generator (G) and DRAFT (D). Pixel MAE and surfaces at a=2 . D − G: mean difference, 95% paired scene-bootstrap interval and scenes with D better. Bold: better mean. At a=1 the F-score and Chamfer distance of DRAFT are worse than those of the generator in every scene ( Table 13 , Appendix K ).
Figure 6: Three routes chosen for display, not representative: predictions recover the room’s extent and main wall positions, not clear walls or furniture. Rule: among routes where DRAFT and Brain-Diffuser + DRAFT layout beat unmodified Brain-Diffuser (Brain-Diffuser alone) in F-score, highest DRAFT F-score first, one per scene, without the route of Figure 4 (rule-chosen routes: Table 9 ). Brain-Diffuser alone: Table 2 . Row 1 (found): true points by distance to the nearest prediction (R: recall). Row 2 (predicted): predicted points by distance to the ground truth (P: precision). Points above 2.4 m or outside the plotted area, and in row 2 stray points (fewer than 4 neighbors or clusters under 40 points, radii scaled with range), are hidden for display, but all points are scored. F: route F-score (bold: best of three).
Depth map
Merged eight-view surface
Layout r↑
Pixel MAE (m) ↓
F@25cm ↑
Chamfer (m) ↓
Scene
G
D
G
D
G
D
BD + L
G
D
BD + L
1
0.536
0.563
0.988
0.856
0.574
0.541
0.488
0.277
0.279
0.304
2
0.356
0.500
1.258
0.846
0.560
0.550
0.606
0.426
0.299
0.252
3
0.010
0.566
2.277
1.756
0.435
0.431
0.434
0.484
0.440
0.403
4
0.435
0.583
1.221
0.705
0.536
0.686
0.635
0.399
0.238
0.259
Appendix
Table 9: On the ten atlas routes DRAFT improves the layout, but its F-score is the lowest of the three methods in seven routes. Scores of the ten atlas routes, whose surface atlas is listed in Appendix K , all at a=2 . G: generator, D: DRAFT, BD + L: Brain-Diffuser + DRAFT layout (surfaces only). Bold per row: best of three for surfaces. Mean: over the ten routes. Bottom: paired differences with 95% route-bootstrap intervals and routes where D is better.
Figure 7: Depth atlas, scenes 1 to 4, failures included: on the ten rule-chosen routes, Brain-Diffuser + DRAFT layout has the highest route layout r in six scenes, DRAFT in three and unmodified Brain-Diffuser in one. All eight views, 1 s apart, of the rule-chosen routes of Table 9 ( Figure 1 is hand-picked). Rows: furnished ground truth (GT, gray: invalid), DRAFT (D, a=2 ), unmodified Brain-Diffuser (U, calibrated DA V2 depth, a=1 ) and Brain-Diffuser + DRAFT layout (B, a=2 ). Numbers: route layout r (bold: best of three) and the scene’s value over all routes (in parentheses). Means over these ten and all 324 routes: 0.493 and 0.480 (D), 0.236 and 0.324 (U), 0.509 and 0.493 (B). Scenes 5 to 10: Appendix K .
(a)
Median-aligned
Log-affine fit
Unaligned pixels
Method
F ↑
Chamfer (m) ↓
F ↑
Chamfer (m) ↓
SI-log ↓
AbsRel ↓
δ<1.25↑
Fixed template
0.569
0.331
0.575
0.319
52.8
1.150
0.281
Generator
0.548
0.340
0.592
0.316
54.9
0.851
0.265
Trajectory head
0.582
0.324
0.622
0.280
50.6
0.857
0.223
Linear depth
0.581
0.322
0.607
0.290
50.1
0.825
0.234
Brain-Diffuser
0.571
0.328
0.601
0.291
57.3
1.064
0.291
Appendix
Table 10: Once per-view depth scale is removed by alignment, the methods with a DRAFT or trajectory layout score best. (a) Means over ten held-out scenes at a=1 . Alignment uses test depth (post hoc). F: F-score at 25 cm. SI-log: ×100 . Bold, underline: best and second best per column, ties share a mark. (b) Paired differences, 95% paired scene-bootstrap intervals, scenes favoring the first method and sign-flip p ( Appendix B ). † First minus second. BD: Brain-Diffuser.
F@25cm, eight views ↑
Change in 3D error (m), 1 s lag ↓
Prediction source
a=1
a=2
a=1
a=2
Current models
Generator
0.469 [0.442, 0.493]
0.456 [0.435, 0.477]
0.585 [0.561, 0.613]
1.227 [1.170, 1.292]
DRAFT
0.418 [0.391, 0.445]
0.467 [0.435, 0.494]
0.372 [0.347, 0.398]
0.627 [0.577, 0.681]
Decoders
Brain-Diffuser
0.492 [0.467, 0.516]
0.411 [0.398, 0.427]
0.915 [0.887, 0.944]
2.404 [2.314, 2.498]
Appendix
Table 11: The DRAFT layout lowers the change in 3D error between views for every source at both factors, but at a=1 it lowers the F-score of DRAFT and Brain-Diffuser. Eight-view F-score and one-second change in 3D error E1 of Equation 11 over the 72 views of each video. Ten-scene means with 95% scene-bootstrap intervals (paired for differences). Bold, underline: best and second best. Bottom: with minus without the DRAFT layout. BD: Brain-Diffuser.
Scored against the evaluated scene
Correct
Correct minus wrong scene
Prediction
a
F@25cm ↑
Precision ↑
Recall ↑
Chamfer (m) ↓
F@25cm ↑
F gain ↑ [95% CI]
Scenes (of 10)
Fixed template
Fixed template
1
0.391
0.399
0.405
0.524
–
–
–
2
0.437
0.390
0.519
0.437
–
–
–
Wrong-scene control: predictions for the next held-out scene
DRAFT
1
0.379
0.379
0.403
0.546
0.416
+0.037 [0.011, 0.063]
6
Appendix
Table 12: Wrong-scene predictions keep most of the F-score. Upper rows: the fixed template scored as a prediction. Lower rows: predictions for the next held-out scene lifted at the evaluated poses (post hoc), and the gain of the correct scene, with its 95% paired scene-bootstrap interval and the number of scenes (of ten) with a positive gain. Ten-scene means. –: not applicable, since the template is the same for every scene. Brain-Diffuser and MindEye at their best factor, a=1 .
Pixel MAE (m) ↓
F@25cm ↑
Chamfer (m) ↓
Method (inputs)
Layout r↑
a=1
a=2†
a=1
a=2†
a=1
a=2†
Generator alone (none)
0.303
1.168
1.507
0.477
0.478
0.426
0.449
+ 0.002 [ − 0.012, + 0.013]
+ 0.023 [ − 0.002, + 0.047]
Readout, 3×3 target (C, M)
0.440
1.097
1.176
0.432
0.485
0.478
0.417
Trajectory head (M)
0.390
1.217
1.264
0.383
0.447
0.535
0.443
Trajectory head (M, T)
0.446
1.185
1.252
0.397
0.465
0.514
0.427
Appendix
Table 13: Without range calibration ( a=1 ), fusion lowers the F-score and raises the Chamfer distance. Calibration raises the F-score of every layout except the generator’s and lowers it for the image decoders. C: EVC, M: camera motion, T: free-space features. Layout r and pixel MAE: 324-route means, F-score and Chamfer distance: ten-scene means, as in Table 1 . Bold, underline: best and second best current method. Small lines: a=2 minus a=1 , 95% paired scene-bootstrap interval. DRAFT minus generator: ten-scene means of per-scene differences ( ∗ : interval excludes zero). † Fixed after earlier results on these scenes. BD: Brain-Diffuser.
(a) Calibration-factor sweep
Calibration factor a
Method
1
1.5
2
2.5
3
F@25cm ↑
DRAFT
0.416
0.464
0.484
0.484
0.473
Generator alone
0.477
0.500
0.478
0.444
0.410
Brain-Diffuser
0.492
0.465
0.424
0.388
0.358
Appendix
Table 14: At each method’s best factor, DRAFT surfaces are no better than those of the generator or the image decoders. (a) F-score and Chamfer distance for a from 1 to 3 (bold: best a , chosen on the held-out scenes, paired differences in Table 15 ). (b) Precision and recall at a=2 and at the best a , and DRAFT minus the row (–: reference, bold: best at a=2 , the a=1 block is summarized in Appendix K ). Ten-scene means, 95% paired scene-bootstrap intervals.
(a) Selected factor and scores
Selected by Chamfer distance
Selected by F-score
Method
a (folds)
F@25cm ↑
Chamfer (m) ↓
a (folds)
F@25cm ↑
Chamfer (m) ↓
DRAFT
2 (10)
0.484
0.405
2 (5), 2.5 (5)
0.477
0.439
Generator alone
1.5 (10)
0.500
0.391
as by Chamfer distance
Brain-Diffuser
1 (10)
0.492
0.389
as above
MindEye
1 (10)
0.494
0.381
as above
Appendix
Table 15: Choosing a without the scored scene leaves DRAFT within noise of, or worse than, the generator and the image decoders. For each held-out scene, a∈{1,1.5,2,2.5,3} is chosen by the Chamfer distance or F-score of the other nine. (a) Chosen a , folds choosing it, and scores (bold: best over both selections). (b) DRAFT minus each method, 95% paired scene-bootstrap interval, scenes of ten favoring DRAFT. Ten-scene means.
Question
Statistic
Estimate [95% CI]
Scenes
p
Holm
Outcome
Does the brain’s gain survive single recordings?
Layout r , DRAFT minus feature replacement, one recording, no pooling
+0.012 [0.005, 0.018]
8
0.010
0.049
support
Does averaging more recordings raise layout r ?
Slope per doubling of recordings (1, 2, 3, 6)
+0.0047 [0.0038, 0.0055]
10
0.002
0.012
support ∗
Is the gain larger where the path covers less of the view?
Within-scene Spearman of path coverage and per-view gain
+0.042 [ −0.021 , 0.110]
7
0.27
0.82
null †
Does the gain over the path grow with path noise?
Slope of DRAFT minus trajectory head over six noise levels
−0.0006 [ −0.0047 , 0.0036]
4
0.81
0.82
null
Is DRAFT more scene-specific than MindEye?
Correct minus wrong-scene F@25cm, DRAFT minus unmodified MindEye
+0.014 [ −0.009 , 0.042]
6
0.37
0.82
null
Does DRAFT gain on MindEye at coarser thresholds?
Slope of the F-score difference per doubling of the threshold (10 cm to 1 m)
−0.0053 [ −0.0079 , −0.0024 ]
2
0.010
0.049
contrary
Appendix
Table 16: Post hoc primary tests: two support a hypothesis, three are null and one goes against it. Estimate: ten-scene mean and 95% paired scene-bootstrap interval. Scenes: scenes of ten with a positive value. p : exact sign-flip. Holm: adjusted over the six tests. Outcome: the decision rule written before the analysis. ∗ A time-misaligned brain component gives the same slope, so this test does not show brain signal (text). † The hypothesis predicted a negative value.
Scene
Recordings (participants)
DRAFT
H1: trajectory head
H2: fixed template
H3: feature replacement
W1
56 (19)
0.285
+0.013 [0.006, 0.020] ∗
+0.107 [0.099, 0.114] ∗
+0.030 [0.022, 0.036] ∗
W2
24 (6)
0.489
+0.031 [0.003, 0.045] ∗
+0.231 [0.202, 0.246] ∗
+0.078 [0.048, 0.092] ∗
W3
6 (6)
0.476
+0.004 [ −0.027 , 0.030]
+0.211 [0.181, 0.236] ∗
+0.034 [0.001, 0.061] ∗
W4
24 (6)
0.579
+0.033 [ −0.001 , 0.044]
+0.197 [0.163, 0.209] ∗
+0.025 [ −0.009 , 0.037]
W5
24 (6)
0.534
+0.013 [ −0.029 , 0.035]
+0.195 [0.152, 0.219] ∗
+0.041 [ −0.004 , 0.065]
Mean, scenes, p
+0.019 , 5, 0.031
+0.188 , 5, 0.031
+0.042 , 5, 0.031
Appendix
Table 17: Five withheld scenes: DRAFT beats the fixed template in every scene, but the brain’s contribution replicates within only two. Layout r , route-pooled. Differences: DRAFT minus the named layout, with 95% participant cluster bootstrap intervals that omit variation between routes. ∗ Replicated (positive, Holm-adjusted p<0.05 over the three contrasts of the scene). Mean: mean of the five differences, positive scenes and exact one-sided sign-flip p (1/32 is the smallest attainable value, and the scenes are not independent, see text). Development: ten-scene mean and range of the differences (layout r of DRAFT 0.359 to 0.578).
Inputs
Layout r↑
Contrast
Difference [95% CI]
Scenes
p
Role
M0: camera motion
0.338
M3 − M1
+0.013 [ −0.006 , 0.033]
6
0.25
primary: null
M1: + image features
0.561
M1 − M0
+0.223 [0.203, 0.244]
10
0.008 (Holm)
secondary
M2: + EVC (paper)
0.440
M2 − M0
+0.098 [0.053, 0.152]
9
0.008 (Holm)
secondary
M3: + image features + EVC
0.573
M3 − M2
+0.138 [0.109, 0.164]
10
0.008 (Holm)
secondary
M2 − M1
−0.125 [ −0.161 , −0.083 ]
1
0.008 (Holm)
secondary
Appendix
Table 18: EVC adds no measurable layout r to low-level image features of the frames. Readout with a 3×3 target, layout r over 324 routes. Differences: ten-scene mean [95% paired scene-bootstrap CI], scenes of ten with a positive value, exact sign-flip p . Secondary tests are Holm-adjusted over four. Left: readouts. Right: contrasts, not tied to the row beside them.
Reconstructing visual images from fMRI data is a central but challenging problem in neuroscience. Despite recent progress, current methods fall short when data and computation are limited-precisely the conditions under which this task is most critical. We introduce Adapter Alignment, a staged, architecture-agnostic training procedure for adapting a pre-trained reconstruction model to a new subject with little data. Rather than letting a shared representation space emerge implicitly, lightweight subject-specific adapters are first explicitly aligned to a pre-trained reference subject on a small set of shared calibration images, and the full pipeline is then fine-tuned while preserving this alignment. The resulting representations not only place different subjects in a shared space but also align them semantically. Our results demonstrate significant gains in low-data regimes, improving reconstruction metrics by up to 8%, and pre-diffusion retrieval shows that these gains reflect additional image-specific information in the fMRI embeddings rather than the generative prior. We also propose a coverage-based algorithm to select representative calibration images. Using both techniques together, we surpass baseline models fine-tuned with one hour of data using only 24 minutes of subject-specific data. The procedure is effective across architectures, reference subjects, and datasets, including adapting 3T subjects to a 7T reference.
Christos Zangos, Danish Ebadulla, Thomas Christopher Sprague +1
Department of Computer Science, University of California, Santa Barbara, USA · Department of Psychological and Brain Sciences, University of California, Santa Barbara, USA
Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the NSD-Imagery benchmark. We propose a latent functional alignment (LFA) approach that maps imagery-evoked activity to the pretrained model's semantic content-enriched conditioning space, by adding a simple alignment module, while keeping the original remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a neural retrieval-based augmentation strategy that selects semantically related NSD perception trials from the same participants. Across four subjects, LFA consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.
Fabrizio Spera, Tommaso Boccato, Michal Olak +2
Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA
Reconstructing natural images from fMRI requires bridging neural activity with both the structural and semantic representations used by modern generative models. Existing diffusion-based decoders often condition on a single global fMRI embedding, which limits their ability to exploit the hierarchical organization of the visual cortex and makes the contribution of different visual areas difficult to inspect. We propose Hi-DREAM, a brain-inspired hierarchical diffusion framework that structures fMRI conditioning according to early, middle, and late visual Regions of Interest (ROI) streams. A ROI adapter converts these streams into a multi-scale cortical pyramid, and a lightweight ROI-conditioned ControlNet injects the resulting anatomy-aware priors into matched U-Net depths during denoising. Experiments on the Natural Scenes Dataset (NSD) show that Hi-DREAM achieves state-of-the-art high-level semantic reconstruction while retaining strong low-level structure. Further ablation and attribution analyses show that the proposed hierarchy-aware conditioning is effective, and that different ROI streams provide complementary, inspectable contributions to reconstruction.
Guowei Zhang, Yun Zhao, Kai Sun +4
Dept of Data Science and AI, Monash University, Melbourne, Australia · Brain and Mind Centre, University of Sydney, Sydney, Australia · School of Psychological Sciences, Monash University, Melbourne, Australia