DRAFT: Decoding and Reconstructing 3D Room Layouts from fMRI and Camera Trajectories

Chi-Wei Lee; An-Kai Chen; Cheng-Han Chiang; Yi-Chun Chang; Po-Chih Kuo

cs.CVOct 22, 2026

DRAFT: Decoding and Reconstructing 3D Room Layouts from fMRI and Camera Trajectories

Authors: Chi-Wei Lee, An-Kai Chen, Cheng-Han Chiang, Yi-Chun Chang, Po-Chih Kuo

Organizations: National Tsing Hua University · National Yang Ming Chiao Tung University

Abstract

While functional magnetic resonance imaging (fMRI) decoders reconstruct images and 3D objects, recovering the metric geometry of rooms from a moving viewpoint remains an open challenge. We introduce, to our knowledge for the first time, the task of reconstructing 3D layouts of held-out scenes using only the known camera trajectories and the fMRI signals of participants watching indoor walkthroughs, without any visual frames. The task is hard because of the temporal mismatch between rapid visual changes and blurred fMRI signals. Our method, DRAFT (Decoding and Reconstructing 3D room lAyouts from fMRI and Trajectories), extracts a compact layout representation from the early visual cortex and fuses this neural code with the geometric free space derived from camera motion. To evaluate it, we propose layout r, the Pearson correlation between predicted and ray-cast log depths. On ten held-out scenes (20 participants), DRAFT reaches a layout r of 0.480, outperforming the camera path alone (0.446) and seven fMRI decoders (at most 0.429). DRAFT also works as a plug-in that improves all seven decoders without retraining, raising their layout r from 0.171-0.429 to 0.487-0.493. Brain activity and camera motion can thus recover the metric layout of unseen rooms, a first step toward reconstructing the 3D spaces people see.

Figures & tables

Figure 1: DRAFT recovers where walls and floor are on a held-out route. (a) Four views of scene 4: frame (not an input), furnished (F) and empty-room (E) ground truth, DRAFT (D), Brain-Diffuser (BD) detail with the DRAFT layout (B), and layout $r$ of D and B against F and E. (b) Merged surface of D from above, blue within 25 cm of F. Hand-picked route, ranked 60th in layout $r$ and 94th in F-score of 324 routes. Rule-chosen routes: Figure 7 and Appendix K .

Figure 2: Overview of DRAFT: the coarse layout is fixed before the detail. (a) The readout maps EVC and camera motion to the layout code, which (b) a renderer and generator turn into eight depth maps. (c) Trajectory and fusion heads give the DRAFT layout, which (d) sets the broad structure of any depth map before range calibration and lifting to 3D. Real data of the Figure 1 route, except the noise ${\bm{z}}_{0}$ .

		Layout			Pixel	Surface
Layout from	Inputs	$r$ $\uparrow$	Row $\uparrow$	L/R $\uparrow$	MAE (m) $\downarrow$	F@25cm $\uparrow$	Chamfer (m) $\downarrow$
Generator alone	none	0.303	0.626	0.602	1.507	0.478	0.449
Readout, $3{\times}3$ target	C, M	0.440	0.656	0.621	1.176	0.485	0.417
Trajectory head	M	0.390	0.665	0.632	1.264	0.447	0.443
Trajectory head	M, T	0.446	0.688	0.658	1.252	0.465	0.427
Fusion head (DRAFT)	C, M, T	0.480	0.696	0.668	1.153	0.484	0.405

Table 1: Main results: fusion gives the best layout at no measurable cost in surface accuracy. Each layout replaces the broad structure of one generator’s depth. C: EVC, M: camera motion (12 readout regressors, 72 trajectory-head features), T: free space. Row: row order, L/R: left/right, bold: best, underline: second. Layout, MAE: 324-route means, others: ten-scene means. Last rows: DRAFT minus generator alone ( ∗ : 95% paired scene-bootstrap CI excludes zero).

	Layout $r$ $\uparrow$			F@25cm $\uparrow$			Chamfer (m) $\downarrow$
Method	alone	+ path	+ DRAFT	alone	+ path	+ DRAFT	alone	+ path	+ DRAFT
Linear depth	0.429	0.454	0.490 ∗	0.393	0.440	0.455 ∗	0.508	0.454	0.436 ∗
CLIP retrieval	0.261	0.451	0.488 ∗	0.455	0.473	0.493 ∗	0.421	0.406	0.383 ∗
Image decoding	0.171	0.450	0.488 ∗	0.442	0.456	0.474 ∗	0.446	0.426	0.403 ∗
Depth decoding	0.322	0.455	0.491 ∗	0.317	0.435	0.449 ∗	0.727	0.462	0.444 ∗
DREAM depth branch †	0.428	0.450	0.488 ∗	0.368	0.435	0.449 ∗	0.561	0.464	0.445 ∗

Table 2: Comparison with fMRI decoders: every decoder scores best with the DRAFT layout, and none matches it given only the camera path. Alone: unmodified. + path, + DRAFT: with the trajectory layout (camera path only) or the DRAFT layout, post hoc except for linear depth and Brain-Diffuser. Averaging as in Table 1 . Bold: row best. ∗ : 95% CI of the gain over alone excludes zero ( Table 5 ). Unmodified MindEye and Brain-Diffuser keep slightly higher F-scores than DRAFT. Decoders: 418 recordings, DRAFT: 416. ‡ DA V2 on the frames. † Our adaptation ( Section C.1 ).

Comparison	Metric	Gain [95% interval]	Scenes
DRAFT vs generator	$r$	+0.172 [0.135, 0.210]	10 a
	Row	+0.066 [0.040, 0.094]	10
	L/R	+0.065 [0.028, 0.101]	8
	MAE	+0.356 [0.304, 0.408]	10
vs fixed template	$r$	+0.144 [0.084, 0.204]	9 b
vs per-second median	$r$	+0.225 [0.149, 0.298]	10 a

Table 3: Layout

r

gains of DRAFT hold in nine or ten scenes, and surface gains are smaller. Ten-scene means of paired differences (positive: first better), 95% paired scene-bootstrap CI, scenes favoring the first.

r

: layout

r

, BD: Brain-Diffuser, M, T: as in Table 1 , cosine: similarity of centered codes. MAE, Chamfer in m. Sign-flip

p

: a 0.002, b 0.004, c 0.012, d 0.025, others untested.

Figure 3: EVC predicts the layout code best of the tested regions. (a) Template masks, not individual localizers. (b) Held-out $r$ per layout-code entry (ten-scene mean), from motion alone or with 1,000 voxels of one mask. (c) Per-scene differences in cosine similarity to the true code.

Figure 4: Surface scores separate methods less than layout scores do. (a to d) Display route (scene 6), one of the top five for DRAFT among routes where DRAFT and BD + DRAFT layout beat unmodified Brain-Diffuser (BD) in F-score. b to d: true points below 2.4 m by distance to the nearest prediction (F: F-score, R: recall, Figure 6 ). On rule-chosen routes, DRAFT has the lowest F-score of the generator, DRAFT and BD + DRAFT layout in seven of ten ( Table 9 ). (e) Layout $r$ gain over the fixed template. (f) Post hoc F-score gain over a wrong scene.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Scene	1	2	3	4	5	6	7	8	9	10	Total
Archive ID suffix	01	02	03	04	05	06	07	09	10	12
Styles	4	4	4	4	4	4	4	3	2	3	36
Recordings	24	24	24	24	74	74	74	38	24	36	416
Participants	6	6	6	6	19	19	19	13	12	12	20
Routes	36	36	36	36	36	36	36	27	18	27	324

Appendix

Table 4: The ten scenes differ in styles and participants. A recording is one participant watching one style video, and each video gives nine routes of eight consecutive one-second views. The participant total counts distinct people. Counts are those of DRAFT, and the decoders use two more recordings (scenes 9 and 10, Appendix C ).

(a) DRAFT minus each method
	Layout $r$ $\uparrow$			F@25cm $\uparrow$
DRAFT minus	Mean [95% CI] (scenes)		$p$	Mean [95% CI] (scenes)		$p$
References
Video frames + DA V2 ‡	$-$ 0.393	[ $-$ 0.445, $-$ 0.340]	–	$-$ 0.258	[ $-$ 0.295, $-$ 0.221]	–
Fixed template	$+$ 0.144	[0.084, 0.204]	–	$+$ 0.093	[0.080, 0.108]	–
Decoder alone (unmodified output)

Appendix

Table 5: Given the same camera path, DRAFT has a higher layout

r

than every decoder, and the DRAFT layout improves every decoder. (a) DRAFT minus each method of Table 2 . (b) The markers of Table 2 : decoder with the DRAFT layout and range calibration (

a=2

) minus unmodified decoder (

a=1

), ∗ if the interval excludes zero. The surface gains of Brain-Diffuser and MindEye are not reliable. Ten-scene means of per-scene differences, 95% paired scene-bootstrap intervals and scenes of ten favoring DRAFT or the layout. Post hoc exact sign-flip

p

( Appendix B ) for (b) and the trajectory-layout rows of (a). Layout rows are post hoc except for linear depth and Brain-Diffuser ( Section C.2 ). ‡ DA V2 on the frames. † Our adaptation ( Section C.1 ).

Figure 5: EVC’s gain over camera motion peaks at a 3 to 4 s lag and exceeds all 20 circular shifts. Gain: cosine similarity of the EVC readout minus that of motion alone. (a) Single-second BOLD lags. Dots: scenes (hollow and numbered at or below zero). Diamonds and bars: mean and 95% paired scene-bootstrap interval. Shading and dashed line: the 2 to 5 s window of the main readout, which averages four lags and can gain more than any single one. (b) Circular shifts of 10 to 70 s (gray) and the unshifted window (teal), on the scale of (a).

Input	Cos. $\uparrow$	Entry $r$ $\uparrow$	MSE $\downarrow$ (std.)	MAE $\downarrow$ (std.)	$\Delta$ Cos. $\uparrow$	Scenes $>0$
EVC (V1 to hV4)	0.334	0.394	0.848	0.678	$+$ 0.219 [ $+$ 0.150, $+$ 0.290]	10
Scene-selective	0.174	0.221	0.971	0.740	$+$ 0.059 [ $+$ 0.019, $+$ 0.088]	9
Union (both)	0.334	0.394	0.847	0.678	$+$ 0.219 [ $+$ 0.151, $+$ 0.287]	10
Motion alone (reference)	0.115	0.185	0.987	0.751	–	–

Appendix

Table 6: With 1,000 voxels each, EVC predicts the layout code best, and the scene-selective regions beat motion alone. Ten-scene means. Cos.: cosine similarity of centered predicted and true codes. Entry

r

: mean per-entry correlation (neither is layout

r

). MSE, MAE: mean squared and absolute error (std.: standardized units).

\Delta

Cos.: minus motion alone, with 95% paired scene-bootstrap interval. Scenes

>0

: of 10, not applicable to the reference row. Bold: within 0.002 of best. Contrasts between regions: Section D.2 and Table 3 .

Feature replacement: intact minus centered EVC features
	Layout $r$ $\uparrow$		Pixel MAE (cm) $\downarrow$				F@25cm (pp) $\uparrow$				Chamfer (cm) $\downarrow$
	G	D	G 1	G 2	D 1	D 2	G 1	G 2	D 1	D 2	G 1	G 2	D 1	D 2
Mean	$+$ 0.064	$+$ 0.037	$-$ 5.7	$-$ 3.5	$-$ 7.7	$-$ 4.2	$+$ 2.1	$+$ 1.1	$+$ 4.0	$+$ 3.6	$-$ 2.3	$+$ 0.3	$-$ 4.7	$-$ 2.5
95% low	$+$ 0.046	$+$ 0.025	$-$ 7.4	$-$ 5.8	$-$ 10.6	$-$ 6.6	$+$ 1.4	$+$ 0.7	$+$ 2.2	$+$ 2.4	$-$ 3.0	$-$ 0.4	$-$ 6.4	$-$ 3.6
95% high	$+$ 0.082	$+$ 0.051	$-$ 4.4	$-$ 1.4	$-$ 5.3	$-$ 2.0	$+$ 2.8	$+$ 1.5	$+$ 5.9	$+$ 5.0	$-$ 1.7	$+$ 0.9	$-$ 3.1	$-$ 1.1
Scenes better	10	10	10	7	10	9	10	10	10	10	10	4	10	9

Appendix

Table 7: Replacing or shifting the brain input worsens layout and surfaces in most scenes. Intact minus control, for feature replacement (readout kept) and circular shift (readout refitted). Ten-scene means with 95% paired scene-bootstrap intervals (low, high). Scenes better: of ten, scenes in which the intact input scores better (per-scene values omitted; Appendix K ). Pixel MAE and Chamfer in cm (m in the other tables), F in percentage points (pp). G: generator, D: DRAFT, subscript:

a

. Only one mean favors the control, G 2 Chamfer under feature replacement, within noise.

		Held-out scene													D
		1	2	3	4	5	6	7	8	9	10	Mean	D $-$ G	95% interval	better
Layout $r$ $\uparrow$	G	0.307	0.336	0.314	0.283	0.343	0.277	0.285	0.275	0.262	0.324	0.301
	D	0.551	0.513	0.578	0.532	0.492	0.359	0.479	0.391	0.408	0.427	0.473	$+$ 0.172	[ $+$ 0.135, $+$ 0.210]	10/10
Pixel MAE (m) $\downarrow$	G	1.419	1.493	1.577	1.434	1.529	1.559	1.410	1.512	1.668	1.572	1.517
	D	1.225	1.131	1.049	1.024	1.242	1.153	1.120	1.172	1.289	1.211	1.162	$-$ 0.356	[ $-$ 0.408, $-$ 0.304]	10/10
F@25cm $\uparrow$	G	0.455	0.478	0.465	0.480	0.534	0.506	0.472	0.462	0.466	0.464	0.478

Appendix

Table 8: DRAFT raises layout

r

in every held-out scene, and its surface comparison with the generator depends on

a

. At

a=2

it lowers pixel MAE in every scene and Chamfer distance in eight, but its F-score is higher in only four. Scene means (equal weights, 324 routes) of the generator (G) and DRAFT (D). Pixel MAE and surfaces at

a=2

. D

-

G: mean difference, 95% paired scene-bootstrap interval and scenes with D better. Bold: better mean. At

a=1

the F-score and Chamfer distance of DRAFT are worse than those of the generator in every scene ( Table 13 , Appendix K ).

Figure 6: Three routes chosen for display, not representative: predictions recover the room’s extent and main wall positions, not clear walls or furniture. Rule: among routes where DRAFT and Brain-Diffuser + DRAFT layout beat unmodified Brain-Diffuser (Brain-Diffuser alone) in F-score, highest DRAFT F-score first, one per scene, without the route of Figure 4 (rule-chosen routes: Table 9 ). Brain-Diffuser alone: Table 2 . Row 1 (found): true points by distance to the nearest prediction (R: recall). Row 2 (predicted): predicted points by distance to the ground truth (P: precision). Points above 2.4 m or outside the plotted area, and in row 2 stray points (fewer than 4 neighbors or clusters under 40 points, radii scaled with range), are hidden for display, but all points are scored. F: route F-score (bold: best of three).

	Depth map				Merged eight-view surface
	Layout $r$ $\uparrow$		Pixel MAE (m) $\downarrow$		F@25cm $\uparrow$			Chamfer (m) $\downarrow$
Scene	G	D	G	D	G	D	BD + L	G	D	BD + L
1	0.536	0.563	0.988	0.856	0.574	0.541	0.488	0.277	0.279	0.304
2	0.356	0.500	1.258	0.846	0.560	0.550	0.606	0.426	0.299	0.252
3	0.010	0.566	2.277	1.756	0.435	0.431	0.434	0.484	0.440	0.403
4	0.435	0.583	1.221	0.705	0.536	0.686	0.635	0.399	0.238	0.259

Appendix

Table 9: On the ten atlas routes DRAFT improves the layout, but its F-score is the lowest of the three methods in seven routes. Scores of the ten atlas routes, whose surface atlas is listed in Appendix K , all at

a=2

. G: generator, D: DRAFT, BD + L: Brain-Diffuser + DRAFT layout (surfaces only). Bold per row: best of three for surfaces. Mean: over the ten routes. Bottom: paired differences with 95% route-bootstrap intervals and routes where D is better.

Figure 7: Depth atlas, scenes 1 to 4, failures included: on the ten rule-chosen routes, Brain-Diffuser + DRAFT layout has the highest route layout $r$ in six scenes, DRAFT in three and unmodified Brain-Diffuser in one. All eight views, 1 s apart, of the rule-chosen routes of Table 9 ( Figure 1 is hand-picked). Rows: furnished ground truth (GT, gray: invalid), DRAFT (D, $a=2$ ), unmodified Brain-Diffuser (U, calibrated DA V2 depth, $a=1$ ) and Brain-Diffuser + DRAFT layout (B, $a=2$ ). Numbers: route layout $r$ (bold: best of three) and the scene’s value over all routes (in parentheses). Means over these ten and all 324 routes: 0.493 and 0.480 (D), 0.236 and 0.324 (U), 0.509 and 0.493 (B). Scenes 5 to 10: Appendix K .

(a)	Median-aligned		Log-affine fit		Unaligned pixels
Method	F $\uparrow$	Chamfer (m) $\downarrow$	F $\uparrow$	Chamfer (m) $\downarrow$	SI-log $\downarrow$	AbsRel $\downarrow$	$\delta{<}1.25$ $\uparrow$
Fixed template	0.569	0.331	0.575	0.319	52.8	1.150	0.281
Generator	0.548	0.340	0.592	0.316	54.9	0.851	0.265
Trajectory head	0.582	0.324	0.622	0.280	50.6	0.857	0.223
Linear depth	0.581	0.322	0.607	0.290	50.1	0.825	0.234
Brain-Diffuser	0.571	0.328	0.601	0.291	57.3	1.064	0.291

Appendix

Table 10: Once per-view depth scale is removed by alignment, the methods with a DRAFT or trajectory layout score best. (a) Means over ten held-out scenes at

a=1

. Alignment uses test depth (post hoc). F: F-score at 25 cm. SI-log:

\times 100

. Bold, underline: best and second best per column, ties share a mark. (b) Paired differences, 95% paired scene-bootstrap intervals, scenes favoring the first method and sign-flip

p

( Appendix B ). † First minus second. BD: Brain-Diffuser.

Current models
	F@25cm, eight views $\uparrow$		Change in 3D error (m), 1 s lag $\downarrow$
Prediction source	$a=1$	$a=2$	$a=1$	$a=2$
Generator	0.469 [0.442, 0.493]	0.456 [0.435, 0.477]	0.585 [0.561, 0.613]	1.227 [1.170, 1.292]
DRAFT	0.418 [0.391, 0.445]	0.467 [0.435, 0.494]	0.372 [0.347, 0.398]	0.627 [0.577, 0.681]
Decoders
Brain-Diffuser	0.492 [0.467, 0.516]	0.411 [0.398, 0.427]	0.915 [0.887, 0.944]	2.404 [2.314, 2.498]

Appendix

Table 11: The DRAFT layout lowers the change in 3D error between views for every source at both factors, but at

a=1

it lowers the F-score of DRAFT and Brain-Diffuser. Eight-view F-score and one-second change in 3D error

E_{1}

of Equation 11 over the 72 views of each video. Ten-scene means with 95% scene-bootstrap intervals (paired for differences). Bold, underline: best and second best. Bottom: with minus without the DRAFT layout. BD: Brain-Diffuser.

		Scored against the evaluated scene				Correct	Correct minus wrong scene
Prediction	$a$	F@25cm $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Chamfer (m) $\downarrow$	F@25cm $\uparrow$	F gain $\uparrow$ [95% CI]	Scenes (of 10)
Fixed template
Fixed template	1	0.391	0.399	0.405	0.524	–	–	–
	2	0.437	0.390	0.519	0.437	–	–	–
Wrong-scene control: predictions for the next held-out scene
DRAFT	1	0.379	0.379	0.403	0.546	0.416	$+0.037$ [0.011, 0.063]	6

Appendix

Table 12: Wrong-scene predictions keep most of the F-score. Upper rows: the fixed template scored as a prediction. Lower rows: predictions for the next held-out scene lifted at the evaluated poses (post hoc), and the gain of the correct scene, with its 95% paired scene-bootstrap interval and the number of scenes (of ten) with a positive gain. Ten-scene means. –: not applicable, since the template is the same for every scene. Brain-Diffuser and MindEye at their best factor,

a=1

		Pixel MAE (m) $\downarrow$		F@25cm $\uparrow$		Chamfer (m) $\downarrow$
Method (inputs)	Layout $r$ $\uparrow$	$a{=}1$	$a{=}2^{\dagger}$	$a{=}1$	$a{=}2^{\dagger}$	$a{=}1$	$a{=}2^{\dagger}$
Generator alone (none)	0.303	1.168	1.507	0.477	0.478	0.426	0.449
				$+$ 0.002 [ $-$ 0.012, $+$ 0.013]		$+$ 0.023 [ $-$ 0.002, $+$ 0.047]
Readout, $3{\times}3$ target (C, M)	0.440	1.097	1.176	0.432	0.485	0.478	0.417
Trajectory head (M)	0.390	1.217	1.264	0.383	0.447	0.535	0.443
Trajectory head (M, T)	0.446	1.185	1.252	0.397	0.465	0.514	0.427

Appendix

Table 13: Without range calibration (

a=1

), fusion lowers the F-score and raises the Chamfer distance. Calibration raises the F-score of every layout except the generator’s and lowers it for the image decoders. C: EVC, M: camera motion, T: free-space features. Layout

r

and pixel MAE: 324-route means, F-score and Chamfer distance: ten-scene means, as in Table 1 . Bold, underline: best and second best current method. Small lines:

a=2

minus

a=1

, 95% paired scene-bootstrap interval. DRAFT minus generator: ten-scene means of per-scene differences ( ∗ : interval excludes zero). † Fixed after earlier results on these scenes. BD: Brain-Diffuser.

(a) Calibration-factor sweep
	Calibration factor $a$
Method	1	1.5	2	2.5	3
F@25cm $\uparrow$
DRAFT	0.416	0.464	0.484	0.484	0.473
Generator alone	0.477	0.500	0.478	0.444	0.410
Brain-Diffuser	0.492	0.465	0.424	0.388	0.358

Appendix

Table 14: At each method’s best factor, DRAFT surfaces are no better than those of the generator or the image decoders. (a) F-score and Chamfer distance for

a

from 1 to 3 (bold: best

a

, chosen on the held-out scenes, paired differences in Table 15 ). (b) Precision and recall at

a=2

and at the best

a

, and DRAFT minus the row (–: reference, bold: best at

a=2

, the

a=1

block is summarized in Appendix K ). Ten-scene means, 95% paired scene-bootstrap intervals.

(a) Selected factor and scores
	Selected by Chamfer distance			Selected by F-score
Method	$a$ (folds)	F@25cm $\uparrow$	Chamfer (m) $\downarrow$	$a$ (folds)	F@25cm $\uparrow$	Chamfer (m) $\downarrow$
DRAFT	2 (10)	0.484	0.405	2 (5), 2.5 (5)	0.477	0.439
Generator alone	1.5 (10)	0.500	0.391	as by Chamfer distance
Brain-Diffuser	1 (10)	0.492	0.389	as above
MindEye	1 (10)	0.494	0.381	as above

Appendix

Table 15: Choosing

a

without the scored scene leaves DRAFT within noise of, or worse than, the generator and the image decoders. For each held-out scene,

a\in\{1,1.5,2,2.5,3\}

is chosen by the Chamfer distance or F-score of the other nine. (a) Chosen

a

, folds choosing it, and scores (bold: best over both selections). (b) DRAFT minus each method, 95% paired scene-bootstrap interval, scenes of ten favoring DRAFT. Ten-scene means.

Question	Statistic	Estimate [95% CI]	Scenes	$p$	Holm	Outcome
Does the brain’s gain survive single recordings?	Layout $r$ , DRAFT minus feature replacement, one recording, no pooling	$+0.012$ [0.005, 0.018]	8	0.010	0.049	support
Does averaging more recordings raise layout $r$ ?	Slope per doubling of recordings (1, 2, 3, 6)	$+0.0047$ [0.0038, 0.0055]	10	0.002	0.012	support ∗
Is the gain larger where the path covers less of the view?	Within-scene Spearman of path coverage and per-view gain	$+0.042$ [ $-0.021$ , 0.110]	7	0.27	0.82	null †
Does the gain over the path grow with path noise?	Slope of DRAFT minus trajectory head over six noise levels	$-0.0006$ [ $-0.0047$ , 0.0036]	4	0.81	0.82	null
Is DRAFT more scene-specific than MindEye?	Correct minus wrong-scene F@25cm, DRAFT minus unmodified MindEye	$+0.014$ [ $-0.009$ , 0.042]	6	0.37	0.82	null
Does DRAFT gain on MindEye at coarser thresholds?	Slope of the F-score difference per doubling of the threshold (10 cm to 1 m)	$-0.0053$ [ $-0.0079$ , $-0.0024$ ]	2	0.010	0.049	contrary

Appendix

Table 16: Post hoc primary tests: two support a hypothesis, three are null and one goes against it. Estimate: ten-scene mean and 95% paired scene-bootstrap interval. Scenes: scenes of ten with a positive value.

p

: exact sign-flip. Holm: adjusted over the six tests. Outcome: the decision rule written before the analysis. ∗ A time-misaligned brain component gives the same slope, so this test does not show brain signal (text). † The hypothesis predicted a negative value.

Scene	Recordings (participants)	DRAFT	H1: trajectory head	H2: fixed template	H3: feature replacement
W1	56 (19)	0.285	$+0.013$ [0.006, 0.020] ∗	$+0.107$ [0.099, 0.114] ∗	$+0.030$ [0.022, 0.036] ∗
W2	24 (6)	0.489	$+0.031$ [0.003, 0.045] ∗	$+0.231$ [0.202, 0.246] ∗	$+0.078$ [0.048, 0.092] ∗
W3	6 (6)	0.476	$+0.004$ [ $-0.027$ , 0.030]	$+0.211$ [0.181, 0.236] ∗	$+0.034$ [0.001, 0.061] ∗
W4	24 (6)	0.579	$+0.033$ [ $-0.001$ , 0.044]	$+0.197$ [0.163, 0.209] ∗	$+0.025$ [ $-0.009$ , 0.037]
W5	24 (6)	0.534	$+0.013$ [ $-0.029$ , 0.035]	$+0.195$ [0.152, 0.219] ∗	$+0.041$ [ $-0.004$ , 0.065]
Mean, scenes, $p$			$+0.019$ , 5, 0.031	$+0.188$ , 5, 0.031	$+0.042$ , 5, 0.031

Appendix

Table 17: Five withheld scenes: DRAFT beats the fixed template in every scene, but the brain’s contribution replicates within only two. Layout

r

, route-pooled. Differences: DRAFT minus the named layout, with 95% participant cluster bootstrap intervals that omit variation between routes. ∗ Replicated (positive, Holm-adjusted

p<0.05

over the three contrasts of the scene). Mean: mean of the five differences, positive scenes and exact one-sided sign-flip

p

(1/32 is the smallest attainable value, and the scenes are not independent, see text). Development: ten-scene mean and range of the differences (layout

r

of DRAFT 0.359 to 0.578).

Inputs	Layout $r$ $\uparrow$	Contrast	Difference [95% CI]	Scenes	$p$	Role
M0: camera motion	0.338	M3 $-$ M1	$+0.013$ [ $-0.006$ , 0.033]	6	0.25	primary: null
M1: + image features	0.561	M1 $-$ M0	$+0.223$ [0.203, 0.244]	10	0.008 (Holm)	secondary
M2: + EVC (paper)	0.440	M2 $-$ M0	$+0.098$ [0.053, 0.152]	9	0.008 (Holm)	secondary
M3: + image features + EVC	0.573	M3 $-$ M2	$+0.138$ [0.109, 0.164]	10	0.008 (Holm)	secondary
		M2 $-$ M1	$-0.125$ [ $-0.161$ , $-0.083$ ]	1	0.008 (Holm)	secondary

Appendix

Table 18: EVC adds no measurable layout

r

to low-level image features of the frames. Readout with a

3{\times}3

target, layout

r

over 324 routes. Differences: ten-scene mean [95% paired scene-bootstrap CI], scenes of ten with a positive value, exact sign-flip

p

. Secondary tests are Holm-adjusted over four. Left: readouts. Right: contrasts, not tied to the row beside them.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 955 papers

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects
May 3, 2025Christos Zangos, Danish Ebadulla, Thomas Christopher Sprague +1fMRI-Based Visual Decoding fMRI Representation Learning
Department of Computer Science, University of California, Santa Barbara, USA · Department of Psychological and Brain Sciences, University of California, Santa Barbara, USA
Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data
Apr 15, 2026Fabrizio Spera, Tommaso Boccato, Michal Olak +2Neural Decoding Visual Neuroscience
Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA
Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping
Nov 14, 2025Guowei Zhang, Yun Zhao, Kai Sun +4Visual Neuroscience Conditional Image Generation
Dept of Data Science and AI, Monash University, Melbourne, Australia · Brain and Mind Centre, University of Sydney, Sydney, Australia · School of Psychological Sciences, Monash University, Melbourne, Australia

May 3, 2025J/K move · Enter open · S save

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects

Christos Zangos, Danish Ebadulla, Thomas Christopher Sprague, Ambuj Singh

Department of Computer Science, University of California, Santa Barbara, USA · Department of Psychological and Brain Sciences, University of California, Santa Barbara, USA

fMRI-Based Visual Decoding fMRI Representation Learning Few-Shot Domain Adaptation Latent Space Alignment

Open paper AskPDF arXiv

Reconstructing visual images from fMRI data is a central but challenging problem in neuroscience. Despite recent progress, current methods fall short when data and computation are limited-precisely the conditions under which this task is most critical. We introduce Adapter Alignment, a staged, architecture-agnostic training procedure for adapting a pre-trained reconstruction model to a new subject with little data. Rather than letting a shared representation space emerge implicitly, lightweight subject-specific adapters are first explicitly aligned to a pre-trained reference subject on a small set of shared calibration images, and the full pipeline is then fine-tuned while preserving this alignment. The resulting representations not only place different subjects in a shared space but also align them semantically. Our results demonstrate significant gains in low-data regimes, improving reconstruction metrics by up to 8%, and pre-diffusion retrieval shows that these gains reflect additional image-specific information in the fMRI embeddings rather than the generative prior. We also propose a coverage-based algorithm to select representative calibration images. Using both techniques together, we surpass baseline models fine-tuned with one hour of data using only 24 minutes of subject-specific data. The procedure is effective across architectures, reference subjects, and datasets, including adapting 3T subjects to a 7T reference.

Abstract

3D Reconstruction

Explore similar work

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data

Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping