Organizations: Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · Monash University, Melbourne, Australia. · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA
Visual brain decoding from functional magnetic resonance imaging (fMRI) data has made remarkable progress in recent years, driven by increasingly powerful artificial intelligence (AI) models and the growing availability of large-scale public datasets. However, despite the impressive advances achieved in static image decoding, the reconstruction of videos from fMRI signals remains challenging due to the temporal mismatch between rapidly changing video stimuli and the slow dynamics of the blood-oxygen-level-dependent (BOLD) response. We evaluate a Bayesian-inspired proposal-and-rerank strategy on CC2017 and the BOLD Moments Dataset (BMD), using a linear semantic decoder, a pretrained video generator, and three neural encoding streams with ridge readouts and learned pooling in two streams. We refer to this framework as REEL (Reranking Encoder Embeddings with Linear decoding). Compared with a single generated candidate, reranking increases mean Video 50-way accuracy from 0.2359 to 0.2434 across three CC2017 participants and from 0.3653 to 0.3920 across ten BMD participants. These findings suggest an alternative and conceptually simpler strategy for improving video brain decoding, highlighting the potential of our approach as a complementary direction to increasingly complex pipelines.
Figures & tables
Figure 1: Overview of the proposed decoding framework for both CC2017 and BMD. A linear ridge decoder maps the observed neural response y to a point estimate c^ in the CogVideoX text-conditioning space, defining a brain-conditioned proposal PG(x∣c^) . Multiple candidate videos are sampled from this proposal and passed through early, ventral and dorsal neural encoding streams. Their agreement with the observed fMRI response provides neural compatibility scores, and the highest-scoring candidate is selected as the final reconstruction.
Table 1: Comparison between REEL and previous fMRI-to-video methods, grouped by dataset. REEL values are averaged across three subjects for CC2017 and ten for BMD; participant coverage of published baselines follows the original studies, with Brain Netflix evaluated on subject 1 . Bold entries indicate numerical maxima among available comparisons, without implying matched statistical superiority. The rows labelled K=1 evaluate the semantic decoder together with the pretrained generator, using one candidate per stimulus without reranking and report the mean across 5 different measures, as done in Section 5.1 . On CC2017, NEURONS results were obtained by reproducing its original pipeline, while NeuroClips and MinD-Video results are taken from [ 14 , 36 ] and SemVideo results from [ 17 ] ; on BMD, published values are taken from [ 13 , 16 ] . Brain Netflix reports ImageNet-based 2-way/50-way scores of 0.767 / 0.145 (single-subject training) and 0.816 / 0.165 (multi-subject training), both evaluated on subject 1 . SemVideo reports a CLIP score of 0.526 with a VIFI-based gate, so it is not included under our ungated CLIP-pcc definition. A dash denotes an unreported or non-comparable value.
Figure 2: Manually selected best video reconstructions for CC2017. These frames show a strong correspondence between the original scene shown in the video and our reconstructions, capturing the overall layout, color palette and coarse semantic category.
Figure 3: Manually selected best video reconstructions for BMD. These frames show a strong correspondence between the original scene displayed on the video and our reconstructions, capturing the overall layout, color palette and coarse semantic category.
Figure 4: Effect of candidate-set size on reconstruction performance on CC2017. Paired difference between brain-guided reranking and random selection for Video 50-way accuracy, Frame 50-way accuracy and SSIM, as a function of the number of candidates K . For each subject and K , the difference is the metric of the reranked candidate minus the random expectation (the mean over the K candidates of each nested subset), computed with per-candidate evaluation metrics and averaged over five subset permutations, which are not treated as independent replicates. Thin lines show the three subjects; the thick line shows their mean, with a 95% bootstrap confidence interval obtained by resampling blocks of 20 consecutive test clips within each movie, jointly across subjects. At K=1 the two strategies coincide. With three subjects, these results are descriptive.
Figure 5: Effect of candidate-set size on reconstruction performance on BMD. Paired difference between brain-guided reranking and random selection for Video 50-way accuracy, Frame 50-way accuracy and SSIM, as a function of the number of candidates K . For each subject and K , the difference is the metric of the reranked candidate minus the random expectation (the mean over the K candidates of each nested subset), computed with per-candidate evaluation metrics and averaged over five subset permutations, which are not treated as independent replicates. Thin lines show the ten subjects; the thick line shows their mean, with a 95% bootstrap confidence interval obtained by resampling test clips jointly across subjects. At K=1 the two strategies coincide.
Table 2: Contribution of each encoding stream and of each pair of streams on CC2017 (averaged over 3 subjects) and BMD (averaged over 10 subjects). Values report the relative change (%) with respect to the full model, which combines all three streams (bottom row, absolute values); a negative value indicates that removing the remaining stream(s) degrades reconstruction quality. All configurations select among the same K=100 candidates. Markers denote uncorrected paired t -tests across subjects ( df=2 for CC2017, df=9 for BMD) and should be read as exploratory: * p<0.05 , ** p<0.01 ; no marker denotes p≥0.05 .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Table 3: Summary of data splits and preprocessing for the two datasets.
Table 4: Assignment of HCP-MMP1 areas [ 45 ] to the three visual streams. All streams are bilateral.
Figure 6: Bayesian motivation for our proposal-and-rerank strategy. Classical Bayesian decoding combines a broad prior over natural videos with neural evidence. Direct candidate search by unconditional sampling from this prior is impractical. Our approach instead uses the observed brain response to construct a semantically restricted proposal distribution, samples a tractable set of candidate videos, and reranks them using neural encoding models.
Table 5: Effect of the candidate-set size K on reconstruction quality on CC2017, with encoding-model reranking versus random selection. Values are means across the three subjects, averaged over five nested random permutations of fixed candidate pools; ± denotes permutation standard deviation only. At K=1 reranking and random selection coincide. At K=100 both are invariant to these permutations, so their permutation standard deviation is zero.
Table 6: Effect of the candidate-set size K on reconstruction quality on BMD, with encoding-model reranking versus random selection. Values are means across the ten subjects, averaged over five nested random permutations of fixed candidate pools; ± denotes permutation standard deviation only. At K=1 reranking and random selection coincide. At K=100 both are invariant to these permutations, so their permutation standard deviation is zero.
Table 7: Shuffled-response and conditioning-only controls. Values are mean ± SEM across subjects (3 for CC2017, 10 for BMD). Random: expected value of a uniform choice among the K=100 candidates of the same pool (mean over candidates). Shuffled: candidates scored against the response to another test stimulus (mean of five permutations; for CC2017, always from a different movie). Conditioning only: candidate most similar to the decoded conditioning, without the measured response. Paired t -test across subjects of reranked versus shuffled: * p<0.05 , ** p<0.01 , *** p<0.001 ; reranked versus conditioning only: † p<0.05 , †† p<0.01 .
Table 8: Comparison between the baseline and our method, averaged over the three subjects. Values are mean ± SEM across subjects. The best value per metric is in bold. * p<0.05 , ** p<0.01 , *** p<0.001 paired t -test across subjects ( df=2 ).
Figure 7: Paired subject-level differences in mean performance with respect to the NEURONS baseline on CC2017, for our reranking method applied to the NEURONS candidate pool (blue) and for our proposed pipeline (orange). Each panel reports one metric on its own scale (top row: semantic metrics; bottom row: structural metrics); the vertical line at zero corresponds to the NEURONS baseline. Diamonds denote the mean difference across the three subjects and whiskers its 95 % confidence interval (paired t -test, df=2 ); filled markers indicate intervals that exclude zero, and faint dots show the individual subjects. * p<0.05 , ** p<0.01 , *** p<0.001 .
Figure 8: Best reconstructions on CC2017. These frames show a strong correspondence between the original scene shown in the video and our reconstructions, capturing the overall layout, color palette and coarse semantic category.
Figure 9: Intermediate reconstructions on CC2017. The reconstructions correctly reproduce the subjects and the environment of the original videos, but show limitations with out-of-distribution elements, such as the moon appearing in the sky, and with scene transitions. Even in these cases, the general setting, such as the presence of a dark sky, is preserved.
Figure 10: Worst reconstructions on CC2017. The reconstructions fail to reproduce the subjects and the environment of the original videos, mostly because these stimuli are out of distribution for our model. Nevertheless, coarse properties such as the presence of an action or the number of subjects in the scene still appear to be captured.
Figure 11: Best reconstructions on BMD. These frames show a strong correspondence between the original scene displayed on the video and our reconstructions, capturing the overall layout, color palette and coarse semantic category.
Figure 12: Intermediate reconstructions on BMD. The reconstructions correctly reproduce the subjects and the environment of the original videos, but show limitations. However, even in these cases, the general setting and the number of subjects resemble the corresponding ground truth, is preserved.
Figure 13: Worst reconstructions on BMD. The reconstructions fail to reproduce the subjects and the environment of the original videos, mostly because these stimuli are out of distribution for our model.