From "What-If" to "What-Is": Counterfactual Thinking-Inspired Semantic Alignment for Visual Brain Decoding

Kaitao Yan; Chi Liu; Congcong Zhu; Huajie Chen; Gengshen Wu; Minghao Wang; Xiaotong Han; Tianqing Zhu

cs.CVAug 15, 2026

From "What-If" to "What-Is": Counterfactual Thinking-Inspired Semantic Alignment for Visual Brain Decoding

Authors: Kaitao Yan, Chi Liu, Congcong Zhu, Huajie Chen, Gengshen Wu, Minghao Wang, Xiaotong Han, Tianqing Zhu

Organizations: Faculty of Data Science, City University of Macau · Zhongshan Ophthalmic Center, Sun Yat-sen University

Abstract

Visual brain decoding reconstructs visual content perceived by a person from neural measurements such as fMRI, providing a computational approach to studying how visual information is represented in the brain. Recent multimodal representations and diffusion priors have improved reconstruction realism. However, visually plausible reconstructions may contain incorrect objects, attributes, or relations because a strong generative prior can complete content not sufficiently specified by the decoded representation. Conventional reconstruction metrics mainly assess the final image and may therefore obscure such semantic errors. We propose ConceptAlign, a counterfactual semantic alignment framework for visual brain decoding. ConceptAlign pools decoded visual tokens and projects them into a frozen text-embedding space, aligning the representation with the ground-truth caption while separating it from scene-preserving near-miss alternatives. Generated offline by an LLM, these alternatives modify one critical object, attribute, or relation while retaining the scene. A margin-based objective learns fine-grained semantic boundaries between the observed stimulus and plausible but incorrect interpretations without requiring LLM calls during inference. We introduce a systematic three-level semantic evaluation framework covering foundational discriminability, counterfactual description discrimination, and representational geometry. Experiments on the Natural Scenes Dataset show that ConceptAlign improves reconstruction measures, counterfactual semantic discrimination, and representational alignment over the MindEye2 backbone. Matched negative-source ablations, independent LLM and human-written alternatives, and human evaluation support the effectiveness and robustness of the supervision, with favorable patterns in fine-grained conflicts, limited-data decoding, and cross-subject structure.

Figures & tables

Figure 1: Motivation and overview of ConceptAlign. ( A ) Conventional visual brain decoding can produce photorealistic reconstructions even when stimulus-specific semantic details drift. ( B ) Inspired by counterfactual thinking, ConceptAlign distinguishes the observed scene from plausible but incorrect alternatives to learn semantic boundaries.

Figure 2: ConceptAlign within the MindEye2 backbone. Subject-specific ridge mappings and a shared decoder map fMRI responses to visual tokens $\mathbf{Z}$ . ConceptAlign pools and projects $\mathbf{Z}$ into a frozen text space, contrasting the ground-truth caption with counterfactual captions generated offline by an LLM. Candidates are filtered before Top- $K$ selection and margin-ranking supervision. The retrieval, low-level, and diffusion-prior branches retain their roles, and no LLM is used at inference.

Figure 3: Semantic evaluation framework: (L1) foundational discriminability with retrieval and 2AFC, (L2) counterfactual description discrimination with CCD-hard, and (L3) representational geometry with RSA.

Figure 4: Semantic fidelity beyond visual realism. Each triplet shows the ground-truth image, the MindEye2 reconstruction, and the ConceptAlign reconstruction. Yellow text marks a semantic anchor derived from the stimulus; red boxes mark missing or incorrect anchor content, and green boxes mark a closer match. Group A focuses on fine-grained details when both methods recover the main scene. Group B shows differences in object identity, count, arrangement, or visible attributes. These selected cases illustrate semantic differences under the stated anchors and are not used to estimate their frequency over the full test set.

Comparison with existing methods
	Low-Level				High-Level			Semantic Evaluation
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex2 $\uparrow$	Alex5 $\uparrow$	Inc. $\uparrow$	CLIP $\uparrow$	SwAV $\downarrow$	CCD-H $\uparrow$	2AFC-B2I $\uparrow$	RSA $\uparrow$
MindEye2 (ICML 2024)	37.40%	43.90%	97.82%	99.10%	96.15%	93.56%	33.80%	62.30%	98.40%	28.05%
Flat-to-Round (ICCVW 2025)	16.50%	30.50%	78.20%	89.00%	85.10%	88.30%	39.80%	–	–	–
MindTuner (AAAI 2025)	32.20%	42.10%	95.80%	98.80%	95.60%	93.80%	34.00%	–	–	–
ZEBRA (NeurIPS 2025)	13.10%	37.50%	74.60%	81.20%	72.20%	71.50%	50.60%	–	–	–

Table 1: Comparison with existing methods and controlled negative source variants on NSD in terms of three-level metrics (%).Bold and underlined values denote the best and second-best results within each block, respectively; tied best results are both bolded. Superscripts in the Ours row report its percentage-point difference from the strongest non-Ours result in the same column. For SwAV, a negative difference is favorable because lower is better. Published methods retain their original training and evaluation settings. Complete ablation values and paired intervals are provided in the Supplement.

Panel of Figure 5 — Figure 5: Subject-level semantic evaluation. ( Top ) CCD@1 on shared982 for the available ConceptAlign and official MindEye2 checkpoints; error bars show 95% bootstrap confidence intervals. ( Bottom ) Brain–image RSA measured by Pearson correlation. CCD comparison is diagnostic because checkpoint availability and budgets differ across subjects.

Panel of Figure 6 — Figure 6: Semantic difficulty and cross-subject structure. ( Left ) CCD@1 relative to chance under counterfactual and random-negative evaluation; random negatives define an easier task. ( Right ) Pairwise cross-subject IS-RSA on shared982. The upper triangle reports MindEye2 and the lower triangle reports ConceptAlign.

Panel of Figure 7 — Figure 7: Data efficiency with limited fMRI training data. ( Top ) CCD@1 and ( bottom ) 2AFC-Hard for Subjects 01 and 05 using one, two, and forty sessions. In both evaluations, the ConceptAlign difference is most visible with one or two sessions, while the full-data results are closer.

Negative source			Margin sensitivity
Variant	Source	Cos. (%)	$m$	CCD@1 (%)	Ret.@1 (%)
Positive-only	None	–	.05	59.20	17.00
Random	Random COCO	34.83	.10	63.60	18.00
CLIP-NN	Nearest CLIP	81.62	.20	62.10	18.00
ConceptAlign	CF near-miss	$82.00\pm 5.00$	.50	61.00	16.00

Table 2: Negative-source characteristics and margin sensitivity. Cos. denotes caption-to-positive cosine similarity. All percentage-valued columns are reported in percent. Ret.@1 denotes forward retrieval. The two table blocks are independent; rows are aligned only for compact presentation.

Human audit ( $N=534$ )
Scene preserved (%)		97.90	Near-miss score (1–5)		3.55
Explicit negation (%)		0.20	Harmful/bias (%)		0.00
Independent evaluation (Acc@1, %)
Set	N	Ours	Ref.	$\Delta$ (pp)	95% CI (pp)
Cross-LLM	979	35.24	36.06	-0.82	$[-3.17,1.53]$
Human-written	100	58.00	61.00	-3.00	$[-11.00,4.00]$

Table 3: Human audit and independent counterfactual evaluation. Human-audit rates and Acc@1 are reported in percent; differences and confidence intervals in percentage points (pp). Ref. denotes the strongest matched control for Cross-LLM and Human-written evaluation and CLIP-nearest for the Human-audited set. Scene preservation denotes the percentage of audited negatives judged to preserve the scene.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Negatives / images	534 / 178
Annotators	3
Total ratings	1,602
Task	Rate each caption by dimension
Use in training	No

Appendix

Table A1: Human Audit protocol. Results are reported for each rating dimension.

Metric (%)	Ours	C1	C2	C3
PixCorr	39.30	27.26	27.05	27.43
SSIM	44.50	22.13	22.41	23.39
AlexNet L2	97.70	49.32	49.37	48.84
AlexNet L5	99.30	43.60	43.94	43.66

Appendix

Table B2: Reconstruction results for the controlled negative-source study on Subject 01 (

N=982

). All variants are evaluated with the same metric implementation.

Variant	Negative source	Cosine (%)
C1: Ours w/o counterfactual	None	–
C2: Ours + random negative	Random COCO caption	34.83
C3: Ours + CLIP-nearest	Nearest CLIP caption	81.62
Ours (ConceptAlign)	Counterfactual near-miss	$82.00\pm 5.00$

Appendix

Table B3: Negative sources used in the controlled study. Cosine similarity is multiplied by 100 for presentation.

Metric	Comparison	$\Delta$ (pp)	CI low (pp)	CI high (pp)
CCD-H	Ours–C1	-0.34	-2.69	2.13
CCD-H	Ours–C2	-0.11	-2.58	2.36
CCD-H	Ours–C3	0.11	-2.36	2.47
2AFC B2I	Ours–C1	0.61	-1.12	2.34
2AFC B2I	Ours–C2	0.31	-1.43	2.14
2AFC B2I	Ours–C3	0.71	-1.02	2.55

Appendix

Table B4: Paired intervals for CCD-H and brain-to-image 2AFC in the controlled negative-source study.

S	Model	F@1 (%)	F@5 (%)	B@1 (%)	B@5 (%)
1	MindEye2	15.00	41.00	5.00	13.00
1	Ours	18.00	45.00	8.00	26.00
2	MindEye2	16.00	41.00	4.00	14.00
2	Ours	14.00	41.00	7.00	23.00
5	MindEye2	38.00	74.00	22.00	52.00
5	Ours	21.00	54.00	16.00	42.00

Appendix

Table B5: Multi-subject pooled retrieval on the 982 shared test images. F and B denote forward and backward retrieval. Values are reported in percent.

Subj	Model	B $\rightarrow$ I (%)	I $\rightarrow$ B (%)
1	MindEye2	97.80	84.70
1	Ours	98.40	96.40
2	MindEye2	97.70	85.00
2	Ours	98.10	96.10
5	MindEye2	99.50	98.10
5	Ours	98.90	98.30

Appendix

Table B6: Multi-subject 2AFC results on the 982 shared test images. Values are reported in percent.

Type (N)	S	MindEye2 (%)	Ours (%)	Higher
Object (682)	1	58.50	65.00	Ours
Object (682)	2	61.30	63.90	Ours
Attribute (182)	1	56.60	57.70	Ours

Appendix

Table B7: CCD results by semantic edit type. Values are reported in percent.

Set	N	Ours (%)	C1 (%)	C2 (%)	C3 (%)
Cross-LLM K2	979	35.24	35.04	36.06	35.75
Human-written	100	58.00	59.00	59.00	61.00
Human-audited	178	78.09	74.72	78.09	71.91

Appendix

Table B8: Image-level Acc@1 (%) on the three evaluation-only counterfactual sets.

Set	Comparison	$\Delta$ (pp)	CI low (pp)	CI high (pp)
Cross-LLM	Ours–C1	+0.20	-2.04	+2.35
Cross-LLM	Ours–C2	-0.82	-3.17	+1.53
Cross-LLM	Ours–C3	-0.51	-3.47	+2.35
Human-written	Ours–C1	-1.00	-7.00	+5.00
Human-written	Ours–C2	-1.00	-7.00	+5.00
Human-written	Ours–C3	-3.00	-11.00	+4.00

Appendix

Table B9: Paired Acc@1 differences.

\Delta

is Ours minus the stated variant. Intervals use 10,000 image-level bootstrap resamples with seed 2027.

Dimension	Result	N
Scene preserved	97.90%	534
Implausible replacement (majority)	6.90%	533
Near-miss mean / median	3.55 / 3.67	534
Invalid edit type	0.00%	527
Other edit type	0.00%	527
Explicit negation	0.20%	534

Appendix

Table B10: Human Audit outcomes by dimension. Seven edit-type items were undetermined.

Dimension	Pairwise (%)	Unanimous (%)
Scene preservation	83.35	75.00
Semantic plausibility	79.49	69.25
Edit type	68.59	53.48
No explicit negation	98.49	97.74
Harmful/bias flag	99.24	98.85

Appendix

Table B11: Raw agreement for the categorical Human Audit ratings. Pairwise agreement is the proportion of equal observed rating pairs; unanimous agreement requires all three ratings to match.

Figure B2: Representational similarity matrices for Subject 01 using 200 images from the shared test set. Left: target-image RSM. Middle: ConceptAlign-decoded RSM. Right: element-wise difference. The panels use separate color ranges and illustrate the matrix structure.

Metric	MindEye2 (%)	Ours (%)	$\Delta$ (pp)
2AFC I $\rightarrow$ B	$90.68\pm 6.40$	$96.78\pm 0.99$	$+6.10\pm 5.43$
2AFC B $\rightarrow$ I	$98.40\pm 0.83$	$98.38\pm 0.33$	$-0.03\pm 0.69$
CCD-H	$62.30\pm 3.84$	$63.65\pm 1.62$	$+1.35\pm 2.98$
RSA Pearson	$28.05\pm 2.40$	$35.50\pm 1.30$	$+7.46\pm 1.91$

Appendix

Table B12: Four-subject mean

\pm

standard deviation on the 982 shared test images. Values are reported in percent, and differences are in percentage points.

Subj / $\Delta$ (pp)	1	2	5	7
1	–	+6.50	+7.63	+8.88
2	+6.50	–	+6.10	+6.87
5	+7.63	+6.10	–	+5.67
7	+8.88	+6.87	+5.67	–

Appendix

Table B13: Pairwise IS-RSA differences in percentage points (

\Delta=\text{Ours}-\text{MindEye2}

after multiplying correlations by 100).

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 180 papers

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data
Apr 15, 2026Fabrizio Spera, Tommaso Boccato, Michal Olak +2Neural Decoding Visual Neuroscience
Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA
Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding
Apr 12, 2026Mohammad Moradi, Morteza Moradi, Marco Grassia +1fMRI-Based Visual Decoding Neural Decoding
University of Catania, Catania, Italy
Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity
May 28, 2026Yizhuo Lu, Changde Du, Qiongyi Zhou +2Neural Decoding Image Reconstruction
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · School of Future Technology, University of Chinese Academy of Sciences, Beijing 100049, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.

Apr 15, 2026J/K move · Enter open · S save

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data

Fabrizio Spera, Tommaso Boccato, Michal Olak, Nicola Toschi+1

Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA

Neural Decoding Visual Neuroscience Neuroimaging Latent Space Alignment fMRI-Based Visual Decoding

Open paper AskPDF arXiv

Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the NSD-Imagery benchmark. We propose a latent functional alignment (LFA) approach that maps imagery-evoked activity to the pretrained model's semantic content-enriched conditioning space, by adding a simple alignment module, while keeping the original remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a neural retrieval-based augmentation strategy that selects semantically related NSD perception trials from the same participants. Across four subjects, LFA consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.

Abstract

Neural Decoding

Explore similar work

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data

Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding

Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity