Toward Generalizable Whole Brain Representations with High-Resolution Light-Sheet Data

Minyoung E. Kim; Dae Hee Yun; Aditi V. Patel; Madeline Hon; Webster Guan; Taegeon Lee; Brian Nguyen

cs.CVMar 31, 2026

Toward Generalizable Whole Brain Representations with High-Resolution Light-Sheet Data

Authors: Minyoung E. Kim, Dae Hee Yun, Aditi V. Patel, Madeline Hon, Webster Guan, Taegeon Lee, Brian Nguyen

Organizations: LifeCanvas Technologies

Abstract

Unprecedented visual details of biological structures are being revealed by subcellular-resolution whole-brain 3D microscopy data, enabled by recent advances in intact tissue processing and light-sheet fluorescence microscopy (LSFM). These volumetric data offer rich morphological and spatial cellular information, however, the lack of scalable data processing and analysis methods tailored to these petabyte-scale data poses a substantial challenge for accurate interpretation. Further, existing models for visual tasks such as object detection and classification struggle to generalize to this type of data. To accelerate the development of suitable methods and foundational models, we present CANVAS, a comprehensive set of high-resolution whole mouse brain LSFM benchmark data, encompassing six neuronal and immune cell-type markers, along with cell annotations and a leaderboard. We also demonstrate challenges in generalization of baseline models built on existing architectures, especially due to the heterogeneity in cellular morphology across phenotypes and anatomical locations in the brain. To the best of our knowledge, CANVAS is the first and largest LSFM benchmark that captures intact mouse brain tissue at subcellular level, and includes extensive annotations of cells throughout the brain.

Figures & tables

Panel of Figure 2 — Figure 2: CANVAS Dataset. (a) Overview of datasets in CANVAS showing six cell type markers imaged using light sheet fluorescence microscopy. Markers include NeuN (cyan), IBA1 (magenta), GFAP (yellow), TH (red), cFOS (green), and PV (grey). All markers except PV are based on immunolabeling; PV is transgenically labeled with fluorescent proteins. All datasets represent whole-brain imaging, except cFOS, which is hemisphere-only. Images are 500 µm maximum intensity projections. Scale bar: 2 mm. (b) Zoomed-in views from the CANVAS dataset showing six cell type markers across brain regions, with single-cell insets at bottom right of each panel: NeuN (cyan), IBA1 (magenta), GFAP (yellow), TH (red), cFOS (green), and PV (grey), presented as 80 µm maximum intensity projections. Images include the following brain regions: HIP (hippocampus), VTA (ventral tegmental area), CP (caudate putamen), ZI (zona incerta), CEA (central amygdalar nucleus), RT (reticular nucleus of the thalamus), SS (somatosensory areas), VIS (visual areas), LC (locus coeruleus), MO (somatomotor areas), and mPFC (medial prefrontal cortex); and fiber tracts: cc (corpus callosum) and fr (fasciculus retroflexus). Scale bar: 100 µm.

Marker	ROI	Size (px)	# Cells
NeuN	HIP	$300\times 300\times 150$	7,233
	CTX(SSp)	$300\times 300\times 150$	1,766
	CP (striatum)	$300\times 300\times 150$	8,321
	Summary		17,320, 5,773 $\pm$ 2,868
cFos	HIP	$500\times 500\times 250$	3,782
	CTX(MOp)	$300\times 300\times 150$	2,827

Table 1: Regions selected for annotation in each dataset, with per-marker summary statistics (HIP: Hippocampal region, SSP: Primary somatosensory area, CP: Caudoputamen, MOp: Primary motor area, CEA: Central amygdalar nucleus, VTA: Ventral tegmental area, LC: Locus ceruleus, ZI/A13: Dopaminergic A13 group, RT: Reticular nucleus of the thalamus, mPFC: medial Prefrontal cortex, VIS: Visual cortex).

Marker	NeuN	cFos	TH	PV	GFAP	IBA1
Tolerance	6.0	6.0	8.0	8.0	8.0	5.0

Table 2: Tolerance threshold (in voxels) per dataset. Each threshold was applied across all regions per marker dataset.

	cFos Model		NeuN Model		TH Model		PV Model		GFAP Model		Iba1 Model
Model	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
cFos	0.64	0.78	0.62	0.76	0.15	0.26	0.59	0.74	0.17	0.29	0.58	0.74
NeuN	0.01	0.02	0.67	0.81	0.12	0.21	0.57	0.73	0.02	0.04	0.40	0.57
TH	0.02	0.04	0.25	0.41	0.40	0.57	0.11	0.20	0.08	0.14	0.16	0.28
PV	0.16	0.28	0.81	0.89	0.52	0.68	0.46	0.63	0.12	0.21	0.45	0.62
GFAP	0.00	0.00	0.03	0.05	0.02	0.04	0.00	0.01	0.20	0.33	0.44	0.61

Table 3: Cross-dataset performance of each model across cell-type datasets. ( Bold indicates evaluation on matched data; underline indicates the highest-performing model.)

Panel of Figure 3 — Figure 3: 3D-MAE training results. Top: Convergence (loss, PSNR, SSIM) comparison across all markers plus the all-markers model (dashed lines) using the best configuration (16 $\times$ 32 $\times$ 32/4 $\times$ 8 $\times$ 8, $m=0.15$ ). Bottom: Effect of crop/patch size. GFAP dataset shows the best performance on (24 $\times$ 48 $\times$ 48/6 $\times$ 12 $\times$ 12), while IBA1 and NeuN datasets show the smallest configuration (blue) outperforms larger alternatives.

Panel of Figure 4 — Figure 4: Example $\beta$ -amyloid (yellow) segmentation results for (a) ground truth, (b) $\mu$ SAM, and (c) CellPose-SAM. (d) $\mu$ SAM segmentation overlaid with GFAP (magenta), nuclear stain (cyan), and detected nuclei cell centers (yellow).

Component	Configuration	Parameters
Encoder		10.75M
Hidden dimension	384
Number of layers	6
Attention heads	6
MLP ratio	4
Decoder		1.90M

Table 5: 3D-MAE Model Configuration

Architecture
Parameter	Value
Crop sizes	16×32×32, 24×48×48, 32×64×64
Patch sizes	4×8×8, 6×12×12, 8×16×16
Mask ratios	0.15, 0.35, 0.55, 0.75
Optimization
Optimizer	AdamW

Table 6: Complete Hyperparameter Configuration

Marker	Config	Mask	Loss $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
NeuN	16 $\times$ 32 $\times$ 32 / 4 $\times$ 8 $\times$ 8	0.15	0.0061	12.00	0.256
cFos	16 $\times$ 32 $\times$ 32 / 4 $\times$ 8 $\times$ 8	0.55	0.0082	14.55	0.291
TH	16 $\times$ 32 $\times$ 32 / 4 $\times$ 8 $\times$ 8	0.55	0.0143	13.38	0.295
PV	16 $\times$ 32 $\times$ 32 / 4 $\times$ 8 $\times$ 8	0.15	0.0087	12.69	0.199
IBA1	16 $\times$ 32 $\times$ 32 / 4 $\times$ 8 $\times$ 8	0.15	0.0103	15.61	0.194
GFAP	16 $\times$ 32 $\times$ 32 / 4 $\times$ 8 $\times$ 8	0.55	0.0194	13.74	0.360

Table 7: 3-MAE training results. Best configuration (16

\times

\times

32 / 4

\times

\times

8).

Weighting Strategy	Final Loss	PSNR (dB)	SSIM
Uniform ( $w_{i}=1.0$ for all)	0.0068	11.42	0.231
Content-aware ( $w_{i}\in[1,10]$ )	0.0061	12.00	0.256
Relative improvement	+11.5%	+5.1%	+10.8%

Table 8: Content-Aware Weighting Ablation (NeuN, 16×32×32 / 4×8×8, m=0.15)

Panel of Figure 5 — Figure 5: NeuN reconstruction with varying mask ratios. Compact neuronal nuclei reconstructed with 16×32×32 crop, 4×8×8 patch. Top to bottom: mask ratio 0.15, 0.55, 0.75. Lower masking preserves more morphological detail.

Panel of Figure 6 — Figure 6: NeuN reconstruction with varying crop sizes. Mask ratio 0.15. Top: 24×48×48 crop (6×12×12 patch). Bottom: 32×64×64 crop (8×16×16 patch). Larger crops capture more spatial context.

Panel of Figure 7 — Figure 7: IBA1 reconstruction with varying mask ratios. Ramified microglial morphology with fine processes. 16×32×32 crop, 4×8×8 patch.

Panel of Figure 8 — Figure 8: cFos reconstruction with varying mask ratios. Sparse activity-dependent signal. 16×32×32 crop, 4×8×8 patch.

Panel of Figure 9 — Figure 9: GFAP reconstruction with varying mask ratios. Complex astrocyte morphology with elongated processes. 16×32×32 crop, 4×8×8 patch.

Panel of Figure 10 — Figure 10: PV reconstruction with varying mask ratios. Sparse interneurons with distinct morphology. 16×32×32 crop, 4×8×8 patch.

Panel of Figure 11 — Figure 11: TH reconstruction with varying mask ratios. Elongated dopaminergic neuron morphology. 16×32×32 crop, 4×8×8 patch.

Panel of Figure 12 — Figure 12: All-markers universal model reconstructions. Training on diverse cell types (60k patches from all 6 markers) produces a generalizable encoder maintaining quality across morphologies. 16×32×32 crop, 4×8×8 patch.

Microscopy Image Analysis

Microscopy image analysis uses computational methods to detect, segment, track, reconstruct, and quantify structures in scientific images.

See all 198 papers

A Multimodal 3D Foundation Model for Light Sheet Fluorescence Microscopy Enables Few-Shot Segmentation, Classification, and Deblurring
May 25, 2026Adina Scheinfeld, Haotan Zhang, Shang Mu +53D Semantic Segmentation Microscopy Image Analysis
Tri-Institutional Program in Computational Biology & Medicine, Weill Cornell Medicine, New York, NY, USA · Department of Radiology, Weill Cornell Medicine, New York, NY, USA · Helen and Robert Appel Alzheimers Disease Research Institute, Feil Family Brain and Mind Research Institute, Weill Cornell Medicine, New York, NY, USA +4
Deep Learning for BioImaging: What Are We Really Learning?
Mar 10, 2026Ivan Svatko, Maxime Sanchez, Ihab Bendidi +2Representation Learning Microscopy Image Analysis
Université Paris Cité, IRD, Inserm, MERIT, F-75006, Paris, France · IBENS, École Normale Supérieure, Université PSL, Paris, France · Institut Curie, Université PSL, Paris, France +4
ConnectomeBench2: A Unified Benchmark for Automated Connectomic Proofreading
Jun 19, 2026Jeff Brown, Tim Farkas, Gleb Razgar +1
MIT · The Open University · Mindspan Institute +2

May 25, 2026J/K move · Enter open · S save

A Multimodal 3D Foundation Model for Light Sheet Fluorescence Microscopy Enables Few-Shot Segmentation, Classification, and Deblurring

Adina Scheinfeld, Haotan Zhang, Shang Mu, Rudolf L. M. van Herten+4

Tri-Institutional Program in Computational Biology & Medicine, Weill Cornell Medicine, New York, NY, USA · Department of Radiology, Weill Cornell Medicine, New York, NY, USA · Helen and Robert Appel Alzheimers Disease Research Institute, Feil Family Brain and Mind Research Institute, Weill Cornell Medicine, New York, NY, USA+4

3D Semantic Segmentation Microscopy Image Analysis Fluorescence Microscopy

Open paper AskPDF arXiv

Light sheet fluorescence microscopy (LSM) enables high-resolution, three-dimensional (3D) imaging of biological specimens, providing rich volumetric data for studying cellular organization, pathology, and vascular networks. However, the size, dimensionality, and annotation burden of LSM data make supervised deep learning approaches costly and difficult to scale. Additionally, despite the abundance of unannotated LSM volumes, foundation models for this modality remain underexplored due to computational challenges and the complexity of volumetric representation learning. In this work, we introduce a 3D foundation model for LSM data, pretrained on a large curated collection of 3D images spanning multiple organisms, stains, and imaging protocols. We learn transferable volumetric representations by jointly optimizing for masked reconstruction and image-text alignment. The pretrained backbone drastically reduces the annotation burden, enabling efficient, few-shot adaptation for varied downstream tasks. We evaluate this approach on downstream segmentation, classification, and deblurring. Our results demonstrate consistent improvements over baselines, (1) when measured using standard evaluation metrics and (2) when rigorously assessed by domain experts. This highlights the potential of foundation model pretraining to reduce annotation requirements while improving performance across diverse LSM analysis tasks. Pretrained model weights and code for pretraining and finetuning are publicly available: https://github.com/AdinaScheinfeld/lsm_fm_public_repo.git.

Abstract

Microscopy Image Analysis

Explore similar work

A Multimodal 3D Foundation Model for Light Sheet Fluorescence Microscopy Enables Few-Shot Segmentation, Classification, and Deblurring

Deep Learning for BioImaging: What Are We Really Learning?

ConnectomeBench2: A Unified Benchmark for Automated Connectomic Proofreading