The microscope is the mask: privileged views and labels from a cryo-ET forward model

Bogdan Toader; Kiarash Jamali; Tanmay A. M. Bharat; Sjors H. W. Scheres

cs.CVSep 3, 2026

The microscope is the mask: privileged views and labels from a cryo-ET forward model

Authors: Bogdan Toader, Kiarash Jamali, Tanmay A. M. Bharat, Sjors H. W. Scheres

Organizations: MRC Laboratory of Molecular Biology Cambridge, UK

Abstract

We explore the use of simulated data for training a model for protein annotation in crowded cryo-electron tomography volumes reconstructed from images collected at limited tilt angles and severely corrupted by the measurement operator. Firstly, we leverage the corruptions imposed by the forward model to generate domain-specific augmented paired views of the exact same scene for an invariance objective integrated into the LeJEPA self-supervised training framework. Secondly, we use additional information from the simulation pipeline such as the positions and identity of proteins in the simulated volumes to inform the architecture of the model and the loss function, so that semantic information is localised at protein positions in the resulting dense feature volume. The resulting model, CARNIVAL, is evaluated without finetuning on classification and detection tasks in real tomograms, using a benchmark dataset containing multiple protein types and two tomogram processing types. We show that CARNIVAL outperforms a state-of-the-art model trained using a contrastive objective on simulated data but without forward model-based paired views or privileged information.

Figures & tables

Figure 1: A Slice of real tomogram from [ Kelley et al., 2026 ] . B From one arrangement of proteins and membranes we generate six views (four shown). Two clean views (left) share those structures and differ only in the small-protein background. Each corrupted view (right) is obtained by applying the measurement operator $\Psi$ to the clean view in the same row, at its own parameter draw $\theta_{1}$ or $\theta_{2}$ (e.g. tilt range, contrast transfer function, coloured noise). Blue and pink boxes mark two protein types; the same colour marks the same type in all four panels. Scale bar 100 nm is common to all panels.

Figure 2: CARNIVAL vs TomoTwin : A Classification

F_{1}

score, per protein type and overall (averaged over protein types). CARNIVAL outperforms TomoTwin on filtered tomograms (overall

0.88

0.83

) and remains stable on denoised tomograms, while TomoTwin’s performance drops to

0.74

. B Detection precision-recall on one filtered tomogram ( TS_5_2 ) containing six protein types. Faint curves are individual template draws (5 templates/draw, 30 draws), bold curves their mean. The square marks the best

F_{4}

over the threshold sweep per method. C As the number of proteins in a simulated block increases, CARNIVAL’s

F_{1}

stays high while the baseline degrades.

Tomogram	Filtered		Denoised
	CARNIVAL	TomoTwin	CARNIVAL	TomoTwin
TS_5_2	0.508	0.431	0.529	0.410
TS_14_1	0.389	0.311	0.357	0.311
TS_43_9	0.542	0.437	0.465	0.425

Table 1: Detection results: weighted mean of the best

F_{4}

over the threshold sweep for each evaluated protein type in that tomogram (six, four and three respectively). Per-type values, with their variability across template draws, are given in Table 4 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Model architecture.

Transformation	Form	Sampled from
Measurement operator $M_{\theta_{M}}$
Bulk medium (ice / cytoplasm)	$v\mapsto v+\phi\,(v_{\max}-v_{\min})\,d(v)$ on the volume before projection, with $d(v)=\mathrm{clamp}(1-v/v_{\max},\,0,\,1)$ so structure displaces medium; fill level $\phi=\alpha$	$\alpha\in[0,0.45]$
Large-scale medium variation	$\phi$ generalised to a smooth field $\phi(x)=\alpha\,(1+r\,z(x))$ of the same mean, $z$ zero-mean unit-std low-passed by $e^{-2\pi^{2}\sigma^{2}f^{2}}$ , $\sigma=\ell/\text{pixel size}$	$r\in[0,0.5]$ $\ell\in[400,3200]\,\text{\AA}$
Tilt scheme (missing wedge)	projections at $\theta\in\{-\theta_{\max},\dots,\theta_{\max}\}$ in steps of $\Delta\theta$ ; the wedge follows from $\theta_{\max}$	$\theta_{\max}\in\{45,50,55,60\}^{\circ}$ $\Delta\theta\in\{2,3,5\}^{\circ}$
Defocus (CTF)	$\hat{I}\mapsto C(k;\Delta f_{\theta})\,\hat{I}$ per tilt, with $\Delta f_{\theta}$ varying with depth along the beam	$\Delta f\in[1,5]\,\mu\mathrm{m}$
CTF envelope	$E(k)=\exp(-Bk^{2}/4)$ from partial coherence, tilt-independent, multiplied into the CTF	$B\in[0,300]\,\text{\AA}^{2}$

Appendix

Table 2: The sampled operators in the forward model

\Psi_{\theta}=R_{\theta_{R}}\circ M_{\theta_{M}}

, grouped by the two blocks of the decomposition. Every simulated tomogram draws its own values (one draw per tomogram, shared by all batch_size blocks extracted from it) and each range is chosen to bracket experimental practice rather than to match any one dataset. Held fixed:

10

Å/voxel,

300

kV,

C_{s}=2.7

mm, amplitude contrast

0.1

, and radiation damage, which is physics rather than a processing choice and so is never switched off. The CTF correction-state randomises a processing choice rather than a physical one, as different reconstruction algorithms differ in their CTF treatment. The notation used in each row is independent from the rest of the paper.

	Filtered $F_{1}$		Denoised $F_{1}$		# particles
	CARNIVAL	TomoTwin	CARNIVAL	TomoTwin
ribosome	0.98	0.96	0.98	0.95	10820
VLP	0.95	0.82	0.94	0.78	1061
thyroglobulin	0.82	0.81	0.83	0.69	5368
$\beta$ -galactosidase	0.72	0.68	0.73	0.51	3725
apoferritin	0.98	0.92	0.98	0.93	9149

Appendix

Table 3: Full classification results with

k=10

nearest neighbours. The

F_{1}

score is shown for each protein type and the overall score is averaged over protein types. The same numbers shown in Figure 2 A.

TS_5_2
	Filtered		Denoised
Protein type	CARNIVAL	TomoTwin	CARNIVAL	TomoTwin
ribosome	$\mathbf{0.878\pm 0.018}$	$0.849\pm 0.023$	$\mathbf{0.790\pm 0.043}$	$0.607\pm 0.049$
VLP	$\mathbf{0.917\pm 0.021}$	$0.536\pm 0.217$	$\mathbf{0.922\pm 0.018}$	$0.470\pm 0.149$
thyroglobulin	$0.417\pm 0.134$	$\mathbf{0.518\pm 0.110}$	$\mathbf{0.494\pm 0.121}$	$0.415\pm 0.105$
$\beta$ -galactosidase	$0.327\pm 0.086$	$\mathbf{0.354\pm 0.112}$	$\mathbf{0.401\pm 0.095}$	$0.375\pm 0.119$

Appendix

Table 4: Per-type detection results. Each entry is the maximum

F_{4}

over the threshold sweep, taken on the precision-recall curve averaged over the 30 template draws,

\pm

one standard deviation of the individual draws’

F_{4}

at that same threshold. These are the operating points marked in Figures 4 – 6 . Bold marks the higher of the two values within a processing condition. The weighted-mean rows reproduce the corresponding entries of Table 1 .

Figure 4: CARNIVAL vs TomoTwin : Detection results, tomogram TS_5_2 . A Filtered. This panel is identical to Figure 2 B, reproduced here for easy visual comparison between the filtered and denoised tomograms. B Denoised.

Figure 5: CARNIVAL vs TomoTwin : Detection results, tomogram TS_14_1 . A Filtered. B Denoised.

Figure 6: CARNIVAL vs TomoTwin : Detection results, tomogram TS_43_9 . A Filtered. B Denoised.

Figure 7: Slices of tomograms and spatial feature volumes from CARNIVAL, with the filtered version on the top row and the denoised version on the bottom row in each panel. As the spatial feature blocks have 64 channels, we show the 3 largest principal components of each feature vector, mapped to RGB, where the principal components were calculated using all feature volumes at the same time. Scale bar 100 nm is common to all panels. A Tomogram TS_5_2 . B Tomogram TS_14_1 . C Tomogram TS_43_9 .

Protein Representation Learning

Protein representation learning builds computational features that capture sequence, structure, function, and molecular relationships.

See all 64 papers

POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET
Jun 8, 2026Jonathan Schwartz, Utz Heinrich Ermel, C. Braxton Owens +6Image Segmentation Cryo-Electron Microscopy
1Biohub, Redwood City, CA 94063, USA · 2Brigham Young University, Provo, UT, 84602, USA
Learning the Shared Latent Structure of Cryo-EM Inverse Problems
Oct 15, 2026Nithin Chalapathi, Theo Sternlieb, Ethan Han +4Cryo-Electron Microscopy Image Inverse Problems
University of California, Berkeley · Flatiron Institute · University of Maryland, College Park
Atelier: Learning Local Self-Supervised Features for CryoEM Volumes via Hypernetworks
Sep 24, 2026Phillip Lo, Sudarshan Babu, Dari Kimanius +13D Representation Learning Cryo-Electron Microscopy
Biohub Chicago, IL · University of Chicago, Biohub Chicago, IL

Jun 8, 2026J/K move · Enter open · S save

POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET

Jonathan Schwartz, Utz Heinrich Ermel, C. Braxton Owens, Zhuowen Zhao+5

1Biohub, Redwood City, CA 94063, USA · 2Brigham Young University, Provo, UT, 84602, USA

Image Segmentation Cryo-Electron Microscopy Microscopy Image Analysis Electron Microscopy

Open paper AskPDF arXiv

Cryo-electron tomography (cryoET) has emerged as a powerful tool in structural and cellular biology by enabling direct visualization of macromolecular structures within intact cells, thereby linking molecular architecture to cellular organization in a native context. Realizing the full potential of cryoET, however, increasingly depends on advances in computational analysis, particularly machine learning (ML), to interpret its complex and information-rich data. Despite rapid progress, ML development for cryoET remains bottlenecked by the lack of standardized, well-annotated benchmarks. Existing evaluations are typically small, task-specific, and are assembled in isolation, limiting robust comparisons across methods. Here, we present POPSICLE, a benchmark suite for cryoET segmentation and macromolecular localization built from the CryoET Data Portal - an open, ML-ready repository of tomographic data, metadata, and annotations. POPSICLE spans eukaryotic and prokaryotic systems, both purified and fully in situ samples, and dense voxel-wise segmentation as well as sparse localization tasks. Built on a living data resource, it can expand as new datasets and annotations become available. Baseline experiments reveal substantial variation in model rankings across tasks, underscoring the need for benchmarks tailored to the unique characteristics of cryoET rather than evaluation practices adapted from adjacent biomedical imaging domains. POPSICLE thus provides an open and extensible foundation for reproducible ML evaluation in cryoET.

Abstract

Protein Representation Learning

Explore similar work

POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET

Learning the Shared Latent Structure of Cryo-EM Inverse Problems

Atelier: Learning Local Self-Supervised Features for CryoEM Volumes via Hypernetworks