At FullTilt: Real-Time Open-Set 3D Macromolecule Detection Directly from Tilted 2D Projections

Ming-Yang Ho; Alberto Bartesaghi

cs.CVApr 12, 2026

At FullTilt: Real-Time Open-Set 3D Macromolecule Detection Directly from Tilted 2D Projections

Authors: Ming-Yang Ho, Alberto Bartesaghi

Organizations: Duke University

Abstract

Open-set 3D macromolecule detection in cryogenic electron tomography eliminates the need for target-specific model retraining. However, strict VRAM constraints prohibit processing an entire 3D tomogram, forcing current methods to rely on slow sliding-window inference over extracted subvolumes. To overcome this, we propose FullTilt, an end-to-end framework that redefines 3D detection by operating directly on aligned 2D tilt-series. Because a tilt-series contains significantly fewer images than slices in a reconstructed tomogram, FullTilt eliminates redundant volumetric computation, accelerating inference by orders of magnitude. To process the entire tilt-series simultaneously, we introduce a tilt-series encoder to efficiently fuse cross-view information. We further propose a multiclass visual prompt encoder for flexible prompting, a tilt-aware query initializer to effectively anchor 3D queries, and an auxiliary geometric primitives module to enhance the model's understanding of multi-view geometry while improving robustness to adverse imaging artifacts. Extensive evaluations on three real-world datasets demonstrate that FullTilt achieves state-of-the-art zero-shot performance while drastically reducing runtime and VRAM requirements, paving the way for rapid, large-scale visual proteomics analysis. All code and data will be publicly available upon publication.

Figures & tables

Figure 1: Runtime and performance comparison. FullTilt achieves inference speeds hundreds to thousands of times faster than current state-of-the-art open-set macromolecule detection methods, while delivering comparable or superior zero-shot performance across three real-world cryo-ET datasets. The exact acceleration multiplier varies based on the tilt-series and tomogram dimensions.

Figure 2: Comparison of detection mechanisms. We show existing architectures and inputs for open-set 3D macromolecule detection in cryo-ET (prompting modules omitted for simplicity). (a) TomoTwin [ 9 ] and ProPicker [ 11 ] process 3D tomograms using 3D models. Due to strict VRAM limits, they rely on sliding-window subvolume extraction, which requires extensive model I/O and bottlenecks inference speed. (b) CryoSAM [ 10 ] applies a 2D SAM model to orthogonal 2D slices of the 3D tomogram, necessitating repeated, costly model I/O operations. (c) Zeng et al. [ 22 ] apply a 2D detector to individual 2D tilt images and back-project the results into 3D space. This approach suffers from low single-image visibility and severe back-projection errors. (d) FullTilt is the first framework to process the 2D tilt-series directly into 3D detection results in an end-to-end manner.

Figure 3: 2D tilt-series vs. 3D tomogram. We show images at varying tilt angles alongside a corresponding reconstructed tomogram slice (CZII dataset, TS_6_6 ). Yellow boxes denote thyroglobulin particles (top half). (a) High noise makes recognizing individual particles in a single 2D tilt image difficult without viewing consecutive frames. Furthermore, particles often overlap at certain angles, and raw images often exhibit severe physical occlusion (red arrow). (b) Conversely, within a tomogram Z-slice, particles are separated along the depth axis with no spatial overlap.

Figure 4: FullTilt architecture. (a) Overview of the FullTilt architecture. (b) The tilt-series encoder ( $\mathcal{E}_{\text{Tilt}}$ ) utilizes alternating local and global row attention to fuse information from the 2D features into 3D-aware features. (c) The multiclass visual prompt encoder ( $\mathcal{E}_{\text{Prompt}}$ ) accommodates flexible prompting scenarios by leveraging masked attention to generate prototypes for all requested classes. (d) The 3D decoder updates the content queries via self-attention, cross-attention with the prototypes, and deformable cross-attention with the 3D-aware features, using 2D coordinates projected from the 3D anchors.

Category	Method	# Prompts	[email protected] $\uparrow$	mAP@1r $\uparrow$	F1 $\uparrow$	T. $\downarrow$	M. $\downarrow$
General MV	DETR3D [ 19 ] $+\mathcal{E}_{\text{Prompt}}$	1	0.048 ± 0.004	0.091 ± 0.005	0.146 ± 0.006	0.214	3430
General MV	PETR [ 20 ] $+\mathcal{E}_{\text{Prompt}}$		0.000 ± 0.000	0.001 ± 0.000	0.010 ± 0.000	0.571	13800
Cryo-ET	Zeng et al. [ 22 ] $+\mathcal{E}_{\text{Prompt}}$		n.a.	n.a.	0.004 ± 0.001	71.3	4050
	TomoTwin [ 9 ]		0.089 ± 0.021	0.132 ± 0.024	0.078 ± 0.012	2960	20800
	CryoSAM [ 10 ]		n.a.	n.a.	0.008 ± 0.003	309	30600
	ProPicker [ 11 ]		n.a.	n.a.	0.061 ± 0.008	79.5	12700

Table 1: Zero-shot intra-instance detection results on the CZII dataset. The best-performing cryo-ET method for each metric is highlighted in bold. FullTilt achieves the best performance across all evaluated metrics.

Category	Method	# Prompts	[email protected] $\uparrow$	mAP@1r $\uparrow$	F1 $\uparrow$	T. $\downarrow$	M. $\downarrow$
General MV	DETR3D [ 19 ] $+\mathcal{E}_{\text{Prompt}}$	1	0.011 ± 0.001	0.091 ± 0.002	0.222 ± 0.003	0.136	2290
General MV	PETR [ 20 ] $+\mathcal{E}_{\text{Prompt}}$		0.000 ± 0.000	0.002 ± 0.000	0.032 ± 0.001	0.233	6020
Cryo-ET	Zeng et al. [ 22 ] $+\mathcal{E}_{\text{Prompt}}$		n.a.	n.a.	0.012 ± 0.001	30.2	2960
	TomoTwin [ 9 ]		0.189 ± 0.057	0.339 ± 0.063	0.532 ± 0.061	1730	20800
	CryoSAM [ 10 ]		n.a.	n.a.	0.002 ± 0.002	388	28400
	ProPicker [ 11 ]		n.a.	n.a.	0.224 ± 0.014	49.2	12500

Table 2: Zero-shot intra-instance detection results on the EMPIAR-10304 dataset. The best-performing cryo-ET method for each metric is highlighted in bold. FullTilt achieves the best performance with the sole exception of [email protected] in the 2- and 4-prompt settings.

Category	Method	# Prompts	[email protected] $\uparrow$	mAP@1r $\uparrow$	F1 $\uparrow$	T. $\downarrow$	M. $\downarrow$
General MV	DETR3D [ 19 ] $+\mathcal{E}_{\text{Prompt}}$	1	0.039 ± 0.002	0.137 ± 0.004	0.271 ± 0.006	0.198	3230
General MV	PETR [ 20 ] $+\mathcal{E}_{\text{Prompt}}$		0.000 ± 0.000	0.000 ± 0.000	0.017 ± 0.000	0.421	10600
Cryo-ET	Zeng et al. [ 22 ] $+\mathcal{E}_{\text{Prompt}}$		n.a.	n.a.	0.008 ± 0.001	26.9	4710
	TomoTwin [ 9 ]		0.035 ± 0.008	0.113 ± 0.020	0.249 ± 0.027	2330	20800
	CryoSAM [ 10 ]		n.a.	n.a.	0.032 ± 0.002	421	29200
	ProPicker [ 11 ]		n.a.	n.a.	0.187 ± 0.011	69.1	12700

Table 3: Zero-shot intra-instance detection results on the EMPIAR-10499 dataset. The best-performing cryo-ET method for each metric is highlighted in bold. FullTilt achieves the best performance across all evaluated metrics.

			CZII			EMPIAR-10304			EMPIAR-10499
Cat.	Method	# P.	[email protected] $\uparrow$	mAP@1r $\uparrow$	F1 $\uparrow$	[email protected] $\uparrow$	mAP@1r $\uparrow$	F1 $\uparrow$	[email protected] $\uparrow$	mAP@1r $\uparrow$	F1 $\uparrow$
WSL	Huang et al. [ 26 ]	1	-	-	-	-	-	-	-	-	-
WSL	TomoPicker [ 12 ]		0.002 ± 0.001	0.035 ± 0.006	0.061 ± 0.010	0.027 ± 0.030	0.056 ± 0.038	0.205 ± 0.108	0.002 ± 0.001	0.017 ± 0.012	0.092 ± 0.055
OS	TomoTwin [ 9 ]		0.077 ± 0.026	0.121 ± 0.029	0.095 ± 0.026	0.203 ± 0.137	0.348 ± 0.189	0.508 ± 0.214	0.049 ± 0.051	0.164 ± 0.136	0.265 ± 0.196
	ProPicker [ 11 ]		n.a.	n.a.	0.076 ± 0.013	n.a.	n.a.	0.221 ± 0.071	n.a.	n.a.	0.196 ± 0.108
	FullTilt (ours)		0.112 ± 0.033	0.137 ± 0.036	0.167 ± 0.037	0.204 ± 0.006	0.439 ± 0.005	0.593 ± 0.004	0.149 ± 0.051	0.271 ± 0.083	0.394 ± 0.134

Table 4: Zero-shot cross-instance detection results on real-world cryo-ET datasets. The best-performing open-set method for each metric is highlighted in bold. FullTilt outperforms all baselines, excepting [email protected] under 2- and 4-prompt settings on EMPIAR-10304.

	1 Prompt				2 Prompts				4 Prompts
# Classes	[email protected] $\uparrow$	mAP@1r $\uparrow$	T. $\downarrow$	M. $\downarrow$	[email protected] $\uparrow$	mAP@1r $\uparrow$	T. $\downarrow$	M. $\downarrow$	[email protected] $\uparrow$	mAP@1r $\uparrow$	T. $\downarrow$	M. $\downarrow$
1	0.118 ± 0.013	0.148 ± 0.013	0.499	3430	0.129 ± 0.010	0.161 ± 0.011	0.503	3430	0.133 ± 0.007	0.180 ± 0.007	0.510	3430
2	0.112 ± 0.005	0.142 ± 0.004	0.518	3430	0.123 ± 0.003	0.155 ± 0.003	0.522	3430	0.127 ± 0.003	0.161 ± 0.003	0.523	3430
3	0.111 ± 0.004	0.142 ± 0.004	0.521	3440	0.119 ± 0.002	0.151 ± 0.002	0.524	3440	0.123 ± 0.001	0.156 ± 0.001	0.525	3440
4	0.111 ± 0.002	0.142 ± 0.003	0.505	3440	0.118 ± 0.002	0.151 ± 0.001	0.507	3440	0.120 ± 0.001	0.153 ± 0.001	0.510	3440
5	0.108 ± 0.003	0.139 ± 0.004	0.514	3440	0.118 ± 0.002	0.150 ± 0.002	0.521	3440	0.119 ± 0.001	0.150 ± 0.002	0.524	3440

Table 5: Multi-class prompting performance comparison (intra-instance). FullTilt maintains stable computational efficiency regardless of the number of target classes.

CZII

EMPIAR-10304

EMPIAR-10499

Configuration

[email protected]

\uparrow

mAP@1r

\uparrow

\downarrow

\downarrow

[email protected]

\uparrow

mAP@1r

\uparrow

\downarrow

\downarrow

[email protected]

\uparrow

mAP@1r

\uparrow

\downarrow

\downarrow

Baseline (100%)

0.118 ± 0.013

0.148 ± 0.013

0.499

3430

0.199 ± 0.004

0.437 ± 0.002

0.229

2410

0.159 ± 0.004

0.288 ± 0.004

0.389

3410

Ablation: Reduced Tilt Images

50%

0.100 ± 0.013

0.133 ± 0.011

0.246

1860

0.184 ± 0.002

0.427 ± 0.002

0.118

1380

0.127 ± 0.006

0.260 ± 0.007

0.202

1950

25%

0.085 ± 0.010

0.120 ± 0.011

0.153

1280

0.157 ± 0.003

0.391 ± 0.003

0.079

860

0.060 ± 0.005

0.165 ± 0.008

0.116

1220

12.5%

0.038 ± 0.007

0.073 ± 0.014

0.087

700

0.084 ± 0.003

0.268 ± 0.003

0.069

600

0.010 ± 0.001

0.048 ± 0.003

0.083

790

Table 6: Ablation study on the number of tilt images and the number of projections per prompt. Reducing the number of tilt images decreases runtime and memory at the cost of overall detection performance; reducing the number of projections per prompt only slightly compromises performance.

CZII

EMPIAR-10304

EMPIAR-10499

\mathcal{E}_{\text{Tilt}}

\mathcal{I}_{\text{Tilt}}

\mathcal{G}

[email protected]

\uparrow

mAP@1r

\uparrow

\downarrow

\downarrow

[email protected]

\uparrow

mAP@1r

\uparrow

\downarrow

\downarrow

[email protected]

\uparrow

mAP@1r

\uparrow

\downarrow

\downarrow

0.001 ± 0.001

0.003 ± 0.001

0.215

3430

0.000 ± 0.000

0.001 ± 0.000

0.133

2280

0.000 ± 0.000

0.200

3170

✓

0.002 ± 0.001

0.008 ± 0.001

0.519

3430

0.000 ± 0.000

0.005 ± 0.001

0.229

2300

0.000 ± 0.000

0.002 ± 0.000

0.380

3190

✓

0.040 ± 0.002

0.099 ± 0.003

0.223

3430

0.004 ± 0.000

0.106 ± 0.003

0.141

2400

0.009 ± 0.001

0.082 ± 0.003

0.207

3380

✓

0.025 ± 0.002

0.063 ± 0.003

0.211

3430

0.000 ± 0.000

0.016 ± 0.001

0.139

2280

0.001 ± 0.000

0.011 ± 0.001

0.198

3170

✓

0.099 ± 0.007

0.131 ± 0.008

0.517

3430

0.008 ± 0.001

0.136 ± 0.009

0.228

2420

0.049 ± 0.003

0.168 ± 0.006

0.392

3390

Table 7: Ablation study on FullTilt components. We ablate the tilt-series encoder (

\mathcal{E}_{\text{Tilt}}

), tilt-aware query initializer (

\mathcal{I}_{\text{Tilt}}

), and auxiliary geometric primitives (

\mathcal{G}

) on detection performance across three real-world datasets. The integration of all three modules yields the optimal performance.

Multi-View 3D Object Detection

Multi-View 3D Object Detection combines images from multiple cameras to locate and classify objects in three-dimensional space.

See all 25 papers

SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection
May 13, 2026Sandro Papais, Lezhou Feng, Charles Cossette +1Multi-View 3D Object Detection Token Pruning
University of Toronto · Zoox Inc.
The microscope is the mask: privileged views and labels from a cryo-ET forward model
Sep 3, 2026Bogdan Toader, Kiarash Jamali, Tanmay A. M. Bharat +1Protein Representation Learning Cryo-Electron Microscopy
MRC Laboratory of Molecular Biology Cambridge, UK
POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET
Jun 8, 2026Jonathan Schwartz, Utz Heinrich Ermel, C. Braxton Owens +6Image Segmentation Cryo-Electron Microscopy
1Biohub, Redwood City, CA 94063, USA · 2Brigham Young University, Provo, UT, 84602, USA

May 13, 2026J/K move · Enter open · S save

SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

Sandro Papais, Lezhou Feng, Charles Cossette, Lingting Ge

University of Toronto · Zoox Inc.

Multi-View 3D Object Detection Token Pruning Efficient ViTs 3D Object Detection Real-Time Object Detection

Open paper AskPDF arXiv

Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense token and query processing across multiple views and large 3D regions. Existing sparsity methods, designed mainly for 2D vision, prune or merge image tokens but do not extend to full-model sparsity or address 3D object queries. We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image tokens and 3D object queries while storing filtered features for reactivation. Mutual 2D-3D relevance heads allocate compute to driving-critical content and preserve other embeddings. Evaluated on nuScenes and our new nuScenes-Relevance benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, establishing real-time large-scale ViT-based 3D detection while maintaining accuracy on planning-critical agents.

Abstract

Multi-View 3D Object Detection

Explore similar work

SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

The microscope is the mask: privileged views and labels from a cryo-ET forward model

POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET