CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

Ilan Theodoro; Jordão Bragantini; Loïc A. Royer

cs.CVOct 22, 2026

CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

Authors: Ilan Theodoro, Jordão Bragantini, Loïc A. Royer

Organizations: Biohub San Francisco, CA

Abstract

Large language models have shown outstanding performance on text-first tasks such as coding, math, and writing. Recent efforts are shifting models' training to include scientific problems that often live in highly specialized domains beyond text, such as protein structure and biological images. However, in these scenarios, their general-purpose accuracy compared with specialized algorithms remains in question. One such domain is cell tracking, which requires spatiotemporal awareness of cell dynamics. To evaluate this, we created a new benchmark, CellTrack-Bench, which assesses Vision-Language Models' (VLMs) cell-tracking-related performance against specialized trackers. The full benchmark draws from 15 standard cell-tracking datasets across four imaging modalities and comprises 1,537 problems corresponding to 9,041 cell-linking decisions. We selected a hard subset requiring greater reasoning effort by identifying failures shared across distinct algorithmic trackers. With this proposed dataset, we: (i) evaluated different environment harnesses for VLMs; (ii) benchmarked 9 frontier closed-source models and 13 open-weight models; and (iii) provided a detailed analysis of their failure modes. Code, the stored model outputs, and a pipeline that regenerates the benchmark from the source datasets are available at https://github.com/royerlab/celltrack-bench.

Figures & tables

Figure 1: Cognitive aspect of cell tracking. Cell tracking is dominated by a majority of trivial associations that can be retrieved using simple proxies such as a proximity-based one. In (a) , we show a human specialist cognitive process on one of those links. However, in some cases of cell-state changes, such as a division in (b) , it requires a more global comprehension and understanding of biological processes.

Figure 2: CellTrack-Bench construction. Three trackers with distinct inductive biases run across four imaging modalities; errors are ranked by a cross-tracker hard-mining score, all-three-wrong first (illustrated: the division $S2\rightarrow\{T2,T3\}$ that every tracker resolves as $S3\rightarrow T3$ ), then by hardness $h(e)$ , backfilled in rank order to a per-dataset cap, split sequence-disjointly, and instantiated as linking, repair, and inspection tasks (scales and counts: Section 3.2 ).

Panel of Figure 3 — Figure 3: Example frames from all 15 source datasets, spanning four imaging modalities: bright-field (BF), differential interference contrast (DIC), phase contrast (PhC), and fluorescence (Fluo).

Condition	Raw	BBox	Seg.	Zoomed crops	Raw	BBox	Seg.	Zoomed crops	Multiturn	Tool	Purpose	gpt-5.5	opus-4.8
	Frame $t$				Frame $t+1$							NN-res. F1
c1	✓	✓	–	–	✓	✓	–	–	✓	crop_and_zoom(...)	Coordinate baseline	$0.497{\scriptstyle\pm.033}$	$0.216{\scriptstyle\pm.028}$
c2	✓	–	✓	–	✓	–	✓	–	✓	crop_and_zoom(...)	Visual localization	$0.329{\scriptstyle\pm.027}$	—
c3	✓	–	✓	–	✓	–	✓	–	✓	cell_region(cell_id: int)	Cell-addressed inspection	$0.347{\scriptstyle\pm.030}$	—
c4	✓	–	✓	Outlined	✓	–	✓	Outlined	–	–	Fixed visual inspection	$0.351{\scriptstyle\pm.029}$	$0.150{\scriptstyle\pm.022}$
c5	✓	–	✓	Plain	✓	–	✓	Plain	–	–	Crop-outline ablation	$0.331{\scriptstyle\pm.028}$	—

Table 1: Validation-time evidence harnesses. Seg. denotes segmentation outlines with displayed cell identifiers. Zoomed crops are either provided directly or obtained through the indicated tool. The two rightmost columns give NN-residual F1 (mean

\pm

standard error over the 178 items with residual links) on the hard validation split (single completion, frozen budget); claude-opus-4.8 was run post hoc on c1/c4/c8 only.

System	Access	Primary task scores				Shortcut resistance				Graph utility
		Link F1	Repair F1	Inspect F1	Mean	NN-res	Div	LNK sub	$\Delta$ NN	Repair $\Delta$ LNK	Inspect $\Delta$ LNK	$\Delta$ /1k
GPT-5.6 (sol) [ 36 ]	API	0.811	0.815	0.631	0.752	0.723	0.787	0.767	$+0.274$	$+0.412$	$+0.518$	0.555
GPT-5.6 (terra) [ 36 ]	API	0.792	0.788	0.590	0.724	0.634	0.707	0.692	$+0.199$	$+0.337$	$+0.518$	0.529
Claude Fable 5 [ 2 ]	API	0.790	0.786	0.698	0.758	0.521	0.573	0.664	$+0.171$	$+0.348$	$+0.513$	0.517
GPT-5.5 [ 35 ]	API	0.717	0.741	0.588	0.682	0.577	0.622	0.638	$+0.145$	$+0.280$	$+0.500$	0.511
Claude Opus 5 [ 4 ]	API	0.758	0.739	0.532	0.676	0.406	0.411	0.623	$+0.130$	$+0.263$	$+0.459$	0.542

Table 2: Unified frozen-c1 hard-test results. Primary task scores are shaded blue, proximity-resistance metrics green, and graph-level repair and inspection utility peach. LNK sub is the graph-edit score,

\Delta

NN is its margin over the nearest-centroid graph, and

\Delta

/1k is inspection gain per 1,000 flagged edges. Reference graphs and trackers are evaluated only where their outputs define the metric.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Scale	Split	Items	Src.	Tgt.	Edges	Div.	Cont.
hard	validation	205	1,025	2,516	1,188	358	830
hard	test	244	1,220	3,200	1,426	424	1,002
full	validation	626	3,130	8,286	3,779	1,350	2,429
full	test	911	4,555	12,003	5,262	1,443	3,819

Appendix

Table 3: Materialized benchmark size. Every neighborhood contains exactly five source cells; targets are the next-frame candidates for those sources. A source with two daughters contributes two division edges.

Probe	Reasoning unit	Information used	Role in construction
Distance Hungarian	Detection centroid	Pairwise displacement	Geometric one-to-one baseline; no divisions
Trackastra	Segmented cell	Learned appearance, motion, and node context	Node-level learned correspondence
HOCT	Cell and candidate link	Node context and link-to-link interactions	High-order correspondence and division reasoning

Appendix

Table 4: The three specialized trackers expose complementary association failure modes. Their predictions are used to identify difficult links, not as estimates of performance on the unfiltered source datasets.

Figure 4: Two evidence-interface families. In the evidence-supplied single-turn interface (top), annotated frames and per-cell crops are included in the prompt before one response. In the evidence-requested multi-turn interface (bottom), the model receives an unannotated overview and normalized bounding boxes, then chooses regions to inspect through crop_and_zoom before returning its answer.

Panel of Figure 5 — Figure 5: All eight evidence-harness conditions (definitions in Table 1 ), instantiated on one fixed division-bearing test item ( BF-C2DL-HSC , sequence 02 , frames 714–715: five source cells and fifteen candidate targets; parent S2 divides into daughters T6 and T14). Every panel is the exact image the harness serves for the cell-linking task. The task text, candidate lists, and answer schema remain fixed, so the conditions differ only in the evidence shown.

Condition	Linking	Repair	Inspection	Mean
c1	0.662	0.690	0.474	0.609
c2	0.423	0.535	0.261	0.406
c3	0.524	0.639	0.367	0.510
c4	0.444	0.546	0.348	0.446
c5	0.447	0.544	0.296	0.429
c6	0.375	0.479	0.283	0.379

Appendix

Table 5: Validation-only harness selection with gpt-5.5 at low reasoning effort. Entries are dataset-macro primary scores; Mean is their unweighted average across the three tasks. Condition c7 is a diagnostic information-loss ablation.

Operation	Charged when	Cost
Addition	A ground-truth endpoint pair is missing	1.5
Deletion	A predicted endpoint pair is spurious	1.0
Semantic change	Endpoints match but the edge label is wrong	1.0

Appendix

Table 6: Edit operations used by Subgraph-LNK.

Split	NN LNK	HOCT LNK	Terra LNK	$\Delta$ NN	Mean	Repair F1	Repair $\Delta$ LNK	Inspection @10	Gold @10
Validation	0.409	0.527	0.616	$+0.207$	0.730	0.791	$+0.154$	$+0.210$	$+0.268$
Test	0.493	0.385	0.692	$+0.199$	0.724	0.788	$+0.337$	$+0.244$	$+0.284$

Appendix

Table 7: Validation–test robustness for GPT-5.6 (terra). Validation contains 205 neighborhoods and test contains 244. NN, HOCT, and Terra report LNK sub ; Mean is the unweighted mean of the three task-primary scores. Repair

\Delta

LNK is measured relative to the split-specific HOCT proposal. Inspection @10 and Gold @10 use the tightest review budget. Validation results are descriptive because that split was used to select the harness and contains one system and completion.

System	Repair F1	$\Delta$ LNK	Add P/R	Delete P/R	Better	Worse
GPT-5.6 (sol)	0.815	$+0.412$	0.85/0.77	0.90/0.77	80%	7%
Claude Fable 5	0.786	$+0.348$	0.77/0.72	0.82/0.77	72%	13%
GPT-5.6 (terra)	0.788	$+0.337$	0.76/0.71	0.91/0.67	72%	12%
GPT-5.5	0.741	$+0.280$	0.69/0.68	0.78/0.73	61%	15%
Claude Opus 5	0.739	$+0.263$	0.76/0.58	0.78/0.71	57%	12%
Claude Opus 4.8	0.729	$+0.201$	0.62/0.58	0.77/0.68	55%	17%

Appendix

Table 8: Track-repair decomposition, sorted by graph-level benefit.

\Delta

LNK is measured relative to the HOCT proposal; the oracle ceiling is

+0.615

. Add and Delete report micro-averaged precision/recall against the gold edit set. Better and Worse denote the percentage of neighborhoods whose per-item LNK increases or decreases; the remainder are unchanged.

System	Inspection F1	Flags	@10%	@25%	@50%	All flags
Gold-flag oracle	–	1061	$+0.284$	$+0.441$	$+0.542$	$+0.615$
GPT-5.6 (sol)	0.631	932	$+0.249$	$+0.265$	$+0.387$	$+0.518$
Claude Fable 5	0.698	993	$+0.249$	$+0.287$	$+0.396$	$+0.513$
GPT-5.6 (terra)	0.590	978	$+0.244$	$+0.288$	$+0.413$	$+0.518$
GPT-5.5	0.588	979	$+0.235$	$+0.264$	$+0.378$	$+0.500$
Claude Opus 5	0.532	847	$+0.223$	$+0.250$	$+0.356$	$+0.459$

Appendix

Table 9: Inspection under fixed review budgets. Each entry reports the

\Delta

LNK recovered after oracle correction of the model’s selected sources. Nominal budgets of 10/25/50% realize 19/38/58% of proposal edges because each neighborhood contains only 4–6 edges. Stored answers do not contain confidence scores, so flags are retained in emission order. Slash-separated values represent the two systems named in the row; ranges are used where only the paired summary was retained.

Stratum	$n$	NN baseline	GPT-5.6 (sol)	Gemma 4
NN-solved	22	1.000	0.992	0.976
NN-resistant	222	0.494	0.766	0.536
Hardness Q1 (easiest)	48	0.706	0.944	0.629
Hardness Q2	49	0.707	0.901	0.718
Hardness Q3	49	0.432	0.689	0.411
Hardness Q4	49	0.510	0.662	0.533

Appendix

Table 10: Per-neighborhood linking LNK sub by difficulty stratum, macro-averaged over datasets represented within each stratum. NN-solved denotes neighborhoods exactly reconstructed by nearest-centroid. EA, EC, and ED denote missing, semantic, and spurious anchor errors.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 999 papers

Can Vision-Language Models Solve the Shell Game?
Mar 9, 2026Tiedong Liu, Wee Sun LeeVisual Object Tracking Temporal Video Understanding
National University of Singapore
DataComp-VLM: Improved Open Datasets for Vision-Language Models
Jun 26, 2026Matteo Farina, Vishaal Udandarao, Thao Nguyen +33VLM Evaluation Training Data Curation
University of Trento · 2Tübingen AI Center, University of Tübingen · University of Cambridge +12
Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
Feb 17, 2026Yuval LeventalVision-Language Models VLM Reasoning
Rochester Institute of Technology

Mar 9, 2026J/K move · Enter open · S save

Can Vision-Language Models Solve the Shell Game?

Tiedong Liu, Wee Sun Lee

National University of Singapore

Visual Object Tracking Temporal Video Understanding VLM Evaluation State Tracking

Open paper AskPDF arXiv

Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .

Abstract

VLM Evaluation

Explore similar work

Can Vision-Language Models Solve the Shell Game?

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families