Learning to Track Cells from Sparse Annotations in 3D Time-Lapse Microscopy
Organizations: GO Drive Inc. Tokyo, Japan
Abstract
Reconstructing cell lineages from 3D time-lapse microscopy requires distinguishing cell motion from division and inferring temporally consistent connections. When only a subset of cells is annotated, however, training procedures that treat unannotated regions as background or incorrect correspondences can be problematic. We propose a cell tracking pipeline that integrates detection from sparse annotations, calibration of candidate connections using image features, and joint selection of lineage graph events. For detection, we combine the outputs of 2.5D and 3D models trained to suppress erroneous background supervision at unannotated cells. We then calibrate ordinary link and division probabilities using motion features from dense flow, correspondence features from a learned cell-matching model, and features from models that identify cells immediately before division, dividing parents, and daughters immediately after division. Ordinary links, divisions represented as daughter pairs, and node inclusion are selected jointly under a surrogate objective derived from the evaluation metric, followed by gap filling and temporal coordinate refinement. The pipeline achieves an overall score of 0.9762 in 5-fold cross-validation on 199 videos from 2 zebrafish embryos. With candidates and calibrated values held fixed, joint selection outperforms a sequential method that allows replacement of existing divisions and reoptimization of ordinary links by 0.0105. The improvement arises primarily from division recovery.
Figures & tables
| Model | Input downsampling factors (z, y, x) | Training region (downsampled voxels) | Epochs | Learning rate | EMA |
|---|---|---|---|---|---|
| 2.5D detection | (1, 1, 1) | 48 192 192 | 800 | 1e-3 | 0.995 |
| 3D detection | (1, 1, 1) | 48 192 192 | 400 | 3e-4 | 0.999 |
| Matcher | (1, 4, 4) | 64 64 64 | 200 | 1e-3 | None |
| Dense flow | (1, 4, 4) | 64 64 64 | 48 | 5e-4 | None |
| Division parent | (1, 2, 2) | 64 128 128 | 20 | 1e-4 | 0.995 |
| Processing stage (cumulative changes) | Adjusted edge Jaccard | Division Jaccard | Overall score |
|---|---|---|---|
| Detection + distance-based assignment with collective-motion correction | 0.8906 | 0.0000 | 0.8906 |
| + Dense-flow motion compensation | 0.9026 | 0.0000 | 0.9026 |
| + Matcher candidate selection and correspondence scores | 0.9028 | 0.0000 | 0.9028 |
| + Division candidates, calibration, and joint optimization | 0.9112 | 0.5185 | 0.9630 |
| + Gap filling | 0.9167 | 0.5185 | 0.9685 |
| + Removal of weakly connected components with fewer than 6 nodes | 0.9184 | 0.5185 | 0.9703 |
| Features used for calibration | Score after optimization | Final adjusted edge Jaccard | Final division Jaccard | Final overall score |
|---|---|---|---|---|
| All features | 0.9630 | 0.9235 | 0.5266 | 0.9762 |
| Without flow features | 0.9612 | 0.9207 | 0.5213 | 0.9728 |
| Without matcher features | 0.9530 | 0.9179 | 0.4607 | 0.9640 |
| Without division-model features | 0.9425 | 0.9227 | 0.3285 | 0.9555 |
| Selection strategy | After optimization | Final output | Final division TP/FP/FN | Final-output difference from joint selection [95% CI] |
|---|---|---|---|---|
| Joint selection | 0.9630 | 0.9762 | 99/37/52 | — |
| Select ordinary links, then add divisions locally | 0.9535 | 0.9657 | 78/34/73 | 0.0105 [ 0.0156, 0.0052] |
| Also allow division replacement and ordinary-link reoptimization | 0.9535 | 0.9657 | 78/34/73 | 0.0105 [ 0.0156, 0.0052] |
| Condition | After optimization | Final output | Final-output difference from control [95% CI] |
|---|---|---|---|
| All features, joint selection, calibration ensemble (control) | 0.9630 | 0.9762 | — |
| Constant Exposure for ordinary links | 0.9603 | 0.9742 | 0.0020 [ 0.0048, +0.0004] |
| Constant Exposure for divisions | 0.9638 | 0.9761 | 0.0000 [ 0.0017, +0.0016] |
| Constant Exposure for both | 0.9622 | 0.9753 | 0.0009 [ 0.0035, +0.0016] |
| Linear calibration with 34 features | 0.9598 | 0.9719 | 0.0042 [ 0.0081, 0.0005] |
| Linear + LightGBM calibration | 0.9625 | 0.9756 | 0.0005 [ 0.0018, +0.0000] |
| Division information used for calibration | After optimization | Final output |
|---|---|---|
| Parent only | 0.9557 | 0.9675 |
| Parent + pre | 0.9571 | 0.9695 |
| Parent + post | 0.9632 | 0.9755 |
| Parent + pre + post | 0.9630 | 0.9762 |
| Division-parent inputs | After optimization | Final output | Final division TP/FP/FN |
|---|---|---|---|
| Five inputs using raw and flow-aligned images (retrained control) | 0.9630 | 0.9752 | 96/34/55 |
| Five inputs using duplicated raw images | 0.9608 | 0.9730 | 95/41/56 |
| Downstream configuration for the public detector | Local diagnostic score | Public | Private |
|---|---|---|---|
| Transfer the original image models and calibrators | 0.9650 | 0.9523 | 0.9432 |
| Retrain calibrators only | 0.9669 | 0.9463 | 0.9319 |
| Retrain matcher, division models, and calibrators | 0.9676 | 0.9473 | 0.9357 |
Microscopy Image Analysis
Microscopy image analysis uses computational methods to detect, segment, track, reconstruct, and quantify structures in scientific images.
See all 206 papers