Deep Learning for BioImaging: What Are We Really Learning?
Organizations: Université Paris Cité, IRD, Inserm, MERIT, F-75006, Paris, France · IBENS, École Normale Supérieure, Université PSL, Paris, France · Institut Curie, Université PSL, Paris, France · Iktos, Paris, France ·
Abstract
Representation learning has driven major advances in natural image analysis by enabling models to acquire high-level semantic features. In microscopy imaging, however, it remains unclear what current representation learning methods really learn. In this work, we conduct a systematic study of representation learning for the two most widely used and broadly available microscopy data types, representing critical scales in biology: cell culture and tissue imaging. We investigate whether, in contrast to natural images, existing models fail to consistently acquire high-level, biologically meaningful features. To this end, we introduce a set of simple yet revealing baselines on curated benchmarks, including untrained models and structural representations of cellular tissue. Our results show that, surprisingly, for a considerable subset of evaluation settings, the baselines are comparable to state-of-the-art methods, demonstrating that many commonly used benchmark metrics are insufficient to assess representation quality and often mask a lack of relevant high-level abstractions. In addition, we investigate how detailed comparisons with these baselines provide ways to interpret the strengths and weaknesses of models for further improvements. Together, our results suggest that progress in representation learning for microscopy requires not only stronger models, but also benchmarks that are more indicative of what is actually learned.
Figures & tables
| Modality | Training | CCRCC | PRAD | SKCM | COAD |
| Full Image | No | 0.12 0.14 | 0.25 0.06 | 0.34 0.06 | 0.27 0.08 |
| Full Image | Yes, ood | 0.26 0.07 | 0.41 0.01 | 0.62 0.07 | 0.30 0.07 |
| Full Image | Yes, iid | 0.37 0.06 | 0.49 0.02 | 0.68 0.03 | 0.38 0.06 |
| Structure | Yes, ood | 0.16 0.04 | 0.38 0.04 | 0.45 0.07 | 0.31 0.01 |
| Model | Training | CCRCC | PRAD | SKCM | COAD |
| Cell count | No | 0.11 0.07 | 0.37 0.05 | 0.40 0.06 | 0.30 0.07 |
| GCN | Yes, synth. data | 0.04 (-0.02) | 0.38 (+0.04) | 0.45 (+0.08) | 0.31 (+0.04) |
| EGNN | Yes, synth. data | 0.04 (-0.01) | 0.36 (+0.005) | 0.22 (-0.05) | 0.28 (-0.02) |
| ResNet152 | Yes, IN1k | 0.11 (+0.04) | 0.34 (+0.09) | 0.37 (+0.23) | 0.24 (+0.14) |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Number of slides | Number of patients (splits) | Condition |
| IDC | 4 | 4 | Invasive ductal carcinoma (breast) |
| PRAD | 23 | 2 | Prostate adenocarcinoma |
| PAAD | 2 | 2 | Pancreatic adenocarcinoma |
| SCKM | 2 | 2 | skin cutaneous melanoma |
| COAD | 4 | 2 | colon adenocarcinoma |
| READ | 4 | 2 | rectal adenocarcinoma |
| Architecture | timm Model Name | timm Weights | Model Stages | Aggregation |
| Convolutional Neural Networks (CNNs) | ||||
| ResNet18 | resnet18 | tv_in1k | layer {1, 2, 3, 4} | Avg-pool |
| ResNet34 | resnet34 | tv_in1k | layer {1, 2, 3, 4} | Avg-pool |
| ResNet50 | resnet50 | tv2_in1k | layer {1, 2, 3, 4} | Avg-pool |
| ResNet152 | resnet152 | tv2_in1k | layer {1, 2, 3, 4} | Avg-pool |
| Vision Transformers (ViTs) | ||||
| Class | Sampling | Noise and/ or density | Cell count |
| 0 | Noisy Grid, 10 voxels | 900 | |
| 1 | Noisy Grid, 10 voxels | 900 | |
| 2 | Noisy Grid, 10 voxels | , | 890 |
| 3 | Noisy Grid, 4 voxels | , | 848 |
| 4 | Noisy Grid, 10 voxels | , | 964 |
| 5 | Noisy Grid, 10 voxels | , | 1028 |
| Model | Average F1 Score | Worst Class (F1-score) | Best Class (F1-score) | Invariance |
| ResNet18 (from scratch) | 0.87 | class 18 (0.38) | classes 1,3,5,7,8,9,14,15,17,19,21 ( 0.99 ) | Learned, data augmentations |
| ViT-l/16 (linear probing) | 0.68 | class 22 (0.15) | classes 7,9,12,15,17,20,21 ( 0.98) | Learned, data augmentations |
| GCN (from scratch) | 0.72 | class 4 (0.15) | classes 7,8,9,12,15,17,21 ( 0.98) | Learned, data augmentations |
| E2GNN (from scratch) | 0.73 | class 4 (0.04) | classes 1,5,12,17,18,19 (=1.0) | Architectural, E( ) |
| Benchmark | IDC | PRAD | PAAD | SKCM | COAD | READ | CCRCC | LUNG | LYMPH IDC | Avg | ||
| Modality | Training | Model | ||||||||||
| 1x1 | Hand-crafted | Cell count (PCA+ridge) | 0.32 0.04 | 0.26 0.02 | 0.32 0.04 | 0.26 0.05 | 0.26 0.004 | 0.02 0.001 | 0.04 0.07 | 0.42 0.003 | 0.11 0.05 | 0.22 0.13 |
| Cell count (XGBoost) | 0.31 0.05 | 0.26 0.02 | 0.34 0.08 | 0.30 0.07 | 0.26 0.01 | 0.05 0.03 | 0.03 0.06 | 0.44 0.01 | 0.11 0.07 | 0.23 0.12 | ||
| 1NN | Hand-crafted | Cell count (PCA+ridge) | 0.38 0.06 | 0.37 0.05 | 0.30 0.04 | 0.32 0.02 | 0.30 0.07 | -0.002 0.03 | 0.11 0.07 | 0.49 0.02 | 0.21 0.11 | 0.28 0.14 |
| Cell count (XGBoost) | 0.37 0.08 | 0.36 0.05 | 0.34 0.07 | 0.40 0.06 | 0.30 0.07 | 0.03 0.02 | 0.04 0.09 | 0.49 0.02 | 0.21 0.16 | 0.28 0.15 | ||
| Synthetic data | EGNN | 0.22 0.07 | 0.36 0.09 | 0.33 0.09 | 0.22 0.06 | 0.28 0.12 | 0.02 0.06 | 0.05 0.068 | 0.19 0.09 | 0.17 0.09 | 0.20 0.12 |
| Benchmark | IDC | PRAD | PAAD | SKCM | COAD | READ | CCRCC | LUNG | LYMPH IDC | Avg | ||
| Modality | Training | Model | ||||||||||
| 1x1-img | Foundation Model | CONCH v1 | 0.536 0.084 | 0.355 0.010 | 0.446 0.071 | 0.579 0.050 | 0.253 0.008 | 0.163 0.049 | 0.218 0.035 | 0.531 0.011 | 0.251 0.042 | 0.370 0.157 |
| CONCH v1.5 | 0.544 0.085 | 0.381 0.010 | 0.457 0.060 | 0.554 0.036 | 0.279 0.013 | 0.159 0.066 | 0.218 0.039 | 0.550 0.007 | 0.270 0.054 | 0.379 0.154 | ||
| CTransPath | 0.511 0.053 | 0.343 0.046 | 0.436 0.067 | 0.512 0.081 | 0.228 0.057 | 0.113 0.077 | 0.228 0.047 | 0.503 0.040 | 0.235 0.048 | 0.345 0.151 | ||
| GigaPath | 0.553 0.073 | 0.370 0.021 | 0.474 0.049 | 0.556 0.067 | 0.291 0.025 | 0.193 0.067 | 0.241 0.038 | 0.544 0.035 | 0.252 0.052 | 0.386 0.148 | ||
| H-Optimus-0 | 0.598 0.084 | 0.379 0.002 | 0.492 0.041 | 0.655 0.058 | 0.302 0.002 | 0.222 0.048 | 0.274 0.035 | 0.561 0.028 | 0.260 0.043 | 0.416 0.163 |
| Benchmark | IDC | PRAD | PAAD | SKCM | COAD | READ | CCRCC | LUNG | LYMPH IDC | Avg | ||
| Modality | Training | Model | ||||||||||
| 1NN-graph | Pretrained | ResNet152 | 0.340 0.036 | 0.338 0.017 | 0.312 0.072 | 0.366 0.091 | 0.243 0.040 | 0.090 0.058 | 0.111 0.030 | 0.382 0.000 | 0.162 0.042 | 0.261 0.113 |
| ResNet18 | 0.338 0.041 | 0.324 0.014 | 0.295 0.060 | 0.333 0.066 | 0.231 0.040 | 0.093 0.062 | 0.109 0.035 | 0.380 0.002 | 0.153 0.055 | 0.251 0.108 | ||
| ResNet34 | 0.347 0.044 | 0.327 0.017 | 0.307 0.072 | 0.315 0.059 | 0.219 0.031 | 0.068 0.046 | 0.115 0.033 | 0.371 0.023 | 0.154 0.047 | 0.247 0.111 | ||
| ResNet50 | 0.341 0.032 | 0.331 0.015 | 0.319 0.067 | 0.361 0.117 | 0.212 0.013 | 0.097 0.051 | 0.116 0.032 | 0.359 0.030 | 0.157 0.044 | 0.255 0.109 | ||
| ViT-b/16 | 0.326 0.031 | 0.339 0.017 | 0.278 0.043 | 0.302 0.063 | 0.198 0.002 | 0.110 0.031 | 0.107 0.026 | 0.363 0.007 | 0.149 0.048 | 0.241 0.101 |
| Hyperparameter | Value | Mean Recall@5% |
| aggregation | avg | 0.19 0.02 |
| cls | 0.20 0.02 | |
| size | s | 0.20 0.02 |
| b | 0.20 0.02 | |
| l | 0.19 0.02 | |
| h | 0.19 0.02 |
| Model | Architecture | Fold | CORUM | HuMAP | Reactome | SIGNOR | StringDB |
| SingleConv | CNN | 1 | 0.284 0.009 | 0.363 0.007 | 0.080 0.001 | 0.037 0.008 | 0.222 0.003 |
| SingleConv | CNN | 2 | 0.281 0.021 | 0.350 0.030 | 0.064 0.001 | 0.066 0.005 | 0.226 0.011 |
| SingleConv | CNN | 3 | 0.263 0.004 | 0.309 0.014 | 0.068 0.001 | 0.046 0.005 | 0.204 0.005 |
| AlexNet | CNN | 1 | 0.111 0.039 | 0.106 0.029 | 0.059 0.005 | 0.057 0.003 | 0.087 0.016 |
| AlexNet | CNN | 2 | 0.114 0.012 | 0.120 0.006 | 0.059 0.017 | 0.057 0.013 | 0.102 0.015 |
| AlexNet | CNN | 3 | 0.103 0.023 | 0.098 0.023 | 0.049 0.010 | 0.058 0.018 | 0.089 0.021 |
| Model | Architecture | Fold | Mean mAP | |
| SingleConv | CNN | 0 | 0.357 | 0.018 |
| SingleConv | CNN | 1 | 0.315 | 0.008 |
| SingleConv | CNN | 2 | 0.366 | 0.017 |
| SingleConv | CNN | 3 | 0.340 | 0.016 |
| SingleConv | CNN | 4 | 0.341 | 0.023 |
| ResNet18 | CNN | 0 | 0.146 | 0.002 |
| Feature type | Model | Average Recall@5% |
| Pixels statistics | Pixel Stats | |
| Human handcrafted features | CellProfiler | |
| Untrained network | ViT-l/16 | |
| Pretrained Fondation Model | OpenPhenom |
| Model | Mean % of Replicating | Mean % of Matching |
| Random | 5.0 | 5.0 |
| Pixel Stats | 22.3 | 10.5 |
| OpenPhenom | 37.8 | 11.4 |
| ResNet152 (untrained) | 27.3 | 11.6 |
| ViT-l/16 (pretrained) | 36.8 | 12.3 |
| SingleConv (untrained) | 30.5 | 12.8 |
| Dataset | BACH | BRACS | ||
| Modality | Training | Model | ||
| Morphological cell graphs | Pretrained IID | ACM-bio | 38.37 2.16 | 21.55 2.62 |
| ACM-UNI | 29.03 9.92 | 19.95 3.27 | ||
| GrapHist | 69.16 3.37 | 60.30 0.46 | ||
| Full RGB Images | Pretrained, IID | DINOv2 | 59.25 3.34 | 52.68 1.78 |
| MAE | 57.94 3.91 | 56.33 0.76 |
| Dataset | BACH | BRACS (448 448) | BRACS (896 896) | ||
| Modality | Model | Training | |||
| Full RGB Images | H-Optimus-1 | Pretrained, IID | 90.88 2.13 | 48.20 1.71 | 75.65 0.99 |
| Morphological Features | mean_std | - | 62.06 2.42 | 56.98 1.02 | 58.02 1.17 |
| Structure Only (Images of Binary Graphs) | ResNet152 | Pretrained, OOD | 54.83 2.86 | 47.29 1.98 | 53.32 1.37 |
| ResNet18 | Pretrained, OOD | 48.21 4.22 | 43.31 1.56 | 46.90 1.71 | |
| ResNet34 | Pretrained, OOD | 53.43 2.22 | 44.95 0.70 | 45.72 1.83 |
Representation Learning
Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.
See all 804 papers