FTU-Seek: Foundation Model-Guided Hard-Negative Learning for Sparse Functional Tissue Unit Segmentation

Characterising the TME from routine H&E-stained histology images requires simultaneous cell segmentation, biological feature extraction, and interpretable clinical reporting. We present SegTME-UNI2, a unified framework addressing all three requirements end-to-end: a segmentation backbone that converts raw H&E patches into per-nucleus class labels, a structured feature-extraction pipeline that turns those labels into quantitative TME descriptors, and a language-model narrative generator that turns those descriptors into clinician-readable text. At its core is UNI2-UperHoVer, a dual-head multiscale segmentation model that pairs UNI2 with two parallel UperNet decoders: one for six-class semantic segmentation and one for HV gradient regression enabling watershed-based nuclear instance separation. It is trained via a three-stage progressive pseudo-label curriculum, scaling from PanNuke (Stage 1, 0.25um/pixel) to TCGA-UT Scale-0 (Stage 2, 0.5um/pixel) and full 1.6M-patch, six-scale TCGA-UT (Stage 3, 0.5 to 1.0um/pixel). TCGA-UT's coarser, broader per-patch context than PanNuke's also permits a larger tile stride during whole-slide inference. This pipeline computes 22 per-patch compositional, morphological, spatial-entropy, and intercellular-distance metrics and translates them into six categorical phenotype labels and a standardised biological-token vocabulary, fine-tuned via NVIDIA BioNeMo that converts into clinically grounded narratives whose individual claims can be spot-checked directly against the underlying features. Qualitative validation on IGNITE NSCLC tiles shows the pipeline produces biologically coherent phenotype classifications and narratives despite inter-institutional stain variability and imperfect segmentation. The pseudo-labelled TCGA-UT dataset and UNI2-UperHoVer checkpoints are publicly released to support large-scale TME profiling and spatial biology research.

Foreground segmentation is the critical first step of every computational pathology pipeline, yet existing methods rely on hand-tuned heuristics or supervised models that overfit to narrow stain and scanner distributions, failing silently on specialised stains such as Jones silver or Elastica van Gieson. We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task and leverages general-purpose vision-language models (VLMs) as zero-annotation oracles. Our key insight is that tissue-versus-background discrimination is a natural-image recognition problem, not a histopathological one, so VLMs trained on internet-scale corpora generalise where domain-specific models cannot. We introduce Leica-75, a benchmark of 75 renal transplant whole-slide images spanning three stain families. On Leica-75, our method achieves the highest segmentation quality on out-of-distribution stains (Dice 0.858 +/- 0.027 on Jones, 0.853 +/- 0.041 on EVG) with 7x lower cross-stain variance than the best supervised baseline, while remaining competitive on in-distribution H&E. Few-shot prompting with automatically curated exemplars (Auto-context) rescues hard cases on Stress-32 (n=32), a curated stress-test subset (Dice 0.470 to 0.819 for the 2B model). VLM-based annotation review matches human expert consensus (kappa=0.989 for blur detection; mean precision/recall grading accuracy 0.708 vs. human 0.646 for segmentation mask review). The resulting pseudo-labels are used to distil lightweight student models that are as performant as the teacher model while running for a fraction of the cost. Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.

Foundation models are reshaping computational histopathology, yet their value for whole-slide image retrieval relative to strong patch-based and supervised aggregation baselines remains unclear. We benchmarked ten pipelines on 9,387 diagnostic slides spanning 17 organs and 60 diagnoses from The Cancer Genome Atlas (TCGA) using patient-level leave-one-patient-out evaluation. Methods included four pre-trained slide foundation models, a supervised attention-based multiple instance learning (ABMIL) aggregator on patch embeddings, and patch-level retrieval across five sampling densities. Performance varied more across organs and diagnoses than across architectures. Although the slide foundation model TITAN achieved the strongest overall results, its advantage was modest; ABMIL and patch-based methods reached comparable Top-1 and Top-3 accuracy, with no model consistently dominant. Morphologically distinctive entities approached ceiling performance, while rare, heterogeneous, and closely related subtypes remained challenging. Misclassifications aligned with organs exhibiting known inter-observer variability, suggesting an intrinsic ceiling for morphology-only retrieval. Performance was driven primarily by patch-level feature representations, with limited benefit from slide-level aggregation, indicating aggregation may be unnecessary in many settings. These findings argue against a universally optimal architecture and instead support organ-resolved benchmarking, diagnosis-aware or ensemble strategies, stronger feature representations, and multimodal retrieval frameworks. Notably, even the best model achieved only $\approx 68\% \pm 21\%$ retrieval accuracy on TCGA, and some subtypes showed $0\%$ accuracy across all methods, highlighting fundamental limitations of morphology-based representations and the need for substantial progress before reliable clinical deployment.