Diagnosis by Composition: Detecting and Localizing Unseen Diseases by Composing Learned Radiological Findings

Kaihua Hou; Wei Liu; Yi Huang; Hao Zhang; Haibo Jin; Yifei Sun; Yuan gao; Zuoou Li

cs.CVNov 3, 2026

Diagnosis by Composition: Detecting and Localizing Unseen Diseases by Composing Learned Radiological Findings

Authors: Kaihua Hou, Wei Liu, Yi Huang, Hao Zhang, Haibo Jin, Yifei Sun, Yuan gao, Zuoou Li,

Organizations: University of California, Berkeley · Alibaba Group · Fudan University · Zhejiang University ·

Abstract

Deep models for radiological diagnosis are typically trained to predict disease labels directly from images, which limits them to diseases seen during training and makes their decisions hard to interpret in radiologists' terms. Radiology, however, proceeds from visual findings to disease through published guidelines that make this decision rule explicit, interpretable, and reproducible. We introduce diagnosis by composition (DbC), a representation learning framework that encodes this findings-first logic. DbC learns a vocabulary of radiological findings by aligning image features with the text embeddings of the findings, each finding represented as a dense map over the scan. A disease is defined by a set of required findings derived from its guidelines, and its diagnosis map is a fixed, parameter-free composition of the corresponding finding maps. Because the finding-to-disease mapping is derived from guidelines rather than learned, DbC is interpretable and auditable by design, can extend to new diseases by specifying their findings without retraining, and can diagnose unseen diseases by composing their findings. On multi-phase liver CT, DbC detects and localizes unseen diseases, rare diseases from a different institution, and an external cohort significantly better than all comparators, including models that align images with disease names. DbC also improves detection and localization in the fully supervised setting and remains superior across training-set sizes. Although demonstrated on liver CT, DbC applies wherever radiological guidelines define diseases through findings.

Figures & tables

Figure 1: Disease-first versus findings-first learning. (a) Disease anchors leave an unseen disease $D$ without a trained anchor, while finding anchors express it as a composition. (b) With hemangioma (Heman) seen and ICC unseen in training, only DbC localizes the ICC, by composing four of its finding maps.

Figure 2: The DbC model. Visual embeddings $v_{n}$ are matched to the text embeddings $t_{m}$ of the finding names (Eq. 2 ), and each diagnosis map is the soft min of its required finding maps (Eq. 1 ).

	Localization				Detection
Model	Dice	IoU	Coverage	AP 50	AUROC	AP	Sens.@ 95% Spec.
SegVol	$.055$ $[.039,.076]$	$.037$ $[.026,.053]$	$.088$ $[.062,.124]$	$.000$ $[.000,.000]$	$.806$ $[.759,.848]$	$.284$ $[.213,.377]$	$.354$ $[.304,.408]$
SAT	$.190$ $[.156,.228]$	$.152$ $[.124,.184]$	$.185$ $[.150,.225]$	$.000$ $[.000,.001]$	$.900$ $[.888,.912]$	$.528$ $[.484,.582]$	$.592$ $[.553,.630]$
VoxTell	$.096$ $[.075,.124]$	$.069$ $[.054,.091]$	$.200$ $[.174,.230]$	$.000$ $[.000,.000]$	$.810$ $[.794,.825]$	$.230$ $[.215,.258]$	$.260$ $[.205,.303]$
Disease-V †	$.288$ $[.255,.326]$	$.233$ $[.205,.266]$	$.287$ $[.254,.326]$	$.153$ $[.117,.212]$	$.915$ $[.903,.928]$	$.628$ $[.578,.678]$	$.678$ $[.633,.714]$
Disease-VL †	$.271$ $[.235,.311]$	$.220$ $[.189,.255]$	$.274$ $[.236,.318]$	$.137$ $[.116,.190]$	$.916$ $[.904,.927]$	$.582$ $[.538,.642]$	$.646$ $[.599,.700]$

Table 1: Fully supervised diagnosis on the internal test set, with 95% confidence intervals. Bold marks the best value, ∗ a significant advantage over every comparator, and † a matched ablation.

		Localization			Detection
Unseen disease	Model	Dice	IoU	AP 50	AUROC	AP	Sens.@ 95% Spec.
HCC	SegVol	$.035$ $[.027,.043]$	$.020$ $[.016,.025]$	$.000$ $[.000,.000]$	$.757$ $[.735,.779]$	$.560$ $[.522,.600]$	$.197$ $[.139,.245]$
	SAT	$.215$ $[.185,.247]$	$.161$ $[.138,.188]$	$.000$ $[.000,.002]$	$.809$ $[.790,.827]$	$.607$ $[.566,.648]$	$\mathbf{.212}$ $\mathbf{[.132,.309]}$
	VoxTell	$.206$ $[.175,.240]$	$.157$ $[.133,.182]$	$.000$ $[.000,.000]$	$.811$ $[.792,.829]$	$.528$ $[.496,.561]$	$.018$ $[.007,.030]$
	Disease-VL †	$.249$ $[.217,.283]$	$.195$ $[.168,.225]$	$.089$ $[.066,.124]$	$.691$ $[.655,.723]$	$\mathbf{.624}$ $\mathbf{[.575,.674]}$	$.092$ $[.034,.175]$
	DbC	$\mathbf{.376}^{*}$ $\mathbf{[.328,.423]}$	$\mathbf{.311}^{*}$ $\mathbf{[.271,.353]}$	$\mathbf{.158}^{*}$ $\mathbf{[.128,.195]}$	$\mathbf{.834}^{*}$ $\mathbf{[.817,.851]}$	$.577$ $[.542,.614]$	$.074$ $[.040,.142]$

Table 2: Zero-shot diagnosis of unseen diseases under leave-one-disease-out. Notation as in Table 1 .

		Localization		Detection
Rare disease	Model	Dice	IoU	AUROC	AP
Abscess	SegVol	$.051$ $[.042,.060]$	$.027$ $[.022,.033]$	$.815$ $[.767,.863]$	$.751$ $[.676,.826]$
	SAT	$.174$ $[.119,.231]$	$.097$ $[.063,.140]$	$.789$ $[.740,.836]$	$.782$ $[.721,.837]$
	VoxTell	$.094$ $[.069,.123]$	$.066$ $[.048,.086]$	$.781$ $[.731,.829]$	$.799$ $[.742,.848]$
	Disease-VL †	$.170$ $[.135,.211]$	$.113$ $[.087,.145]$	$.830$ $[.794,.861]$	$.840$ $[.800,.893]$
	DbC	$\mathbf{.238}^{*}$ $\mathbf{[.203,.284]}$	$\mathbf{.163}^{*}$ $\mathbf{[.136,.196]}$	$\mathbf{.890}^{*}$ $\mathbf{[.851,.924]}$	$\mathbf{.865}^{*}$ $\mathbf{[.807,.916]}$

Table 3: Rare diseases from a different institution, scored against CTs without a liver lesion.

		Localization		Detection
Setup	Model	Dice	IoU	AUROC	AP
Zero-shot	SegVol	$.044$ $[.039,.051]$	$.026$ $[.023,.030]$	$.489$ $[.458,.520]$	$.245$ $[.232,.264]$
	SAT	$.094$ $[.083,.109]$	$.065$ $[.056,.076]$	$.528$ $[.497,.559]$	$.281$ $[.259,.313]$
	VoxTell	$.064$ $[.055,.075]$	$.042$ $[.036,.050]$	$.466$ $[.441,.490]$	$.238$ $[.227,.254]$
	Disease-VL †	$.103$ $[.091,.119]$	$.073$ $[.065,.085]$	$.593$ $[.560,.625]$	$.278$ $[.263,.302]$
	DbC	$\mathbf{.173}^{*}$ $\mathbf{[.156,.195]}$	$\mathbf{.134}^{*}$ $\mathbf{[.120,.152]}$	$\mathbf{.639}^{*}$ $\mathbf{[.606,.673]}$	$\mathbf{.326}$ $\mathbf{[.306,.358]}$

Table 4: External transfer to MCT-LTDiag from the zero-shot and full-training checkpoints.

Figure 3: Data scaling, DbC against Disease-VL. A star marks a significant advantage of DbC.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Finding	Phase	Required in
Density	low density	any	ICC, Meta, Heman
	low density on NC	NC	HCC, FNH
	water density	any	Cyst
	very high density	any	Calc
	iso-density	any	–
	heterogeneous density	any	–

Appendix

Table 5: The finding catalog. The Phase column gives the contrast phase on which a finding is supervised and read (NC non-contrast, A arterial, PV portal venous, D delayed), where “any” means phase-invariant, “and” means that every listed phase must be present and “or” that any one of them suffices. The Required in column lists the diagnostic compositions that contain the finding (Meta metastasis, Heman hemangioma, Calc calcification), and a dash marks a finding supervised from reports and the size rule only.

Disease	Diagnostic composition (required findings)
HCC	low density on NC $\wedge$ non-rim arterial hyperenhancement $\wedge$ non-peripheral washout † $\wedge$ enhancing capsule †
ICC	low density $\wedge$ rim arterial enhancement $\wedge$ progressive delayed enhancement $\wedge$ upstream biliary dilatation † $\wedge$ infiltrative growth pattern †
Metastasis	low density $\wedge$ rim arterial enhancement $\wedge$ peripheral washout † $\wedge$ ill-defined margin †
Hemangioma	low density $\wedge$ peripheral nodular discontinuous enhancement † $\wedge$ centripetal fill-in † $\wedge$ progressive delayed enhancement $\wedge$ well-defined margin
FNH	low density on NC $\wedge$ non-rim arterial hyperenhancement $\wedge$ isoenhancing portal phase † $\wedge$ well-defined margin $\wedge$ homogeneous enhancement †
Cyst	water density † $\wedge$ no arterial enhancement † $\wedge$ well-defined margin $\wedge$ homogeneous density † $\wedge$ round or oval shape †

Appendix

Table 6: Diagnostic compositions, each the soft minimum

\wedge

of Eq. ( 1 ) over the required findings of a disease. The two rare diseases were declared at evaluation time from published imaging criteria and were never trained on. A dagger marks a required finding that belongs to exactly one of the seven compositions.

Figure 4: The finding vocabulary and the seven diagnostic compositions drawn as a flow, with the 40 findings on the left, grouped by the twelve catalog categories in the order of Table 5 , and the seven diseases on the right (Meta metastasis, Heman hemangioma, Calc calcification), ordered to reduce ribbon crossings. A ribbon joins a finding to every disease whose composition requires it. A finding required by several diseases (dark bar) fans out into several ribbons, a finding required by one disease (bar in the color of that disease) has a single ribbon, and a finding required by none (light bar, grey label) is supervised from reports and the size rule only.

Figure 5: Extraction of supportive findings from the report. For every CT and each disease that both its mask and its report carry, Qwen3-Max receives the finding catalog and the diagnostic compositions in its system prompt and the report sections, the disease and the lesion long axes in its user prompt, and answers in JSON with the supported findings, their source phrases and a confidence (example values shown). The supportive findings supervise the finding maps during training and are never used at inference.

Hyperparameter	Value
Hardware	2 NVIDIA H200 GPUs, distributed data parallel
Micro-batch and accumulation	2 CTs per GPU, 8 micro-batches
Effective batch	32 CTs per optimizer step
Optimizer	AdamW [ Loshchilov and Hutter, 2019 ] , peak learning rate $5.7\times 10^{-5}$ , weight decay 0.1
Schedule	linear warm-up over the first 3% of optimizer steps, then constant
Gradient clipping	global norm 1.0

Appendix

Table 7: Shared training recipe of DbC, Disease-VL and Disease-V.

		Localization		Detection
Target	Model	Dice	IoU	AUROC	AP
HCC 91 CT / 191 L	SegVol	$.046$ $[.031,.063]$	$.028$ $[.019,.041]$	$.539$ $[.472,.605]$	$.205$ $[.159,.262]$
	SAT	$.092$ $[.062,.124]$	$.067$ $[.044,.093]$	$.558$ $[.490,.624]$	$\mathbf{.236}$ $\mathbf{[.176,.311]}$
	VoxTell	$.064$ $[.043,.088]$	$.043$ $[.029,.060]$	$.482$ $[.415,.549]$	$.211$ $[.162,.269]$
	Disease-VL †	$.128$ $[.098,.163]$	$.090$ $[.067,.117]$	$\mathbf{.615}$ $\mathbf{[.552,.678]}$	$.201$ $[.170,.236]$
	DbC	$\mathbf{.225}$ $\mathbf{[.180,.271]}$	$\mathbf{.176}$ $\mathbf{[.142,.219]}$	$.581$ $[.512,.648]$	$.188$ $[.149,.243]$

Appendix

Table 8: MCT-LTDiag per disease in the zero-shot regime, each disease scored with the leave-one-disease-out checkpoint that never saw it (95% CI, with the number of CTs and lesions under each disease). Macro averages over the four diseases give the zero-shot rows of Table 4 .

		Localization		Detection
Target	Model	Dice	IoU	AUROC	AP
HCC 91 CT / 191 L	SegVol	$.204$ $[.166,.245]$	$.151$ $[.122,.184]$	$.646$ $[.578,.713]$	$.371$ $[.283,.471]$
	SAT	$.121$ $[.088,.155]$	$.083$ $[.060,.108]$	$.626$ $[.562,.689]$	$.275$ $[.209,.361]$
	VoxTell	$.179$ $[.143,.218]$	$.127$ $[.101,.159]$	$.623$ $[.551,.693]$	$.322$ $[.244,.424]$
	Disease-VL †	$.182$ $[.148,.218]$	$.133$ $[.106,.164]$	$.638$ $[.568,.706]$	$.338$ $[.249,.434]$
	DbC	$\mathbf{.302}$ $\mathbf{[.263,.346]}$	$\mathbf{.254}$ $\mathbf{[.221,.292]}$	$\mathbf{.658}$ $\mathbf{[.585,.731]}$	$\mathbf{.413}$ $\mathbf{[.319,.522]}$

Appendix

Table 9: MCT-LTDiag per disease after full training on the internal data (95% CI, with the number of CTs and lesions under each disease). Macro averages over the four diseases give the full-training rows of Table 4 .

		Training fraction
Curve	Row	10%	20%	40%	60%	80%	100%
Detection AUROC all diseases	DbC	$.725$ $(\mathrm{sd}\,.009)$	$.768$ $(\mathrm{sd}\,.011)$	$.836$ $(\mathrm{sd}\,.016)$	$.864$ $(\mathrm{sd}\,.022)$	$.898$ $(\mathrm{sd}\,.005)$	$.918$ $(\mathrm{sd}\,.011)$
	Disease-VL †	$.700$ $(\mathrm{sd}\,.006)$	$.758$ $(\mathrm{sd}\,.007)$	$.822$ $(\mathrm{sd}\,.004)$	$.855$ $(\mathrm{sd}\,.013)$	$.882$ $(\mathrm{sd}\,.008)$	$.903$ $(\mathrm{sd}\,.014)$
	Significance	$\checkmark$	ns	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Localization Dice all diseases	DbC	$.070$ $(\mathrm{sd}\,.010)$	$.103$ $(\mathrm{sd}\,.011)$	$.164$ $(\mathrm{sd}\,.019)$	$.219$ $(\mathrm{sd}\,.014)$	$.275$ $(\mathrm{sd}\,.019)$	$.312$ $(\mathrm{sd}\,.006)$
	Disease-VL †	$.054$ $(\mathrm{sd}\,.001)$	$.094$ $(\mathrm{sd}\,.014)$	$.149$ $(\mathrm{sd}\,.010)$	$.208$ $(\mathrm{sd}\,.009)$	$.248$ $(\mathrm{sd}\,.021)$	$.282$ $(\mathrm{sd}\,.010)$

Appendix

Table 10: Seed means behind Figure 3 , the detection AUROC and localization Dice of DbC and Disease-VL at each training fraction, averaged over seeds 42, 43 and 44 with the across-seed standard deviation, over all seven diagnoses and the malignant subset (HCC, ICC, metastasis). The Sig. column shows a check mark when the 95% CI of the paired difference DbC minus Disease-VL, from the CT-level bootstrap of the seed mean, excludes zero, and ns otherwise.

Figure 6: Composition on six CTs from the test set, shown on one arterial-phase slice each, where every case has a coexisting lesion of another disease. Each row shows the CT with the ground truth outlined in orange for the target disease and in green for the coexisting disease, the finding maps of the required findings of the target disease as a blue overlay (probability 0 to 1), and the composed diagnosis map. In row 1, metastasis with hemangioma, low density is active on both lesions, the findings specific to metastasis (rim arterial enhancement, peripheral washout, ill-defined margin) are active on the metastasis, and the composed map follows them. In row 4, hemangioma with ICC, low density covers both lesions, the enhancement findings specific to hemangioma cover the hemangioma only, and the composed map excludes the ICC. In row 6, cyst with hemangioma, water density, no arterial enhancement and homogeneous density separate the cyst from the hemangioma although well-defined margin and round or oval shape are active on both, and the composed map excludes the hemangioma.

Medical Image Analysis

Medical image analysis uses computational methods to interpret clinical images, measure anatomy and disease, and support diagnosis and treatment.

See all 300 papers

Apr 15, 2026

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.

Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5

Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3

Radiology VLMs Explainable Medical Image Analysis Radiology Report Generation Medical Visual Question Answering

Ask PDF

Nov 2, 2026

Findings-aware Language-Image Pretraining for CT Volume Interpretation

Computed tomography (CT) provides diverse patient information and various clinical tasks, for which numerous deep learning models have been investigated. Recent years have seen the development of vision-language foundation models that can comprehensively extract clinical features, driven by representation learning that combines CT volumes with radiology reports. Existing approaches typically adopt Contrastive Language-Image Pre-training (CLIP)-based representation learning, where pairs from different cases are, in principle, treated as negative pairs such that the volume and report embeddings are pushed apart. In the biomedical imaging, however, treating different cases as negative pairs is not always appropriate, particularly when clinical similarity exists. In this study, we propose a supervised representation learning framework grounded in clinical knowledge from language foundation models, rather than the weak supervision used in CLIP. Specifically, we utilized a large language model to extract abnormal finding phrases from reports. We then explicitly constructed the target features that the image encoder should output, as the semantic representation of the extracted findings using a biomedical text embedding model. Our Findings-aware Language-Image Pre-training (FiLIP) outperformed weakly supervised approaches in multi-label zero-shot finding classification in both in-domain and external-domain evaluations, and also proved effective when fine-tuned for the downstream task of radiology report generation.

Takuro Shimaya, Junya Inoh, Masahiro Saiko

Biometrics Research Labs, NEC corporation, Kawasaki, Japan

Medical VLMs Medical Image Representation Learning Vision-Language Pretraining Zero-Shot Learning

Ask PDF

Oct 21, 2026

An interpretable deep learning method for early hepatocellular carcinoma diagnosis on multiphase CT

Early diagnosis of hepatocellular carcinoma is critical to preserve access to curative treatment options. The Liver Imaging Reporting and Data System (LI-RADS) provides a standardized framework for non-invasive diagnosis on multiphase computed tomography, based on major visual imaging features. However, these features may be difficult to assess consistently, particularly in small liver lesions. We developed dLI-RADS, an interpretable artificial intelligence method designed to estimate the probability of hepatocellular carcinoma from multiphase computed tomography images of manually segmented liver lesions, while remaining aligned with LI-RADS-related imaging concepts. dLI-RADS combines neural network-derived predictions with handcrafted quantitative imaging measurements reflecting LI-RADS major features and integrates these inputs using logistic regression. In this multicenter retrospective study, patients with multiphase computed tomography and at least one liver lesion were included, while lesions larger than 40 mm were excluded. Histology served as the reference standard for hepatocellular carcinoma, and a composite reference standard was used for non-hepatocellular carcinoma lesions. The lesion-level accuracy of dLI-RADS and LI-RADS were compared on the internal evaluation set using 10,000-repetition bootstrap non-inferiority tests, with 5% margins and significance levels. On the external test set, sensitivities were similarly compared. dLI-RADS was non-inferior to LI-RADS for lesion-level accuracy in the internal evaluation set, achieving 80% accuracy compared with 74% for LI-RADS (p = 0.04)...

Emma Sarfati, Alexandre Bône, Pietro Gori +8

Guerbet Research, Villepinte, France · LTCI, Télécom Paris, Institut Polytechnique de Paris, Saclay, France · MAP5 (UMR 8145), Université Paris Cité, 45 rue des Saints-Pères, Paris, France +7

Medical Diagnosis Medical Image Classification Explainable Medical Image Analysis Computed Tomography

Ask PDF

Abstract

Medical Image Analysis

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Findings-aware Language-Image Pretraining for CT Volume Interpretation

An interpretable deep learning method for early hepatocellular carcinoma diagnosis on multiphase CT