SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis

Hangyul Yoon; Hyungyung Lee; Edward Choi; Eunho Yang

cs.CVSep 28, 2026

SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis

Authors: Hangyul Yoon, Hyungyung Lee, Edward Choi, Eunho Yang

Organizations: Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Vision-language (VL) pretraining using paired chest X-ray (CXR) images and radiology reports has shown strong potential for medical image understanding. However, existing methods often remain dependent on task-specific finetuning because radiology reports are lengthy, clinically dense, and difficult to align with simple zero-shot prompts. Recent sentence-level approaches partially address this limitation using clinical phrases extracted by large language models (LLMs), but they largely overlook the intrinsic characteristics of radiology discourse. In particular, limited positive-pair diversity constrains further gains, while clinically equivalent sentences frequently recur across patients, creating false negatives in contrastive learning. To address these issues, we propose SentZero, an enhanced sentence-centric VL pretraining framework for zero-shot, multi-task CXR analysis. SentZero introduces LLM-based abstract-level sentence structuring and mapping to expand positive-pair diversity, together with an additional loss term to mitigate false negatives. We further introduce sentence-conditioned residual modulation of visual embeddings, enabling visual features to adapt to the semantic characteristics of each input sentence. Across diverse downstream tasks and datasets, SentZero improves zero-shot generalization and outperforms prior multi-task zero-shot methods.

Figures & tables

Figure 1 : Overview of the SentZero training framework. SentZero performs multi-pair contrastive learning between sentence embeddings and weighted-sum patch embeddings (bold red line, lower-right panel). Structured tuples extracted from the findings generate augmented positive pairs through abstract-level mapping. The model is trained with a combination of the contrastive loss ( $\mathcal{L}_{con}$ ) and the false-negative mitigation loss ( $\mathcal{L}_{fn}$ ). Before contrastive learning, the patch embeddings are refined via a residual connection with a text-conditioning module (lower left). At inference, the similarity logits and patch attention scores support multiple zero-shot downstream tasks.

Method	Venue	Classification					Grounding
		(AUROC)					(Pointing Acc.)
		OpenI	CXR14	PadChest	CXD10	CheXPert	CXD10	MS-CXR
GLoRIA	ICCV’21	0.589	0.610	0.565	0.645	0.750	0.367	-
BioViL-T	CVPR’23	0.702	0.729	0.655	0.708	0.789	0.351	0.719
MedKLIP	ICCV’23	0.759	0.726	0.629	0.713	0.879	0.481	0.407
KAD	Nat. Comm.’23	0.807	0.789	0.750	0.735	0.905	0.391	-

Table 1: Zero-shot performance comparison results. The first header row indicates the task type and corresponding evaluation metric. The best and second-best results are shown in bold and underlined , respectively. CXR14 and CXD10 refer to the ChestXray14 and ChestXDet10 datasets, respectively.

Component				Classification					Grounding
MF	ALM	TC	FN	OpenI	CXR14	PadChest	CXD10	CheXPert	CXD10	MS-CXR
				0.8747	0.8140	0.8592	0.8090	0.9080	0.6513	0.8922
✓				0.8741	0.8164	0.8559	0.8093	0.9109	0.6655	0.9222
✓	✓			0.8856	0.8323	0.8635	0.8395	0.8978	0.6811	0.9162
✓		✓		0.8828	0.8236	0.8652	0.8226	0.9162	0.7032	0.8922
✓			✓	0.8774	0.8239	0.8607	0.8230	0.9155	0.6556	0.8862

Table 2: Ablation results for the model components. MF – Multi-Layer Feature Aggregation; ALM – Abstract-Level Mapping; TC – Text Conditioned Residual Connection; FN – False Negative Mitigation. Best results in each column are in bold .

Method	Classification					Grounding
Method	OpenI	CXR14	PadChest	CXD10	CheXPert	CXD10	MS-CXR
None	0.8858	0.8376	0.8627	0.8431	0.9047	0.7121	0.9042
False Negative Masking	0.8815	0.8334	0.8642	0.8444	0.8981	0.6923	0.8982
False Negative Transition	0.8492	0.8139	0.7904	0.7951	0.9067	0.5138	0.7066
Ours	0.8891	0.8389	0.8609	0.8433	0.9037	0.7315	0.9222

Table 3: Comparison of false negative mitigation strategies. Best mitigation results in each column are in bold .

Panel of Figure 2 — Figure 2 : Examples of attention heatmaps for relatively small lesions from the ChestXDet10 dataset. Each example shows the original image and its attention map for the given text prompt. Green boxes are the ground-truth bounding boxes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3 : Instruction prompt used to extract structured tuples for abstract-level sentence mapping.

	Classification					Grounding
	OpenI	CXR14	PadChest	CXD10	CheXPert	CXD10	MS-CXR
w/o text feature	0.8804	0.8358	0.8599	0.8523	0.9000	0.7298	0.9042
w/ text feature	0.8891	0.8389	0.8609	0.8433	0.9037	0.7315	0.9222

Appendix

Table 4: Ablation results of text feature conditioning in feature modulation units.

$\lambda_{fn}$	Classification					Grounding
$\lambda_{fn}$	OpenI	CXR14	PadChest	CXD10	CheXPert	CXD10	MS-CXR
0.01	0.8891	0.8389	0.8609	0.8433	0.9037	0.7315	0.9222
0.03	0.8851	0.8381	0.8637	0.8490	0.9074	0.7018	0.8922
0.05	0.8882	0.8377	0.8591	0.8510	0.9047	0.7209	0.9341

Appendix

Table 5: Effect of false negative loss coefficient.

$M$	Classification					Grounding
$M$	OpenI	CXR14	PadChest	CXD10	CheXPert	CXD10	MS-CXR
10%	0.8815	0.8350	0.8621	0.8457	0.9039	0.6998	0.9281
20%	0.8891	0.8389	0.8609	0.8433	0.9037	0.7315	0.9222
30%	0.8818	0.8335	0.8613	0.8489	0.8951	0.7168	0.9341

Appendix

Table 6: Effect of false negative patch sampling ratio, with

\lambda_{fn}=0.01

Panel of Figure 4 — Figure 4 : Additional attention heatmap examples from the ChestXDet10 dataset.

Panel of Figure 5 — Figure 5 : Additional attention heatmap examples from the MS-CXR dataset.

Explore similar work

ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification
Apr 20, 2026Florian Kittler, Sheethal Bhat, Andreas MaierChest X-Ray Vision-Language Model Adaptation
Friedrich-Alexander University Erlangen-Nuremberg, Erlangen 91054, Germany
Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings
May 19, 2026Chenyu Lian, Hong-Yu Zhou, Chun-Ka Wong +1Chest X-Ray Adaptive Boundary Suppression
The Center for Smart Health, School of Nursing, the Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Smart Ageing, the Hong Kong Polytechnic University,2026 Hong Kong, China · School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China +1
PLRS-IC: A Dual-Calibration Framework for Chest X-Ray Vision-Language Alignment
Sep 30, 2026Qixing Zhao, Jinpeng LiVision-Language Alignment Chest X-Ray
South China University of Technology

Apr 20, 2026, cs.LGJ/K move · Enter open · S save

ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification

Florian Kittler, Sheethal Bhat, Andreas Maier

Friedrich-Alexander University Erlangen-Nuremberg, Erlangen 91054, Germany

Chest X-Ray Vision-Language Model Adaptation

Open paper AskPDF arXiv

Zero-shot vision-language models (VLMs) have shown promise for chest radiograph classification, but their performance is often limited by confounding label co-occurrence, long-tail class imbalance, and transfer instability under domain shift. We propose ProtoCLIP, a refinement strategy for CLIP-style VLMs that improves zero-shot discrimination through targeted data curation and distilled anchor alignment. Specifically, we construct pathology-focused training subsets with curated negative samples to reduce co-occurrence bias. We also introduce a representation-preserving distillation objective to stabilize adaptation while maintaining semantic structure and improving discrimination of clinically relevant co-occurring pathologies. Evaluated on an unseen dataset VinDr-CXR, ProtoCLIP improves AUC by 2-10 percentage points over a strong CLIP-based baseline across multiple findings. For pneumothorax specifically, ProtoCLIP achieves a state-of-the-art AUC of 0.94. These results demonstrate that anchor-guided refinement, coupled with curated supervision and controlled adaptation, can mitigate common zero-shot transfer failures in medical VLMs without requiring large-scale retraining.

Abstract

Explore similar work

ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification

Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings

PLRS-IC: A Dual-Calibration Framework for Chest X-Ray Vision-Language Alignment