WeNLEX: Weakly Supervised Natural Language Explanations for Multilabel Chest X-ray Classification

Isabel Rio-Torto; Jaime S. Cardoso; Luís F. Teixeira

cs.CVMar 19, 2026

WeNLEX: Weakly Supervised Natural Language Explanations for Multilabel Chest X-ray Classification

Authors: Isabel Rio-Torto, Jaime S. Cardoso, Luís F. Teixeira

Organizations: INESC TEC · Universidade do Porto

Abstract

Natural language explanations provide an inherently human-understandable way to explain black-box models, closely reflecting how radiologists convey their diagnoses in textual reports. Most works explicitly supervise the explanation generation process using datasets annotated with explanations. Thus, though plausible, the generated explanations are not faithful to the model's reasoning. In this work, we propose WeNLEX, a weakly supervised model for the generation of natural language explanations for multilabel chest X-ray classification. Faithfulness is ensured by matching images generated from their corresponding natural language explanations with original images, in the black-box model's feature space. Plausibility is maintained via distribution alignment with a small database of clinician-annotated explanations. We empirically demonstrate, through extensive validation on multiple metrics to assess faithfulness, simulatability, diversity, and plausibility, that WeNLEX is able to produce faithful and plausible explanations, using as little as 5 ground-truth explanations per diagnosis. Furthermore, WeNLEX can operate in both post-hoc and in-model settings. In the latter, i.e., when the multilabel classifier is trained together with the rest of the network, WeNLEX improves the classification AUC of the standalone classifier by 2.21%, thus showing that adding interpretability to the training process can actually increase the downstream task performance. Additionally, simply by changing the database, WeNLEX explanations are adaptable to any target audience, and we showcase this flexibility by training a layman version of WeNLEX, where explanations are simplified for non-medical users.

Figures & tables

Figure 1: Architecture of WeNLEX , a weakly supervised model that generates natural language explanations (NLEs) for a multilabel X-ray classifier. For each predicted diagnosis (e.g., atelectasis, edema), it produces an NLE (only the atelectasis NLE is shown). A pretrained, frozen text-only Encoder–Decoder is adapted with soft prompt tuning ( Vision-to-Adapter and Soft Prompt ) to take as input the image features, the entire prediction vector (including diagnosis and evidence labels), and the textual label of the diagnosis being explained. The NLE Generator Encoder outputs an NLE embedding, which is compared to ground-truth NLE embeddings for that diagnosis to ensure plausibility ( Plausibility Loss ). Each NLE is also given to a Text Embedding to Image model, which generates an image depicting its content. This image is then processed by the model being explained (MBE) to extract features. To enforce faithfulness, the average of these features across all NLEs for an image is compared with the original image features ( Faithfulness Loss: Reconstruction ). Finally, each NLE must recover the MBE’s original diagnosis prediction: the MBE’s output for the generated image/NLE is compared against the original prediction ( Faithfulness Loss: Classification ). Trainable layers/parameters are represented by the fire icon, while frozen blocks are represented by the snowflake.

Fully Supervised
Model	Image-Related			NLE-Related
	Clf.	AUC	$\hat{y}\|img$	Plaus.	Recons.	Clf.	# NLEs	Faithfulness			Simulatability		Diversity ↓		Plausibility
									Deletion		$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS
								CLEV (F1)	Flip (%)	$\Delta p$	$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS
1		64.08	83.29	MSE	✗	✗	1107 (128)	10.37	97.63	0.565	89.97	88.26	8.785	0.7263	48.00
Weakly Supervised

Table 1: Comparison of WeNLEX with the fully supervised baseline of [ 21 ] in terms of Faithfulness, Simulatability, Diversity, and Plausibility. refers to the in-model scenario - the MBE is trained alongside the rest of WeNLEX ), while ❄ refers to the post-hoc scenario (MBE has been pretrained and is frozen during training of WeNLEX ). “Clf.” refers either to the image or the NLE classification loss. “Plaus.” refers to the plausibility loss and “Recons.” to the feature reconstruction loss. “# NLEs” is the total number of generated NLEs; values in parentheses are for correctly classified cases (diagnosis and evidence). All metrics use the total # NLEs, except CXBS, which uses only the parenthesised values. The ↓ means that for the diversity metrics lower is better. Best results are in bold; second-best are underlined.

Figure 2: Depiction of the deletion faithfulness metric: an image and a generated NLE are given to CheXagent, which grounds the text in the image. The identified regions are occluded, and the masked image is given to the model being explained (MBE). If the NLE is faithful, occluding these regions should significantly alter the MBE’s prediction.

Figure 3: CheXbert identified evidence is correct if it matches the evidence predicted by the model being explained (MBE) (which is also the evidence used to generate the NLE in the first place), but it is incorrect if judged against the ground-truth evidence. Since we want NLEs faithful to the MBE, the target evidence for computing the CLinical EVidence (CLEV) score should be the MBE’s predicted evidence.

Layer	# Feats.	Faithfulness			Simulatability		Diversity ↓		Plausibility
			Deletion		$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS
		CLEV (F1)	Flip (%)	$\Delta p$	$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS
classifier	30	9.623	91.81	0.3377	91.81	89.31	5.481	0.6580	46.96
gap	1024	7.993	92.39	0.3279	92.49	88.34	11.38	0.6927	46.98
denseblock4	50176	8.887	91.62	0.3483	86.13	88.05	2.424	0.5907	46.80
denseblock3	200704	8.631	91.16	0.3344	86.03	87.57	1.992	0.5767	46.46

Table 2: Ablation on the layer used for the feature reconstruction loss. “# Feats” is the dimension of the flattened feature map. “classifier” is the output layer, “gap” the GAP layer, and “denseblock” the outputs of DenseNet layers (with “denseblock1” closest to the input). The underlined layer is used in all other experiments. Best results are in bold; second-best are underlined.

# NLEs	Faithfulness			Simulatability		Diversity ↓		Plausibility
	CLEV (F1)	Deletion		$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS
	CLEV (F1)	Flip (%)	$\Delta p$	$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS
2	8.663 (0.9852)	92.76 (0.2796)	0.3530 (0.0342)	85.61 (4.097)	85.64 (0.6675)	8.544 (1.727)	0.6852 (0.0382)	47.40 (1.672)
5	9.615 (1.157)	92.14 (0.2282)	0.3539 (0.0324)	91.78 (1.404)	88.63 (0.4199)	3.560 (1.551)	0.6331 (0.0158)	44.88 (3.371)
10	8.396 (0.1789)	92.33 (0.4732)	0.3749 (0.0016)	92.29 (3.424)	88.34 (0.3474)	1.827 (0.0461)	0.6276 (0.0348)	45.22 (1.422)
20	8.672 (1.735)	92.52 (0.4358)	0.3616 (0.0106)	92.77 (2.569)	88.05 (0.4199)	2.090 (0.6986)	0.5976 (0.0400)	46.45 (0.9763)

Table 3: Ablation on the number of NLEs per diagnosis present in the GT NLE database. The underlined # NLEs is used in all other experiments. Best results are in bold; second-best are underlined.

Database	Faithfulness			Simulatability		Diversity ↓		Plausibility	Readability ↓
	CLEV (F1)	Deletion		$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS	Gen.	DB
	CLEV (F1)	Flip (%)	$\Delta p$	$\hat{y}\|NLE$	$\hat{y}\|(img,NLE)$	S-BLEU	Retrv.	CXBS	Gen.	DB
Medical	9.401	91.48	0.3328	90.46	88.82	4.037	0.6286	47.35	13.21	15.00
Layman	4.025	91.43	0.2842	64.35	84.30	4.534	0.5295	39.49	10.42	10.46

Table 4: Comparison of WeNLEX trained on clinician vs. simplified layperson NLEs. The simplified version shows higher readability (lower required grade level). “Gen.” denotes generated NLEs, and “DB” the GT NLE from the medical and non-medical databases.

Figure 4: Qualitative examples of WeNLEX for three different diagnoses, comparison with ground-truth NLEs, and with the layman version of WeNLEX , in which the generated NLEs are simplified to adapt to a non-medical audience.

Natural Language Explanation Generation

See all 13 papers

A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation
Apr 1, 2026Yabin Zhang, Chong Wang, Yunhe Gao +19Radiology VLMs VLM Reasoning
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA. +4
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
May 19, 2026Guangzhi Xiong, Qiao Jin, Sanchit Sinha +2Medical VQA VLM Interpretability
University of Virginia · National Institutes of Health
Explanation-Aware Learning for Enhanced Interpretability in Biomedical Imaging
May 11, 2026Zubair Faruqui, Rahul DubeyMedical Image Classification Explainable Medical Image Analysis
Department of Computer Science, Missouri State University

Apr 1, 2026J/K move · Enter open · S save

A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation

Yabin Zhang, Chong Wang, Yunhe Gao, Jiaming Liu+18

Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA.+4

Radiology VLMs VLM Reasoning Explainable Medical Image Analysis Radiology Report Generation Chest X-Ray Classification

Open paper AskPDF arXiv

Chest X-rays (CXRs) are among the most frequently performed imaging examinations worldwide, yet rising imaging volumes increase radiologist workload and the risk of diagnostic errors. Although artificial intelligence (AI) systems have shown promise for CXR interpretation, most generate only final predictions, without making explicit how visual evidence is translated into radiographic findings and diagnostic predictions. We present CheXOne, a reasoning-enabled vision-language model for CXR interpretation. CheXOne jointly generates diagnostic predictions and explicit, clinically grounded reasoning traces that connect visual evidence, radiographic findings, and these predictions. The model is trained on 14.7 million instruction and reasoning samples curated from 30 public datasets spanning 36 CXR interpretation tasks, using a two-stage framework that combines instruction tuning with reinforcement learning to improve reasoning quality. We evaluate CheXOne in zero-shot settings across visual question answering, report generation, visual grounding and reasoning assessment, covering 17 evaluation settings. CheXOne outperforms existing medical and general-domain foundation models and achieves strong performance on independent public benchmarks. A clinical reader study demonstrates that CheXOne-drafted reports are comparable to or better than resident-written reports in 55% of cases, while effectively addressing clinical indications and enhancing both report writing and CXR interpretation efficiency. Further analyses involving radiologists reveal that the generated reasoning traces show high clinical factuality and provide causal support for the final predictions, offering a plausible explanation for the performance gains. These results suggest that explicit reasoning can improve model performance, interpretability and clinical utility in AI-assisted CXR interpretation.

Abstract

Natural Language Explanation Generation

Explore similar work

A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation

Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

Explanation-Aware Learning for Enhanced Interpretability in Biomedical Imaging