CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

Geon Choi; Hangyul Yoon; Nalee Kim; Jeong Yun Jang; Hyunju Shin; Hyunki Park; Sang Hoon Seo; Edward Choi

cs.CVJun 19, 2026

CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

Authors: Geon Choi, Hangyul Yoon, Nalee Kim, Jeong Yun Jang, Hyunju Shin, Hyunki Park, Sang Hoon Seo, Edward Choi

Organizations: 1KAIST · 2Samsung Medical Center · 3Konkuk University Medical Center

Abstract

The evaluation of vision-language models (VLMs) for chest X-ray (CXR) analysis has largely been limited to disease-presence classification without visual grounding. Such evaluations fail to verify the expert-level lesion perception necessary to ensure the clinical reliability of VLMs. To address these limitations, we introduce CheXpercept, a sequential, multi-level perception benchmark that mirrors a radiologist's cognitive workflow across coarse-level detection, fine-level contour evaluation and revision, and semantic-level attribute extraction. To ensure high clinical fidelity at scale, we construct the dataset using a semi-automated generation pipeline paired with a review by six medical experts. CheXpercept contains 10,400 QA items derived from 2,100 CXRs, covering seven clinically critical pulmonary and cardiac lesions. To demonstrate the current landscape of VLM perception, we benchmark 14 general and medical VLMs on CheXpercept. The models achieve adequate performance only at the coarse level, with accuracy degrading precipitously on deeper visual tasks. Notably, medical VLMs show almost no perceptual advantage over their general-domain counterparts, highlighting a systemic flaw in current domain adaptation. The code and dataset will be publicly available.

Explore similar work

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation
Apr 29, 2026Sonali Sharma, Jin Long, George Shih +7Chest X-Ray Medical Vision-Language Models
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
May 19, 2026Guangzhi Xiong, Qiao Jin, Sanchit Sinha +2Medical Vision-Language Models Chest X-Ray
CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs
Jun 7, 2026Sergios Gatidis, Curtis Langlotz, Christian BluethgenMedical Vision-Language Models Chest Radiography

Apr 29, 2026 · cs.CVJ/K move · Enter open · S save

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

Sonali Sharma, Jin Long, George Shih, Sarah Eid+6

Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA · Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA · Department of Radiology, Weill Cornell Medicine, New York, NY, USA · Department of Radiology, Brigham and Women’s Hospital, Boston, MA, USA

Chest X-ray interpretation is one of the most frequently performed diagnostic tasks in medicine and a primary target for AI development, yet current vision-language models are primarily trained on datasets of paired images and reports, not the cognitive processes and visual attention that underlie clinical reasoning. Here, we present CheXthought, a global, multimodal resource containing 103,592 chain-of-thought reasoning traces and 6,609,082 synchronized visual attention annotations across 50,312 multi-read chest X-rays from 501 radiologists in 71 countries. Our analysis reveals clinical reasoning patterns in how experts deploy distinct visual search strategies, integrate clinical context, and communicate uncertainty. We demonstrate the clinical utility of CheXthought across four dimensions. First, CheXthought reasoning significantly outperforms state-of-the-art vision-language model chain-of-thought in factual accuracy and spatial grounding. Second, visual attention data used as an inference-time hint recovers missed findings and significantly reduces hallucinations. Third, vision-language models trained on CheXthought data achieve significantly stronger pathology classification, visual faithfulness, temporal reasoning and uncertainty communication. Fourth, leveraging CheXthought's multi-reader annotations, we predict both human-human and human-AI disagreement directly from an image, enabling transparent communication of case difficulty, uncertainty and model reliability. These findings establish CheXthought as a resource for advancing multimodal clinical reasoning and the development of more transparent, interpretable vision-language models.

Chest X-Ray Medical Vision-Language Models Multimodal Clinical Data Large Vision Language Models Multimodal Dataset

Open paper Ask about it PDF

Abstract

Explore similar work

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs