Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis

Shaoxuan Wu; Jingkun Chen; Chong Ma; Cong Shen; Xiao Zhang; Jun Feng

cs.CVFeb 25, 2026

Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis

Authors: Shaoxuan Wu, Jingkun Chen, Chong Ma, Cong Shen, Xiao Zhang, Jun Feng

Organizations: College of Computer Science, Northwest University, Xi’an, China, 710127 · Department of Engineering Science, University of Oxford, Oxford, OX3 7DQ, United Kingdom · School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, China · Department of PET/CT, The First Affiliated Hospital of Xi’an Jiaotong University, Xi’an, 710061, China

Abstract

Computer-aided diagnosis (CAD) has significantly advanced automated chest X-ray diagnosis but remains isolated from clinical workflows and lacks reliable decision support and interpretability. Human-AI collaboration seeks to enhance the reliability of diagnostic models by integrating the behaviors of controllable radiologists. However, the absence of interactive tools seamlessly embedded within diagnostic routines impedes collaboration, while the semantic gap between radiologists' decision-making patterns and model representations further limits clinical adoption. To overcome these limitations, we propose a visual cognition-guided collaborative network (VCC-Net) to achieve the cooperative diagnostic paradigm. VCC-Net centers on visual cognition (VC) and employs clinically compatible interfaces, such as eye-tracking or the mouse, to capture radiologists' visual search traces and attention patterns during diagnosis. VCC-Net employs VC as a spatial cognition guide, learning hierarchical visual search strategies to localize diagnostically key regions. A cognition-graph co-editing module subsequently integrates radiologist VC with model inference to construct a disease-aware graph. The module captures dependencies among anatomical regions and aligns model representations with VC-driven features, mitigating radiologist bias and facilitating complementary, transparent decision-making. Experiments on the public datasets SIIM-ACR, EGD-CXR, and self-constructed TB-Mouse dataset achieved classification accuracies of 88.40%, 85.05%, and 92.41%, respectively. The attention maps produced by VCC-Net exhibit strong concordance with radiologists' gaze distributions, demonstrating a mutual reinforcement of radiologist and model inference. The code is available at https://github.com/IPMI-NWU/VCC-Net.

Figures & tables

Fig. 1: The collaborative paradigm (right) leverages radiologists’ visual cognition to bridge radiologists’ cognition and model inference. Compared with conventional CAD (left), it enhances the consistency and reliability in clinical decisions.

Fig. 2: The proposed VCC-Net comprises two main components: (1) The VAG employs GNN and CNN to model both global and local visual search patterns of radiologists, and supervises learning through three pathways. The VAG generates radiologist-like visual attention based on the input medical images. (2) The VCC leverages radiologists’ VC to construct a graph structure and align visual and feature differences across regions, ultimately producing diagnostic outputs.

Fig. 3: The CGCM aligns feature and visual distances for each image node by incorporating radiologists’ VC, optimizing the feature space structure. With the addition of soft visual attention. The CGCM also guides the network in constructing a graph structure concentrated on high-attention regions, establishing a robust model aligned with radiologists’ cognition.

Method	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
$\spadesuit$ ResNet18 [ 1 ]	83.20	82.35	85.25
$\spadesuit$ ResNet50 [ 1 ]	84.00	85.81	83.63
$\spadesuit$ ResNet101 [ 1 ]	84.40	86.13	81.08
$\spadesuit$ ViT [ 2 ]	83.60	84.16	83.77
$\spadesuit$ SwinT [ 3 ]	84.40	83.31	83.69
$\spadesuit$ ViG [ 4 ]	83.20	84.89	82.81

TABLE I: Quantitative comparison between our method and other approaches on the SIIM-ACR dataset. Bold values represent the best results.

Method	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
$\spadesuit$ ResNet18 [ 1 ]	71.96	85.02	72.17
$\spadesuit$ ResNet50 [ 1 ]	72.90	86.43	72.45
$\spadesuit$ ResNet101 [ 1 ]	74.77	85.63	74.90
$\spadesuit$ ViT [ 2 ]	70.09	85.43	69.19
$\spadesuit$ SwinT [ 3 ]	71.96	86.44	74.39
$\spadesuit$ ViG [ 4 ]	75.70	85.71	75.62

TABLE II: Quantitative comparison between our method and other approaches on the EGD-CXR dataset. Bold values represent the best results.

Fig. 4: Visualization of attention maps, generated with Grad-CAM, comparing different methods on the SIIM-ACR dataset. In the first column, the pneumothorax region is highlighted in yellow. The red areas in the attention maps reflect the focus regions of each network. The second column shows that VCC-Net achieves superior anomaly localization by accurately focusing on the pneumothorax region in comparison to other models.

Fig. 5: Visualization of attention maps obtained from Grad-CAM comparing methods on the TB-Mouse dataset. In the first column, regions with exudation and nodules are marked with bounding boxes. Red areas indicate the focused regions in each network’s attention map. Compared to other methods, VCC-Net accurately localizes lesion areas.

Fig. 6: Comparison of visual attention generated by various methods on the SIIM-ACR dataset. Compared to others, the visual attention generated by our method closely matches that of real radiologists. The yellow circle highlights an area missed by our method that is non-abnormal on the X-ray, underscoring that VCC-Net predicted visual attention is highly aligned with actual abnormalities.

Method	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
$\spadesuit$ ResNet18 [ 1 ]	83.73	92.23	83.75
$\spadesuit$ ResNet50 [ 1 ]	88.95	95.87	88.96
$\spadesuit$ ResNet101 [ 1 ]	87.86	96.06	87.80
$\spadesuit$ ViT [ 2 ]	88.23	95.64	88.20
$\spadesuit$ SwinT [ 3 ]	88.18	94.95	88.05
$\spadesuit$ ViG [ 4 ]	88.40	95.17	88.33

TABLE III: Quantitative comparison between our method and other methods on the TB-Mouse dataset. Bold values represent the best results.

SIIM-ACR
$\mathcal{L}_{soft}$	$\mathcal{L}_{hard}$	$\mathcal{L}_{aux}$	$\mathcal{L}_{align}$	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
✓				85.20	85.53	84.66
✓	✓			86.40	85.90	86.58
✓		✓		86.40	85.88	85.97
✓	✓	✓		86.80	86.91	86.32
✓			✓	87.20	87.59	87.37

TABLE IV: Quantitative results from ablation studies on VCC-Net components for the SIIM-ACR and TB-Mouse datasets. Bold values represent the best results.

Fig. 7: Ablation study on the hyperparameters $\lambda_{align}$ and $\lambda_{VAG}$ , which adjust feature alignment strength and the contribution of visual attention generation, respectively. Acc, F1, and AUC for the SIIM-ACR data are shown across different parameter settings. Optimal performance is achieved when both parameters are set to $0.5$ , with consistency observed across various settings, underscoring model stability.

SIIM-ACR
Method	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
VCC	83.20	84.89	82.81
VAG (CNN) +VCC	85.60	82.94	85.14
VAG (GNN) +VCC	87.20	85.07	86.14
VAG (GNN+CNN) +VCC	88.40	86.12	88.08
TB-Mouse

TABLE V: Quantitative ablation results on VAG structure for the SIIM-ACR and TB-Mouse datasets. Bold values represent the best results

SIIM-ACR
Method	Acc $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
Random	82.40	82.64	82.64
Radiologists’ VC	86.00	84.84	85.08
VAG	88.40	86.12	88.08
VAG+Radiologists’ VC	88.80	88.56	88.54
TB-Mouse

TABLE VI: Comparison of performance using different types of visual attention. Bold indicates the best results.

Fig. 8: Visualization reveals instances where radiologists, due to subjectivity or fatigue, misinterpret non-lesion structures as suspicious areas or overlook pneumothorax regions. The yellow-circled areas exemplify such occurrences.

Fig. 9: Visualization of graph structures under different distance metrics. Examples from the SIIM-ACR and TB-Mouse datasets are shown. The red circle represents the central node with yellow neighboring nodes. Yellow areas highlight abnormal regions.

Fig. 10: Visualization of distance maps based on various distance metrics. Examples from the SIIM-ACR and TB-Mouse datasets are shown. Darker colors indicate proximity to the central white point.

Explore similar work

Apr 15, 2026

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.

Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5

Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3

Radiology VLMs Explainable Medical Image Analysis Radiology Report Generation Medical VQA

Ask PDF

Mar 27, 2026

Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays

Despite recent advances in medical vision-language pretraining, existing models still struggle to capture the diagnostic workflow: radiographs are typically treated as context-agnostic images, while radiologists' gaze -- a crucial cue for visual reasoning -- remains largely underexplored by existing methods. These limitations hinder the modeling of disease-specific patterns and weaken cross-modal alignment. To bridge this gap, we introduce CoGaze, a Context- and Gaze-guided vision-language pretraining framework for chest X-rays. We first propose a context-infused vision encoder that models how radiologists integrate clinical context -- including patient history, symptoms, and diagnostic intent -- to guide diagnostic reasoning. We then present a multi-level supervision paradigm that (1) enforces intra- and inter-modal semantic alignment through hybrid-positive contrastive learning, (2) injects diagnostic priors via disease-aware cross-modal representation learning, and (3) leverages radiologists' gaze as probabilistic priors to guide attention toward diagnostically salient regions. Extensive experiments demonstrate that CoGaze consistently outperforms state-of-the-art methods across diverse tasks, achieving up to +2.0% CheXbertF1 and +1.2% BLEU2 for free-text and structured report generation, +23.2% AUROC for zero-shot classification, and +12.2% Precision@1 for image-text retrieval. Code is available at https://github.com/mk-runner/CoGaze.

Kang Liu, Zhuoqi Ma, Siyu Liang +5

Xidian University, Xi’an, China · Wuhan University, Wuhan, China

Radiology VLMs Medical VLMs Medical Image Analysis Radiology Report Generation Chest X-Ray Classification

Ask PDF

Apr 29, 2026

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

Chest X-ray interpretation is one of the most frequently performed diagnostic tasks in medicine and a primary target for AI development, yet current vision-language models are primarily trained on datasets of paired images and reports, not the cognitive processes and visual attention that underlie clinical reasoning. Here, we present CheXthought, a global, multimodal resource containing 103,592 chain-of-thought reasoning traces and 6,609,082 synchronized visual attention annotations across 50,312 multi-read chest X-rays from 501 radiologists in 71 countries. Our analysis reveals clinical reasoning patterns in how experts deploy distinct visual search strategies, integrate clinical context, and communicate uncertainty. We demonstrate the clinical utility of CheXthought across four dimensions. First, CheXthought reasoning significantly outperforms state-of-the-art vision-language model chain-of-thought in factual accuracy and spatial grounding. Second, visual attention data used as an inference-time hint recovers missed findings and significantly reduces hallucinations. Third, vision-language models trained on CheXthought data achieve significantly stronger pathology classification, visual faithfulness, temporal reasoning and uncertainty communication. Fourth, leveraging CheXthought's multi-reader annotations, we predict both human-human and human-AI disagreement directly from an image, enabling transparent communication of case difficulty, uncertainty and model reliability. These findings establish CheXthought as a resource for advancing multimodal clinical reasoning and the development of more transparent, interpretable vision-language models.

Sonali Sharma, Jin Long, George Shih +7

Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA · Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA +6

Chest X-Ray Classification Medical Image Analysis Medical VLMs Clinical Reasoning Multimodal CoT Reasoning

Ask PDF