Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

Dimitrios P. Panagoulias; Evangelia-Aikaterini Tsichrintzi; Georgios Savvidis; Evridiki Tsoureli-Nikita

cs.AIFeb 26, 2026

Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

Authors: Dimitrios P. Panagoulias, Evangelia-Aikaterini Tsichrintzi, Georgios Savvidis, Evridiki Tsoureli-Nikita

Organizations: Department of Informatics, University of Piraeus, Karaoli ke Dimitriou 80, Piraeus, 18534, Greece · Department of Research & Development, Noetiv PC, Valaoritou 18, Athens, 10671, Greece · Department of Dermatology, Dermacen SA, Valaoritou 18, Athens, 10671, Greece

Abstract

Human-in-the-loop validation is essential in safety-critical clinical AI, yet the transition between initial model inference and expert correction is rarely analyzed as a structured signal. We introduce a diagnostic alignment framework in which the AI-generated image based report is preserved as an immutable inference state and systematically compared with the physician-validated outcome. The inference pipeline integrates a vision-enabled large language model, BERT- based medical entity extraction, and a Sequential Language Model Inference (SLMI) step to enforce domain-consistent refinement prior to expert review. Evaluation on 21 dermatological cases (21 complete AI physician pairs) em- ployed a four-level concordance framework comprising exact primary match rate (PMR), semantic similarity-adjusted rate (AMR), cross-category alignment, and Comprehensive Concordance Rate (CCR). Exact agreement reached 71.4% and remained unchanged under semantic similarity (t = 0.60), while structured cross-category and differential overlap analysis yielded 100% comprehensive concordance (95% CI: [83.9%, 100%]). No cases demonstrated complete diagnostic divergence. These findings show that binary lexical evaluation substantially un- derestimates clinically meaningful alignment. Modeling expert validation as a structured transformation enables signal-aware quantification of correction dynamics and supports traceable, human aligned evaluation of image based clinical decision support systems.

Figures & tables

Metric	Physician 1	Physician 2	Combined
Exact Primary Matches	13/16	2/5	15/21 (71.4%)
Similarity-Adjusted	13/16	2/5	15/21 (71.4%)
Cross-Category Cases	3/16	2/5	5/21 (23.8%)
Cases with Any Match	16/16	5/5	21/21
CCR (%)	100	100	100
95% CI (CCR)	–	–	[83.9%, 100%]

Table 1: Diagnostic Concordance Summary (N=21)

Figure 1: Primary diagnosis distribution and agreement composition for $R_{0}$ and $R_{1}$ .

Figure 2: Differential counts and modification patterns between $R_{0}$ and $R_{1}$ . Bars show number of differentials per case; pie charts summarize addition, removal, and unchanged proportions.

AI	Artificial Intelligence
AMR	Adjusted Match Rate
CCR	Comprehensive Concordance Rate
CDSS	Clinical Decision Support System
CI	Confidence Interval
CNN	Convolutional Neural Network
HITL	Human-in-the-Loop

Table 4

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Apr 15, 2026Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5Radiology VLMs Explainable Medical Image Analysis
Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-Judge Healthcare
aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa +3
LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
Feb 4, 2026Ruixiao Yang, Yuanhe Tian, Di Dong +3Radiology Report Generation Language Model Decoding
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1

Apr 15, 2026J/K move · Enter open · S save

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang, Yue Yang+4

Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK+3

Radiology VLMs Explainable Medical Image Analysis Radiology Report Generation Medical VQA

Open paper AskPDF arXiv

Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.

Abstract

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?

LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation