Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study

Zihao Zhao; Frederik Hauke; Juliana De Castilhos; Sven Nebelung; Daniel Truhn

cs.CVFeb 26, 2026

Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study

Authors: Zihao Zhao, Frederik Hauke, Juliana De Castilhos, Sven Nebelung, Daniel Truhn

Organizations: Department of Diagnostic and Interventional Radiology, University Hospital Aachen, 52074 Aachen, Germany

Abstract

The rapid progress of multimodal large language models (MLLMs) has led to increasing interest in agent-based systems. While most prior work in medical imaging concentrates on automating routine clinical workflows, we study an underexplored yet clinically significant setting: distinguishing visually hard-to-separate diseases in a zero-shot setting. We benchmark representative agents on two imaging-only proxy diagnostic tasks, (1) melanoma vs. atypical nevus and (2) pulmonary edema vs. pneumonia, where visual features are highly confounded despite substantial differences in clinical management. We introduce a multi-agent framework based on contrastive adjudication. Experimental results show improved diagnostic performance (an 11-percentage-point gain in accuracy on dermoscopy data) and reduced unsupported claims on qualitative samples, although overall performance remains insufficient for clinical deployment. We acknowledge the inherent uncertainty in human annotations and the absence of clinical context, which further limit the translation to real-world settings. Within this controlled setting, this pilot study provides preliminary insights into zero-shot agent performance in visually confounded scenarios.

Figures & tables

Figure 1: Illustration of hard-to-separate disease pairs. Despite highly overlapping visual patterns, their typical etiologies and managements differ significantly, which makes imaging-only differentiation challenging and high-stakes.

Figure 2: The overview of Contrastive Agent Reasoning (CARE). Two disease-specific agents generate opposing evidence from the same input image (e.g., melanoma vs. atypical nevus). A judge agent adjudicates the arguments, flags unsupported evidence, and outputs the final diagnosis in a training-free, zero-shot setting.

Method	Melanoma vs. Atypical Nevus			Edema vs. Pneumonia
Method	ACC	F1	Youden	ACC	F1	Youden
SigLIP2 [ 22 ]	0.611	0.606	0.219	0.502	0.349	0.006
BiomedCLIP [ 28 ]	0.607	0.569	0.209	0.557	0.554	0.114
MedVLM-R1 [ 17 ]	0.505	0.368	0.001	0.498	0.336	-0.003
InternVL3-14B [ 3 ]	0.549	0.534	0.071	0.502	0.361	0.002
Gemma-3-12B [ 19 ]	0.499	0.427	0.091	0.476	0.455	-0.041

Table 1: Benchmarking results on visually confounded diagnostic tasks. Gray : best single-agent method. Bold : best performance across all methods.

Figure 3: Representative cases showing how CARE exposes contradictory findings, implements cross-agent evidence recalibration, and verifies claims against the image.

Explore similar work

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making
May 14, 2026Yize Liu, Siyuan Yan, Ming Hu +5Medical Image Analysis LLM Agent Orchestration
AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom
Meissa: Multi-modal Medical Agentic Intelligence
Mar 9, 2026Yixiong Chen, Xinyi Bai, Yue Pan +2Medical LLMs Multimodal Agents
Johns Hopkins University, Baltimore, MD, USA · Cornell University, Ithaca, NY, USA
BioMed-Agent-RL: A Meta Learning, All You Need for Biomedical Applications
Aug 22, 2026Md Asaduzzaman Jabin, Zihao Wu, Tianming LiuLLM Agents Multimodal Agents
The University of Georgia, Athens, Georgia, GA 30605 USA

May 14, 2026J/K move · Enter open · S save

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

Yize Liu, Siyuan Yan, Ming Hu, Lie Ju+4

AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom

Medical Image Analysis LLM Agent Orchestration Tool-Using Vision-Language Agents Medical VLMs Explainable Medical Image Analysis

Open paper AskPDF arXiv

Dermatological diagnosis requires integrating fine-grained visual perception with expert clinical knowledge. Although Multimodal Large Language Models (MLLMs) facilitate interactive medical image analysis, their application in dermatology is hindered by insufficient domain-specific grounding and hallucinations. To address these issues, we propose DermAgent, a collaborative multi-tool agent that orchestrates seven specialized vision and language modules within a Plan-Execute-Reflect framework. DermAgent delivers stepwise, traceable diagnostic reasoning through three core components. First, it employs complementary visual perception tools for comprehensive morphological description, dermoscopic concept annotation, and disease diagnosis. Second, to overcome the lack of domain prior, a dual-modality retrieval module anchors every prediction in external evidence by cross-referencing 413,210 diagnosed image cases and 3,199 clinical guideline chunks. To further mitigate hallucinations, a deterministic critic module conducts strict post-hoc auditing via confidence, coverage, and conflict gates, automatically detecting inter-source disagreements to trigger targeted self-correction. Extensive experiments on five dermatology benchmarks demonstrate that DermAgent consistently outperforms state-of-the-art MLLMs and medical agent baselines across zero-shot fine-grained disease diagnosis, concept annotation, and clinical captioning tasks, exceeding GPT-4o by 17.6% in skin disease diagnostic accuracy and 3.15% in captioning ROUGE-L. Our code is available at https://github.com/YizeezLiu/DermAgent.

Abstract

Explore similar work

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

Meissa: Multi-modal Medical Agentic Intelligence

BioMed-Agent-RL: A Meta Learning, All You Need for Biomedical Applications