MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data

Zhenghao Zhu; Chuxue Cao; Sirui Han; Yuanfeng Song; Xing Chen; Caleb Chen Cao; Yike Guo

cs.AIDec 15, 2025

MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data

Authors: Zhenghao Zhu, Chuxue Cao, Sirui Han, Yuanfeng Song, Xing Chen, Caleb Chen Cao, Yike Guo

Organizations: The Hong Kong University of Science and Technology, Hong Kong, China · ByteDance, China

Abstract

In medical data analysis, extracting deep insights from complex, multi-modal datasets is essential for improving patient care, increasing diagnostic accuracy, and optimizing healthcare operations. However, there is currently a lack of high-quality datasets specifically designed to evaluate the ability of large multi-modal models (LMMs) to discover medical insights. In this paper, we introduce MedInsightBench, the first benchmark that comprises 332 carefully curated medical cases, each annotated with thoughtfully designed insights. This benchmark is intended to evaluate the ability of LMMs and agent frameworks to analyze multi-modal medical image data, including posing relevant questions, interpreting complex findings, and synthesizing actionable insights and recommendations. Our analysis indicates that existing LMMs exhibit limited performance on MedInsightBench, which is primarily attributed to their challenges in extracting multi-step, deep insights and the absence of medical expertise. Therefore, we propose MedInsightAgent, an automated agent framework for medical data analysis, composed of three modules: Visual Root Finder, Analytical Insight Agent, and Follow-up Question Composer. Experiments on MedInsightBench highlight pervasive challenges and demonstrate that MedInsightAgent can improve the performance of general LMMs in medical data insight discovery.

Figures & tables

Dataset	Input	Output	Task Type	Topic & Area	Multi-Step	Size	Construction Method
Spider 2.0 Lei et al. (2025)	Question	SQL Query	Text-to-SQL	Enterprise-level	Limited	632	Machine & Human-Labeled
VisEval Chen et al. (2025)	Question+Chart	Evaluation	Visual Understanding	Data Visualization	No	500+	Human-Labeled
MatPlotBench Yang et al. (2024)	Question+Table	Vis Image	Visual Generation	Data Visualization	Partial	100	Machine & Human-Labeled
InfiAgent-DABench Hu et al. (2024)	Question+Table	Answer	Data Analysis	Data Analysis	Partial	603	Machine-Labeled
InsightBench Sahu et al. (2025)	Goal+Table	Insights	Insight Discovery	Business Analysis	Extensive	100	Human-Labeled
MedRepBench Shang et al. (2026)	Image+Report	Interpretation	Medical Report Analysis	Clinical Analysis	Limited	1K+	Human-Annotated

Table 1: Comparison of MedInsightBench with existing benchmarks. Our benchmark is the first to evaluate goal-driven multi-step medical insight discovery from image-modal medical data.

Figure 1: The dataset construction pipeline of MedInsightBench. This pipeline consists of 3 steps: 1) Pathological Image Processing . WSIs are standardized and quality-checked. 2) Report Processing & Insight Extraction . Reports are converted to text, insights and questions are generated, and verified by experts. 3) Goal Generation . An overarching analysis goal is synthesized from the questions and validated for guiding the analysis.

Figure 2: Quality Comparison Between LLM Evaluation and Human Annotation.

Figure 3: Comparison of average TF - IDF cosine (TC) similarity, Self - BLEU, and Distinct - 2 diversity scores between Question and Insight in MedInsightBench.

Figure 4: Overview of our proposed agentic framework, MedInsightAgent , for goal-driven medical insight discovery. It consists of three collaborative components: (1) Visual Root Finder for image summarization and root-question generation; (2) Analytical Insight Agent for grounded question-guided analysis; and (3) Follow-Up Question Composer generates deeper follow-up questions for progressive multi-step medical reasoning.

Baselines	Insights Recall		Insights Precision		Insights $F_{1}$		Insights Novelty
Baselines	ROUGE-1	G-Eval	ROUGE-1	G-Eval	ROUGE-1	G-Eval	Original	Innovation
Direct Decoding (LMMs-Only)
GPT-4o	0.1799	0.2978	0.2091	0.3577	0.1934	0.3250	0.1293	0.2094
GPT-5	0.1866	0.3052	0.1851	0.3649	0.1858	0.3324	0.1322	0.2131
Deepseek-VL2	0.1816	0.3226	0.2274	0.4070	0.2019	0.3599	0.1959	0.2712
InternVL3-38B	0.1774	0.3390	0.2002	0.3993	0.1881	0.3667	0.1607	0.2553

Table 2: Performance comparison of baseline models on the MedInsightBench across insight extraction metrics.

Methods	Insights Recall	Insights Precision	Insights $F_{1}$	Insights Novelty
Direct Decoding (Qwen3-VL-32B-Instruct)	0.4295	0.5372	0.4774	0.4488
MedInsightAgent (Qwen3-VL-32B-Instruct)	0.4703	0.5921	0.5242	0.4995
w/o Image-Summarization Module	0.4615 $\downarrow$ 0.0088	0.5764 $\downarrow$ 0.0157	0.5126 $\downarrow$ 0.0116	0.4780 $\downarrow$ 0.0215
w/o Web-Retrieval Module	0.4623 $\downarrow$ 0.0080	0.5739 $\downarrow$ 0.0182	0.5121 $\downarrow$ 0.0121	0.4705 $\downarrow$ 0.0290
w/o Image-Analysis Tool	0.4537 $\downarrow$ 0.0166	0.5604 $\downarrow$ 0.0317	0.5014 $\downarrow$ 0.0228	0.4842 $\downarrow$ 0.0153
w/o Follow-Up Question Composer	0.4391 $\downarrow$ 0.0312	0.5511 $\downarrow$ 0.0410	0.4888 $\downarrow$ 0.0354	0.4653 $\downarrow$ 0.0342

Table 3: Effect of each method and module within the MedInsightAgent framework. The blue arrows and values indicate the performance drop compared to the full MedInsightAgent(Qwen3-VL-32B-Instruct).

Figure 5: Comparison of novelty scores between model-generated and human-assessed insights across baselines.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The distribution of different Insight Types in MedInsightBench and the average token count of Question and Insight in each type of data.

Figure 7: The distribution of different difficulty levels in MedInsightBench.

Figure 8: Comparison of G-Eval scores in Insight Recall, Insight

F1

, and Expert Scores in Human Evaluation.

Comparison	G-Eval $F1$ (Mean $\pm$ 95% CI)	$p$ -value
MedInsightAgent	$0.5242\pm 0.0090$	–
ReAct + same tools	$0.5067\pm 0.0096$	$0.010$
Reflexion	$0.5063\pm 0.0097$	$0.011$
CoT	$0.4826\pm 0.0105$	$<0.001$
Qwen3-VL-32B	$0.4774\pm 0.0109$	$<0.001$
GPT-4o	$0.3250\pm 0.0138$	$<0.001$

Appendix

Table 4: Bootstrap significance test on the primary results (

B=10{,}000

). G-Eval

F1

is reported as mean

\pm

95% confidence interval.

p

-values are computed against MedInsightAgent.

Configuration	ISM	WRM	IAT	FQC	G-Eval $F1$	Novelty	$\boldsymbol{\Delta}$ $F1$
MedInsightAgent	✓	✓	✓	✓	0.5242	0.4995	–
w/o ISM & WRM	$\times$	$\times$	✓	✓	0.4996	0.4607	$-0.0246$
w/o ISM & WRM & IAT	$\times$	$\times$	$\times$	✓	0.4889	0.4524	$-0.0353$

Appendix

Table 5: Joint ablation results for analyzing interactions among the specialized tools in MedInsightAgent.

Ground-Truth Insight	Output of GPT-4o	Output of MedInsightAgent(GPT-4o)
Case 139 : TCGA-EJ-A7NM with Difficulty Level 2
Lymphovascular invasion and extensive perineural invasion suggest increased metastatic potential; consider systemic therapy evaluation.	Absence of perineural invasion in the visible sections may impact staging.	Perineural invasion suggests a more aggressive tumor , which might increase the likelihood of cancer recurrence and affect treatment decisions.
Multifocal extracapsular extension and seminal vesicle invasion indicate locally advanced disease with higher recurrence risk; discuss adjuvant therapy options.	Stromal desmoplasia is present, indicating possible tumor-host interaction and invasive behavior.	A strong stromal reaction, known as desmoplasia, often indicates a more aggressive tumor behavior and higher grade in prostate cancer, potentially impacting treatment strategies and prognosis.
Case 122 : TCGA-IN-A7NT with Difficulty Level 3
Angiolymphatic invasion is present.	No definitive lymphovascular invasion observed, although detailed review and further sections may be warranted.	The presence of lymphovascular invasion can suggest a higher risk of metastasis, informing treatment decisions and prognosis.
Poorly differentiated (G3) adenocarcinoma suggests more aggressive tumor biology; consider comprehensive treatment approach.	There are poorly formed glands and irregular cell clustering, which may indicate a higher histologic grade.	The presence of pronounced cellular atypia and high mitotic rate suggests a more aggressive tumor behavior , potentially correlating with a higher risk of nodal metastasis and impacting therapeutic decisions.

Appendix

Table 6: Case study of insight discovery. We selected three cases across different difficulty levels, with the bolded statements highlighting instances where MedInsightAgent demonstrated superior performance and the underlined parts show the defects in the output of GPT-4o.

Model	Reviewed	Category A	Category B	Category C	Human Approval Rate
MedInsightAgent	30	13 (43.3%)	13 (43.3%)	4 (13.4%)	26 (86.6%)
Qwen3-VL-32B	30	8 (26.7%)	13 (43.3%)	9 (30.0%)	21 (70.0%)
Reflexion	30	9 (30.0%)	13 (43.3%)	8 (26.7%)	22 (73.3%)

Appendix

Table 7: Supplementary human audit of insights that passed the Novelty filter. Category A denotes true clinical novelty, Category B denotes clinically relevant but already known information, and Category C denotes trivial or clinically uninformative statements. The human approval rate is the proportion of insights categorized as A or B.

Method	API Calls / Sample	Avg. Input Tokens	Avg. Output Tokens	Wall-clock (s/Sample)	G-Eval $F1$
Direct Decoding	1	2,352	521	15.8	0.4774
CoT	1	2,685	769	27.6	0.4826
ReAct	12.4	10,907	2,854	133.4	0.5067
MedInsightAgent	32	65,372	7,692	958.9	0.5242

Appendix

Table 8: Inference cost and performance comparison across different methods. All values are averaged per sample.

Goal	Correlate histopathologic features of the tongue carcinoma with staging parameters, margin status, nodal metastasis, and HPV status to guide prognostic assessment and treatment planning.
Pathological Image

Appendix

Table 9: Goal and pathological image of Case 4 in MedInsightBench.

Insights Details

Question 1 : What is the diagnosis, location, size, and depth of invasion as documented in the pathology report for the excised tongue specimen? Insight 1 : Invasive keratinizing squamous cell carcinoma identified in the right lateral tongue dorsum and lateral tongue, approximately 2.0 cm in greatest dimension with muscle invasion. Type 1 : Diagnostic

Question 2 : Does the pathology report confirm the presence of metastatic carcinoma in a lymph node from the right neck level 2, and what is the greatest dimension measurement of the identified tumor deposit? Insight 2 : Metastatic carcinoma identified in one lymph node from right neck level 2 with a tumor deposit measuring 1.9 cm. Type 2 : Diagnostic

Question 3 : What is the status of all surgical margins regarding tumor presence in the final pathology report? Insight 3 : Final surgical margins are negative for tumor in all specimens. Type 3 : Descriptive

Question 4 : What are the specific pathologic stage descriptors for the primary tumor, regional lymph nodes, and the number of lymph nodes examined versus involved as documented in the report? Insight 4 : Pathologic staging is pT1 pN1 with 44 lymph nodes examined and 1 involved. Type 4 : Descriptive

Question 5 : What are the reported findings regarding histologic grade, extracapsular extension, perineural invasion, and bony/cartilage invasion? Insight 5 : Tumor is moderately-differentiated squamous cell carcinoma without extracapsular extension, perineural invasion, or bony/cartilage invasion. Type 5 : Descriptive

Question 6 : What were the results of HPV testing for both p16 immunohistochemistry and high risk HPV in situ hybridization? Insight 6 : HPV testing performed shows p16 negative by immunohistochemistry and high risk HPV negative by in situ hybridization. Type 6 : Descriptive

Appendix

Table 10: Insight details of Case 4 in MedInsightBench (1).

Insights Details

Question 7 : Does the pathology report indicate both the number of lymph nodes involved by metastatic carcinoma and the total number examined, along with the presence or absence of extracapsular extension? Insight 7 : Single lymph node metastasis (1/44) without extracapsular extension suggests intermediate recurrence risk; consider adjuvant therapy based on multidisciplinary discussion. Type 7 : Predictive

Question 8 : Does the pathology report confirm that all intraoperative frozen section consultations for mucosal margins were benign, indicating adequate surgical clearance? Insight 8 : All intraoperative frozen section consultations for mucosal margins were benign, confirming adequate surgical clearance. Type 8 : Evaluative

Question 9 : What are the specific benign findings and lymph node levels documented as negative for tumor in the report? Insight 9 : Additional benign findings include minor salivary gland tissue and multiple lymph node levels negative for tumor. Type 9 : Descriptive

Question 10 : What is the HPV status of the oral cavity squamous cell carcinoma as determined by testing documented in the report? Insight 10 : HPV-negative status in an oral cavity squamous cell carcinoma suggests non-HPV driven etiology; consider additional molecular profiling for treatment guidance. Type 10 : Exploratory

Appendix

Table 11: Insight details of Case 4 in MedInsightBench (2).

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 106 papers

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models
Jul 24, 2026Bannapol Limanond, Masanori Suganuma, Takayuki OkataniMedical VLMs Multimodal Model Evaluation
MedUAG: Unified Understanding and Generation for Medical Multimodal Models
Aug 19, 2026Zijie Meng, Yuncheng Zhang, Hualiang Wang +8Unified Multimodal Models Multimodal Generation
Zhejiang University · Hong Kong University of Science and Technology · Tsinghua University +1

Jun 8, 2025J/K move · Enter open · S save

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied+14

LASA Team · DAMO Academy, Alibaba Group

Medical Report Generation Multimodal Pretraining Multimodal QA Multimodal Reasoning Medical VLMs Multimodal Foundation Models+2

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

Abstract

Data Analysis Agents

Explore similar work

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models

MedUAG: Unified Understanding and Generation for Medical Multimodal Models