TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis

Sijing Li; Zhongwei Qiu; Jiang Liu; Wenqiao Zhang; Tianwei Lin; Yihan Xie; Jianxiang An; Boxiang Yun

cs.CVMar 6, 2026

TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis

Authors: Sijing Li, Zhongwei Qiu, Jiang Liu, Wenqiao Zhang, Tianwei Lin, Yihan Xie, Jianxiang An, Boxiang Yun,

Organizations: Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab · Shanghai Institution of Pancreatic Disease ·

Abstract

Accurate tumor analysis is central to clinical radiology and precision oncology, where early detection, reliable lesion characterization, and pathology-level risk assessment guide diagnosis and treatment planning. Chain-of-Thought (CoT) reasoning is particularly important in this setting because it enables step-by-step interpretation from imaging findings to clinical impressions and pathology conclusions, improving traceability and reducing diagnostic errors. Here, we target the clinical tumor analysis task and build a large-scale benchmark that operationalizes a multimodal reasoning pipeline, spanning findings, impressions, and pathology predictions. We curate TumorCoT, a large-scale dataset of 1.5M CoT-labeled VQA instructions paired with 3D CT scans, with step-aligned rationales and cross-modal alignments along the trajectory from findings to impression to pathology, enabling evaluation of both answer accuracy and reasoning consistency. We further propose TumorChain, a multimodal interleaved reasoning framework that tightly couples 3D imaging encoders, clinical text understanding, and organ-level vision-language alignment. Through cross-modal alignment and iterative interleaved causal reasoning, TumorChain grounds visual evidence, aggregates conclusions, and issues pathology predictions after multiple rounds of self-refinement, improving traceability and reducing hallucination risk. Experiments show consistent improvements over strong baselines in lesion detection, impression generation, and pathology classification, and demonstrate strong generalization on the DeepTumorVQA benchmark. These results highlight the potential of multimodal reasoning for reliable and interpretable tumor analysis in clinical practice. Detailed information about our project can be found on our project homepage at https://github.com/ZJU4HealthCare/TumorChain.

Figures & tables

Figure 1: (a) TumorChain focuses on tumor analysis of the five major organs in the digestive system. (b) We establish a corresponding medical reasoning logic chain from radiology findings to radiology impressions to pathological conclusions. (c) The system tackles four diverse tasks across 1.5 million multimodal VQA instructions, enabling advanced clinical reasoning.

Figure 3: The overview of the interactive-validated CoT Data Engine, which is a system of multi-agent collaboration, including the procedure of raw data processing, diagnostic knowledge graph importing, and CoT data building and validation.

Figure 4: The overview of TumorChain . Given CT volumes and task prompts, TumorChain analyzes the task requirements and performs iterative interleaved reasoning by integrating a segmentation expert model for ROI localization, auxiliary classification for local feature enhancement, and organ-guided iterative reasoning for clinical tumor analysis.

Commercial Models
Model	Position		Lesion Attributes						TNM Prediction			CoT- Report	Avg.
Model	Organ Pos.	Tumor Pos.	Seg. Loc.	Shape	Boundary	Density	Count	Others	Tumor	Node	Met.
Claude3-Haiku	32.61	33.05	52.38	58.51	50.00	50.74	37.51	50.95	59.19	56.59	43.40			33.21	46.51
Gemini2.0-Flash	43.27	17.69	46.81	35.97	38.22	39.11	40.22	52.86	34.03	42.21	49.06	56.08	41.29
GPT-5-Mini	44.18	37.73	44.96	47.72	62.58	57.13	64.85	66.67	34.25	47.40	49.60	62.02	51.59
Generalist Models

Table 1: Comparison of TumorChain with other LVLMs on TumorCoT benchmark. Bold and underlined text indicates the best and second-best performance, respectively.

Commercial Models
Reasoning Models	$S_{FC}$	$S_{IC}$	$S_{LRC}$	$CoT_{e}$
Claude3-Haiku	59.31	43.66	30.80	43.21
Gemini2.0-Flash	63.10	57.69	45.12	54.28
GPT-5-Mini	64.22	66.42	55.09	61.23
Open-Source Models
Qwen2.5-VL-7B	49.15	22.89	12.09	26.45

Table 2: TumorChain-Eval with other reasoning LVLMs on TumorCoT .

Figure 5: (a) Ablation on classification loss weight $\alpha$ . (b) Model performance on five organs of TumorCoT . (c) Workflow of reasoning chains extraction and evaluation for task 4.

Figure 6: Baseline model improvements by finetuning with TumorCoT-1.5M.

Model	Recog.	V.R.	M.R.	Avg.
Commercial Models
Claude3-Haiku	38.70	31.53	34.81	35.01
Gemini2.0-Flash	50.88	33.81	35.06	39.92
GPT-5-mini	49.29	35.57	29.87	38.24
Open-Source Models
Qwen2.5-VL-7B	45.31	36.94	37.04	39.76

Table 3: Results on DeepTumorVQA.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{F}(\cdot)$	Represents the TumorChain framework.
$\mathcal{{E}}_{v}(\cdot)$	3D vision encoder
$\mathcal{S}eg(\cdot)$	Organ segmentation expert
$\mathcal{C}ls(\cdot)$	Auxiliary classification model
$\mathcal{P}(\cdot)$	Multi-layer perceptron (MLP) projector
$\mathcal{LLM}(\cdot)$	Backbone of LLM

Appendix

Table 4: Notation in the TumorChain Pipeline.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$3\times 10^{-5}$
Learning rate scheduler	cosine decay
Weight decay	0.0
Warmup ratio	0.03
Mixed precision	bf16 (fp16 disabled)

Appendix

Table 5: Hyperparameter Configuration

Figure 7: Semantic Consistency Evaluation Prompt.

Reasoning Models	Liver	Pancreas	Colon	Stomach	Esophagus	Average
Claude3-Haiku	34.70	56.13	44.08	17.11	39.77	38.36
Gemini2.0-Flash	42.35	29.03	56.24	71.15	38.64	47.48
GPT-5-Mini	44.48	35.71	59.76	76.85	65.12	56.38
Qwen2.5-VL-7B	47.94	58.15	54.60	69.79	52.41	56.58
InternVL-2.5-8B	48.89	49.20	48.89	55.84	52.54	51.07
MiniCPM-V-4.5-9B	46.07	43.29	45.69	50.26	47.59	46.58

Appendix

Table 6: CoT-Report comparison of TumorChain with other Reasoning VLMs on TumorCoT .

Model	Liver	Pancreas	Colon	Average
Generalist Models
Qwen2.5-VL-7B	36.32	32.39	46.33	38.35
InternVL-2.5-8B	34.52	35.86	52.75	41.04
MiniCPM-V4.5-9B	40.02	45.20	52.93	46.05
Commercial Models
Claude3-Haiku	32.67	32.62	18.40	27.90

Appendix

Table 7: Comparison of TumorChain with other VLMs on the public DeepTumorVQA benchmark.

$\alpha$	Position		Lesion Attributes						TNM Prediction			CoT- Report	Avg.
$\alpha$	Organ Pos.	Tumor Pos.	Seg. Loc.	Shape	Boundary	Density	Count	Others	Tumor	Node	Met.	CoT- Report	Avg.
0.0	99.92	97.01	82.46	81.45	75.82	83.01	80.89	82.35	86.11	57.88	65.22	77.41	80.79
0.1	99.90	95.58	84.01	79.78	75.09	82.10	83.85	81.85	88.69	58.16	69.54	78.29	81.40
0.5	99.90	97.14	83.14	81.60	81.15	83.51	84.01	82.45	86.43	60.85	70.65	79.54	82.53
1.0	99.97	97.57	86.88	82.28	84.52	85.05	86.20	86.57	88.83	61.63	71.07	82.36	84.41
1.5	99.94	97.91	83.21	79.40	74.46	75.37	81.83	77.41	86.23	57.51	57.46	79.21	79.16

Appendix

Table 8: Ablation study of the effect of classfication loss weight

\alpha

in TumorChain.

Avg. Score (Before finetuning / After finetuning)
Model	Position	Lesion Attributes	TNM Prediction	CoT Report	Avg.
M3D-Phi-3-4B	30.36 / 86.92	33.23 / 61.78	37.16 / 59.48	34.79 / 61.18	32.84 / 66.98
Lingshu-7B	28.93 / 85.46	49.91 / 77.36	43.21 / 67.65	54.30 / 65.98	45.10 / 74.11
Qwen2.5-VL-7B	31.05 / 91.02	47.17 / 76.45	40.94 / 62.37	60.54 / 74.21	44.04 / 76.01
TumorChain-7B	- / 98.77	- / 85.25	- / 73.84	- / 82.36	- / 84.41

Appendix

Table 9: Ablation study of baselines before and after finetuning on TumorCoT.

Figure 8: The overview of the diagnostic knowledge graph of 5 organs by doctors.

Figure 9: Prompt example: Traceable CoT Reasoner for extracting lesion attributes.

New Label	Segmentator Name	Included Organs	TotalSeg. Label
0	background	background	0
1	spleen	spleen	1
2	kidney_right	kidney_right	2
3	kidney_left	kidney_left	3
4	gallbladder	gallbladder	4
5	liver	liver	5

Appendix

Table 10: Mapping Table of TotalSegmentator Organ Indexes After Aggregation.

Prompt for Medical Fact Triple Extraction

You are a medical fact–extraction assistant specialized in parsing knowledge from the given clinical text. Your task is to analyze the input and extract all factual triples

(\textit{subject},\textit{relation},\textit{object})

, where each triple expresses a single clear medical fact. Ensure that all information is presented strictly in triple form, focusing on organs/structures , relations , and lesions . Input Example: First, examine the right hemicolon, cecum, ascending colon, and hepatic flexure; no wall thickening or abnormal density is observed, indicating no definite mass or inflammatory change in the right hemicolon. Second, check the left hemicolon, transverse colon, and splenic flexure, descending colon, sigmoid colon, and rectum; again no abnormal thickening or abnormal density is found, supporting the absence of significant lesions in the left hemicolon. Third, evaluate colon-related internal structures and surrounding tissues; no pericolonic vascular abnormality is detected, and the liver, gallbladder, pancreas, spleen, kidneys, and pelvic structures show no colon-related significant lesion, enlarged lymph nodes, or free fluid. These findings collectively do not support tumor or active inflammatory changes. Overall, the imaging features do not suggest colorectal cancer or other qualifying pathology. Output Requirements: In each triple, the first and third elements must be an organ or lesion, and the second element must describe their relationship. If the second and third positions are reversed, correct them. The output format should be:

(\text{Right hemicolon},\text{not observed},\text{wall thickening});

Appendix

Table 11: Prompts for medical triples extraction in stage 1 of CoT evaluation

Chain Level	Scoring Dimension	Description	Scoring Criteria
Finding Chain	Existence Match	Degree to which predicted facts match the ground-truth (GT) facts.	10 : Prediction contains all key GT facts with no omission or redundancy.
			8–9 : Prediction covers the vast majority of GT facts, with only minor omissions or redundancies (¡10%).
			6–7 : Prediction covers some GT facts but exhibits moderate omissions or redundancies (10–30%).
			4–5 : Prediction covers only a small portion of GT facts; substantial omissions or redundancies (30–50%).
			1–3 : Low match rate; only a few facts are correct and the error rate is very high.
			0 : Prediction includes none of the GT facts.

Appendix

Table 12: Finding Chain (FC) scoring criteria.

Chain Level	Scoring Dimension	Description	Scoring Criteria
Impression Chain	Clarity	Medical clarity of the impression statement.	10 : Impression is completely clear, logically coherent, and unambiguous.
			8–9 : Description is clear with only minor incomplete expressions or slight ambiguity ( $<$ 10%).
			6–7 : Basically clear but contains some ambiguities that moderately hinder understanding.
			4–5 : Blurry description with numerous ambiguities significantly affecting medical interpretation.
			1–3 : Very difficult to understand; almost unusable due to severe ambiguity.
			0 : Impression chain is empty or entirely invalid and unclear.

Appendix

Table 13: Impression Chain (IC) scoring criteria.

Chain Level	Scoring Dimension	Description	Scoring Criteria
Long Reasoning Chain	Logical Completeness	Logical closure and completeness of higher-order reasoning.	10 : The reasoning chain perfectly covers all key points, with no logical gaps or omissions.
			8–9 : Reasoning is largely complete, with only minor ( $<$ 10%) logical gaps or omitted details.
			6–7 : Some logical interruptions or omissions exist, but most reasoning paths remain valid (10–30%).
			4–5 : Significant logical gaps, missing many key points, and notable interruptions in reasoning (30–50%).
			1–3 : Most of the reasoning chain is invalid; significant logical flaws, key derivations incomplete.
			0 : No reasoning process or the reasoning chain completely fails.

Appendix

Table 14: Long Reasoning Chain (LRC) scoring criteria.

Figure 10: A case of CoT report generation.

Radiology VLMs

Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.

See all 110 papers

E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis
Jun 22, 2026Sijing Li, Zhongwei Qiu, Zhuoya Wang +6Medical Image Grounding VLM Hallucination Mitigation
Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab +3
RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology
May 11, 2026Wenxuan Li, Pedro R. A. S. Bassi, Xinze Zhou +3Healthcare Medical VQA
Department of Computer Science, Johns Hopkins University · Clinic of Radiology and Nuclear Medicine, University Hospital Basel · Department of Oncology, Johns Hopkins School of Medicine
Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering
Mar 14, 2026Lin Fan, Yafei Ou, Zhipeng Deng +8Medical VQA Visual CoT
Southwest Jiaotong University · RIKEN · Hokkaido University +4

Jun 22, 2026J/K move · Enter open · S save

E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis

Sijing Li, Zhongwei Qiu, Zhuoya Wang, Boxiang Yun+5

Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab+3

Medical Image Grounding VLM Hallucination Mitigation RL for VLMs

Open paper AskPDF arXiv

While Vision-Language Models (VLMs) show great promise in volumetric medical report generation, they frequently suffer from visual hallucinations and a lack of grounding in 3D CT data. Current Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) strategies typically optimize text fidelity alone, essentially rewarding correct diagnoses derived from language priors rather than genuine visual perception. To address this, we propose cross-view aligned Evidence-driven Multimodal Reinforcement Learning (Evidence-MRL, noted as E-MRL), a reliable RL reasoning framework that formulates the generation process as a Markov Decision Process of "diagnosis-localization-verification". Unlike standard approaches, our model is explicitly trained to identify a "key evidence slice" alongside the global diagnostic report, grounding its findings in verifiable visual evidence. Crucially, we introduce a novel cross-view consistency reward, which validates the semantic alignment between the golden-standard report and a local visual re-query of the selected key slice, providing additional rewards for correctly-localized reasoning. Experiments on large-scale 3D CT tumor datasets demonstrate that E-MRL significantly reduces hallucinations and improves diagnostic accuracy compared to SFT and RL baselines, offering a clinically interpretable solution for visually-grounded and tumor analysis.

Abstract

Radiology VLMs

Explore similar work

E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis

RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology

Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering