Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering

Lin Fan; Yafei Ou; Zhipeng Deng; Pengyu Dai; Hou Chongxian; Jiale Yan; Yaqian Li; Kaiwen Long

cs.CVMar 14, 2026

Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering

Authors: Lin Fan, Yafei Ou, Zhipeng Deng, Pengyu Dai, Hou Chongxian, Jiale Yan, Yaqian Li, Kaiwen Long,

Organizations: Southwest Jiaotong University · RIKEN · Hokkaido University · Westlake University ·

Abstract

Chain-of-thought (CoT) reasoning has advanced medical visual question answering (VQA), yet most existing CoT rationales are free-form and fail to capture the structured reasoning process clinicians actually follow. This work asks: Can traceable, multi-step reasoning supervision improve reasoning accuracy and the interpretability of Medical VQA? To this end, we introduce Step-CoT, a large-scale medical reasoning dataset with expert-curated, structured multi-step CoT aligned to clinical diagnostic workflows, implicitly grounding the model's reasoning in radiographic evidence. Step-CoT comprises more than 10K real clinical cases and 70K VQA pairs organized around diagnostic workflows, providing supervised intermediate steps that guide models to follow valid reasoning trajectories. To effectively learn from Step-CoT, we further introduce a teacher-student framework with a dynamic graph-structured focusing mechanism that prioritizes diagnostically informative steps while filtering out less relevant contexts. Our experiments show that using Step-CoT can improve reasoning accuracy and interpretability. Benchmark: github.com/hahaha111111/Step-CoT. Dataset Card: huggingface.co/datasets/fl-15o/Step-CoT

Figures & tables

Dataset	Instances	CoT Construction	Expert Involvement	Clinical Workflow
PMC-VQA [ 47 ]	227K	$\times$ No CoT	$\checkmark$ Yes	$\times$ No
Med-GRIT-270k [ 19 ]	270k QA pairs	$\checkmark$ GPT-generated	$\times$ No experts	$\times$ No
MedCoT [ 26 ]	50k Extends Med-VQA	$\checkmark$ Expert validation	$\checkmark$ Yes	$\checkmark$ Partial
ReasonMed [ 3 ]	370k reasoning samples	$\checkmark$ Agent-generated	$\checkmark$ Multi-agent validation	$\times$ No
HVCR [ 11 ]	31k QA pairs	Expert validation	$\checkmark$ Yes	$\times$ No
V2T-CoT [ 41 ]	39k examples	$\checkmark$ GPT-generated CoTs	$\times$ No experts	$\times$ No

Table 1: Comparison of recent medical reasoning datasets with Step-CoT.

Figure 1: Overview of the Step-CoT dataset. (A) Conventional Med-VQA approaches, where models take an image and a question as input, perform multi-modal feature fusion and output a diagnostic answer. Although leveraging multi-modal knowledge, this paradigm lacks interpretability and often yields limited diagnostic accuracy. (B) Enhances interpretability by integrating large language models with CoT reasoning to generate intermediate explanations; however, such reasoning is often unreliable. (C) Our proposed Step-CoT dataset and training framework, which introduces explicit intermediate supervision. By guiding the model to learn structured clinical reasoning steps, Step-CoT not only improves interpretability through trustworthy intermediate reasoning but also enhances diagnostic accuracy.

Component	Description and Implementation
Clinical Questions	Sequential questions derived from radiological diagnostic pathways, progressing from abnormality detection to final diagnosis.
Answer Logic	• Direct answers from explicit report content. • “N/A” when the question is clinically irrelevant. • “No Answer” when information is missing, with inference attempted from available evidence.
Reasoning Chains	Each reasoning step required explicit justifications derived directly from report content or clinically valid inferences. This requirement ensures that model outputs remain grounded in available clinical evidence while maintaining the structured analytical rigor characteristic of expert radiological interpretation.

Table 2: Components of individual case analysis

Figure 2: Distribution and statistics for the data sources, disease prevalence, answer distributions, and reasoning lengths in the Step-CoT dataset. (A) The inner ring illustrates the proportional distribution across different datasets, while the outer ring represents the distribution of various disease categories within the datasets. (B) This confusion matrix, organized by disease categories and reasoning steps, visualizes the average reasoning chain length. Each cell contains a pie chart representing the statistical distribution of samples across different chain lengths, while the marginal histograms on the axes display the sample count distributions by chain length for individual steps (x-axis) and disease categories (y-axis). (C) This diagram presents the outcome transition statistics between consecutive reasoning steps, mapping the flow of diagnostic conclusions throughout the clinical reasoning pathway. The name of each annotation (e.g., A1) can be referred to in the dataset description section of Appendix Sec. B.

LVLMs
Model	Accuracy	mAUC	Sensitivity	Specificity
LLaVA-Med [ 20 ]	42.7	58.3	42.7	79.4
Med-Flamingo [ 28 ]	30.1	61.2	28.4	89.8
Visual foundation models
VisualBERT [ 24 ]	56.2(+9.3)	48.8(+14.3)	8.5(+1.6)	89.1(+2.7)
CLIP [ 31 ]	64.7(+4.5)	48.8(+3.8)	10.1(+1.9)	87.5(+2.1)

Table 3: Test results of diagnosis step on different models using Step-CoT(%). Entries are reported as

a({+}b)

, where

a

is the performance without Step-CoT and

b

is the improvement with it. The best results in each column are highlighted in bold , and the second-best values are underlined .

Figure 3: Fine-tuning an LVLM with Step-CoT-based intermediate constraints under verifiable instructions. The testing of different steps is conducted in independent dialogue sessions. The model progressively adjusts its stepwise reasoning, produces coherent intermediate steps, and converges to the correct final diagnosis; this demonstrates that Step-CoT’s structured intermediate constraints strengthen model reasoning and reliably guide it to accurate conclusions.

Model	Accuracy	mAUC	Sensitivity	Specificity
VisualBERT [ 24 ]	25.7(+6.3)	52.1(+4.3)	30.8(+4.7)	73.9(+3.9)
CLIP [ 31 ]	26.8(+7.3)	53.3(+4.6)	18.4(+3.3)	79.7(+3.7)
ALBEF [ 23 ]	30.4(+5.2)	58.3(+3.1)	38.4(+3.5)	81.5(+2.9)
BLIP [ 22 ]	27.5(+6.1)	52.7(+3.7)	17.9(+2.9)	78.2(+3.5)
FLAVA [ 35 ]	28.9(+5.5)	56.8(+2.9)	35.5(+4.1)	77.3(+3.0)
biomedclip [ 46 ]	33.6(+5.8)	61.9(+3.6)	42.7(+3.9)	85.6(+2.6)

Table 4: Transfer performance comparison with and without using Step-CoT on a different test dataset (%). Entries are reported as

a({+}b)

, where

a

is the performance without Step-CoT and

b

is the improvement with it. The best value in each column is bold and the second best is underlined .

	Detection	Distribution	Location	Diagnosis
Clinician $\dagger$	72.1	66.0	66.0	73.1
Teacher $\dagger$	88.5	78.4	72.8	79.8
Student $\dagger$	80.4	72.6	69.5	68.5

Table 5: Comparison of clinical expert evaluation across four of seven reasoning steps (%). The best results in each column are highlighted in bold , and the second-best values are underlined .

	Detection	Distribution	Location	Diagnosis
w/o Memory	73.7	69.6	63.2	65.5
w/o Text	81.5	76.1	69.3	72.1
Ours (Teacher)	91.8	84.6	77.1	78.3
Ours (Student)	91.8	83.4	76.9	77.5

Table 6: Comparison of memory ablation results across four of seven reasoning steps (%). The best results in each column are highlighted in bold , and the second-best values are underlined .

Figure 4: The feature attention visualization across multi-step reasoning demonstrates an evolution from broad attention in the initial query steps to highly targeted attention in the final diagnostic step, reflecting the multi-step capability of Step-CoT and visually verifying the effectiveness of the reasoning chain.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: This study collected a total of 16,782 CXR samples in PNG format from three datasets, containing 3,999, 8,788, and 3,995 samples, respectively. After filtering, 10,068 samples were retained, yielding 10,068*7 QA pairs for training the stepwise Med-VQA task.

Figure 6: Preprocessing pipeline of the Step-CoT dataset.

Category	Absolute Counts			Relative Distribution (%)
Category	Train	Validation	Test	Train	Validation	Test
Diagnosis Categories
Normal	4,750	1,018	1,019	70.0	15.0	15.0
Nodule	595	127	129	69.9	14.9	15.2
Atelectasis	511	109	111	69.9	14.9	15.2
Infiltration	506	108	109	69.9	14.9	15.2

Appendix

Table 7: Dataset distribution analysis for Step-CoT.

Model	Step	Accuracy	AUC	Sensitivity	Specificity	F1-Score	Precision
VisualBERT	Detection step	75.0	71.0	13.5	93.5	33.1	34.0
	Lesion distribution step	70.6	58.1	13.0	93.0	30.0	28.1
	Radiographic pattern step	74.1	62.0	14.6	93.2	20.1	17.5
	Anatomical location step	68.0	60.0	13.0	92.8	17.5	15.0
	Morphologic feature step	84.0	58.5	13.5	93.1	33.0	32.1
	Secondary effects/associated signs step	66.1	61.0	12.8	92.5	16.0	14.5

Appendix

Table 8: Stepwise benchmark results: Vision foundation models (%). The best value in each column is bold and the second best is underlined .

Model	Step	Accuracy	AUC	Sensitivity	Specificity	F1-Score	Precision
Teacher	Detection step	91.8	88.3	63.1	94.6	61.6	60.3
	Lesion distribution step	84.6	89.7	51.4	93.8	49.8	61.7
	Radiographic pattern step	84.8	90.2	45.0	96.5	43.6	53.2
	Anatomical location step	77.1	89.5	45.7	95.3	44.9	57.7
	Morphologic feature step	89.4	95.1	46.1	94.1	40.5	39.2
	Secondary effects/associated signs step	79.7	86.4	41.4	95.3	40.0	41.4

Appendix

Table 9: Comprehensive performance comparison (per-step metrics) between Teacher, Student, and ablation variants (%). The best value in each column is bold and the second best is underlined .

Model	Step	Accuracy	Sensitivity	Specificity	F1-Score
Clinician	Detection step	72.1	36.8	82.7	37.9
	Lesion distribution step	66.0	28.0	85.3	27.1
	Radiographic pattern step	69.1	25.1	88.7	27.1
	Anatomical location step	66.0	34.0	91.1	24.7
	Morphologic feature step	66.1	21.3	85.3	19.0
	Secondary effects/associated signs step	91.0	33.4	93.7	25.0

Appendix

Table 10: Clinical expert evaluation (%), per-step comparison between Clinicians, Teacher, and Student (N=200). The best value in each column is bold .

Model	Params	Inf_sample	Inf_batch	Memory_peak
CLIP	151.81 M	1.55 ms	6.20 ms	599.43 M
FLAVA	242.54 M	4.39 ms	17.57 ms	971.42 M
BLIP	225.25 M	1.46 ms	5.86 ms	904.27 M
VisualBERT	111.98 M	0.03 ms	0.12 ms	438.22 M
Teacher	283.66 M	22.81 ms	91.23 ms	1219.06 M
Student	151.56 M	1.51 ms	6.02 ms	1219.72 M

Appendix

Table 11: Computational Requirements Comparison (batchsize=4). Note: Bold = best (lowest); underline = 2nd best (2nd lowest).

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

Improving Medical VQA through Trajectory-Aware Process Supervision
Apr 10, 2026Halil Ibrahim Gulluk, Olivier GevaertMedical VQA Medical VLMs
Electrical Engineering, Stanford University · Stanford Center For Biomedical Informatics Research
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Mar 2, 2026Yuan Wu, Zongxian Yang, Jiayu Qian +5Medical VLMs VLM Reasoning
Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University

Apr 10, 2026J/K move · Enter open · S save

Improving Medical VQA through Trajectory-Aware Process Supervision

Halil Ibrahim Gulluk, Olivier Gevaert

Electrical Engineering, Stanford University · Stanford Center For Biomedical Informatics Research

Medical VQA Medical VLMs VLM Reasoning Process Supervision RL for VLMs

Open paper AskPDF arXiv

Reasoning capabilities are crucial for reliable medical visual question answering (VQA); however, existing datasets rarely include reasoning explanations. We address this by generating reasoning trajectories for six medical VQA benchmarks using the COMCTS algorithm with open-source vision-language models, with an LLM serving as the verification judge. Building on these generated datasets, we propose a two-stage training framework: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO) with a novel process-based reward. While standard approaches rely solely on exact-match rewards for final answers, we introduce a trajectory-aware reward that measures the similarity between generated and ground-truth reasoning processes. Specifically, we embed reasoning steps using sentence transformers and compute the Dynamic Time Warping (DTW) distance between the resulting vector sequences. Experiments across six benchmarks demonstrate that combining the DTW-based process reward with exact-match reward consistently outperforms SFT-only training, raising mean accuracy from 0.598 to 0.689, mean BERTScore from 0.845 to 0.881, and mean ROUGE-L from 0.665 to 0.748. Our results highlight the importance of process supervision in training reasoning-capable medical VLMs. We make our code and generated reasoning datasets publicly available at https://anonymous.4open.science/r/MICCAI-R1-MED-VQA-code-B14B/

Abstract

Medical VQA

Explore similar work

Improving Medical VQA through Trajectory-Aware Process Supervision

LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering

Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine