InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

Guanghao Zhu; Zeyu Liu; Zhitian Hou; Pengkai Wang; Zhijie Sang; Shuo Cai; Yang Yu; Yuanyi Wang

cs.CLSep 28, 2026

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

Authors: Guanghao Zhu, Zeyu Liu, Zhitian Hou, Pengkai Wang, Zhijie Sang, Shuo Cai, Yang Yu, Yuanyi Wang, +4 more

Organizations: The Hong Kong Polytechnic University · PolyU-Daya Bay Technology and Innovation Research Institute

Abstract

Recent medical multimodal models have benefited from larger corpora, broader modality coverage, and stronger reasoning-oriented training, yet effective data design across continued pretraining (CPT) and post-training remains challenging. Medical sources vary substantially in structure, granularity, and information density, and their utility shifts as training progresses from broad knowledge acquisition to late-stage consolidation. Meanwhile, post-training is often dominated by short-form visual question answering, providing limited supervision for informative and answer-consistent explanations. We introduce InfiMed2, a family of 4B and 27B generalist medical multimodal foundation models built around stage-aware data design. We curate a 55.68B-token corpus that combines broad clinical knowledge with context-rich biomedical visual evidence through source-specific processing. Our CPT pipeline first adapts the vision encoder, then builds broad medical knowledge, and finally transitions to an evidence-focused data mixture during learning-rate decay. For supervised fine-tuning (SFT), we regenerate visual question-answering responses using answer stability, answer-masked reconstruction, and correctness-constrained selection to produce more informative and answer-consistent supervision. The 4B model is further optimized with reinforcement learning with verifiable rewards (RLVR). Across five medical multimodal benchmarks, InfiMed2-4B achieves 66.73% mean accuracy after RLVR, surpassing the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, the highest among the evaluated open-weight models.

Figures & tables

Figure 1: Overview of the InfiMed2 data construction and training pipeline. (a) Source-specific processing forms the warmup–stable CPT corpus and a compact, high-quality mixture for learning-rate decay. (b) Post-training combines fine-grained quality filtering, stability-aware response regeneration, and visual-modality-aware sampling. (c) The resulting data support vision-encoder adaptation, CPT, SFT, and subsequent RLVR for the selected 4B model.

General-purpose models
Model	MMMU Med.	MMMU-Pro Med.-10	MedXQA	PMC-VQA clean	OmniMed VQA	Avg.
Gemini-3-Pro	81.84	74.13	74.85	70.40	84.55	77.15
Gemini-3-Flash	81.51	70.63	68.00	69.75	84.45	74.87
Claude-Opus-4.7	82.47	70.63	69.40	67.15	81.60	74.25
GPT-5	83.39	70.90	71.70	67.30	76.40	73.94
GLM-5V-Turbo	73.91	64.69	53.25	66.55	81.95	68.07

Table 1: Medical multimodal accuracy (%) under a common evaluation protocol. Selected InfiMed2 models are shaded in gray. Bold and underlined entries denote the best and second-best scores.

Figure 2: Parameter efficiency and stage-wise gains. (a) Five-benchmark average accuracy versus model size for open-weight models with reported parameter counts. (b) Benchmark-wise score changes after adding CPT or RLVR to the corresponding matched configuration.

Parameters	Vision-encoder adaptation	Warmup– stable CPT	Learning-rate decay	Post- training	MMMU Med.	MMMU-Pro Med.-10	MedXQA	PMC-VQA clean	OmniMedVQA	Avg.
4B	–	–	–	SFT	69.53	56.64	39.35	62.15	92.12	63.96
	1B	30B	2.5B	SFT	71.18	56.99	42.70	63.00	90.11	64.80
	1B	55B	2.5B	SFT	70.55	55.94	43.05	63.90	90.04	64.69
	1B	30B	2.5B	SFT+RLVR	72.43	58.39	46.25	65.80	90.79	66.73
27B	–	–	–	SFT	75.51	68.88	56.55	66.20	92.44	71.92
	1B	8B	2.5B	SFT	78.88	66.78	57.45	66.20	92.69	72.40

Table 2: CPT checkpoint selection and RLVR ablation. CPT columns report token budgets.

SFT target	MMMU Med.	MMMU-Pro Med.-10	MedXQA	PMC-VQA clean	OmniMedVQA	Avg.
Answer only	58.56	40.21	31.25	65.70	83.19	55.78
Random generated response	68.04	52.10	34.85	59.25	75.60	57.97
Stability-aware selected response	69.81	51.75	35.00	59.20	77.00	58.55

Table 3: Matched lightweight-SFT ablation of response targets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Tokens	Share
Medical e-books	1.000B	40.00%
Biomedical scientific literature	0.500B	20.00%
General multimodal replay	0.396B	15.84%
Medical web articles	0.094B	3.76%
Medical web exercises and selected VQA	0.510B	20.40%
Total	2.500B	100.00%

Appendix

Table 4: Composition of the 2.5B-token mixture used during learning-rate decay.

Dimension	Score	Criterion	Description
Image–question alignment	0	Non-medical image	The image is unrelated to the medical context required by the question.
	0	Modality mismatch	The visual modality required by the question is inconsistent with the provided image, e.g., a question referring to CT while the input is an X-ray.
	0	Anatomical mismatch	The organ, anatomical region, or laterality specified by the question is absent from the provided image(s).
	0	Missing image	The question refers to an image or panel that is not provided, or the available images are insufficient for answering the question.
	1	Image-independent question	The question can be answered without using the visual input.
Information completeness	0	Invalid answer space	The candidate answers do not contain a valid option or are not mutually compatible with the intended question.

Appendix

Table 5: Fine-grained quality assessment criteria used for multimodal data filtering.

Category	Examples	Tokens	Share	Vision	Text
General multimodal instruction	50,000	26.021M	10.76%	16.909M	9.112M
Medical visual question answering	90,734	52.644M	21.77%	41.699M	10.945M
Medical reasoning data	49,808	54.868M	22.69%	16.469M	38.399M
Stability-aware supervision	106,183	108.316M	44.79%	45.382M	62.933M
Total	296,725	241.848M	100.00%	120.459M	121.389M

Appendix

Table 6: Composition of the supervised fine-tuning mixture.

Stage	Count	Retention
Input questions	176,948	100.00%
Generated candidates ( $8$ per question)	1,415,584	100.00%
Candidates passing answer checking	704,046	49.74%
Questions with at least one valid candidate	121,544	68.69%
Long-form examples exported after stability filtering	108,293	61.20%
Final stability-aware SFT examples	106,183	60.01%

Appendix

Table 7: Data flow through stability-aware response regeneration. Percentages for candidates use all generated candidates as the denominator; percentages for questions use the original question set.

Stage	Trainable components	Epochs	Max. length	Batch	Learning-rate schedule
Vision-encoder adaptation	Vision encoder, projector	1	8,192	256 / 2	3% warmup to $2\times 10^{-6}$ , cosine decay to $2\times 10^{-7}$
Warmup–stable CPT	All components	1	12,288	256 / 1	3% warmup to $1\times 10^{-5}$ , then constant
Learning-rate decay	All components	1	12,288	256 / 1	Cosine decay from $1\times 10^{-5}$ to $2\times 10^{-6}$
Supervised fine-tuning	All components	5	9,000	64 / 2	10% warmup, then cosine decay to zero. Peak $2\times 10^{-6}$ for language model and projector and $1\times 10^{-6}$ for vision encoder

Appendix

Table 8: Optimization settings for the main training stages. Batch sizes are reported as global and per-device micro-batch sizes.

Hyperparameter	Value
Learning rate	$1\times 10^{-6}$
Global batch size	128
Training epochs	3
Rollouts per prompt	8
Sampling temperature	1
Format-to-accuracy reward ratio	$1{:}9$

Appendix

Table 9: Optimization settings for reinforcement learning with verifiable rewards.

Category	Source	Train	Test	Total
Medical VQA	PMC-VQA	7,210	790	8,000
	MedSynVQA	5,371	629	6,000
	GMAI-VL-5.5M	2,721	279	3,000
	Internally curated medical data	1,848	206	2,054
	OmniMedVQA + RadImageNet	426	49	475
Text QA	MedQA (English USMLE)	1,800	200	2,000

Appendix

Table 10: Composition of the medical reasoning resources prepared for RLVR.

Figure 3: Qualitative comparison on a multiple-myeloma question from MMMU-Medical-test.

Figure 4: Qualitative comparison on an outbreak-analysis question from MMMU-Medical-test.

Figure 5: Qualitative comparison on a drug-induced megaloblastic-anemia question from MedXpertQA-MM.

Figure 6: Qualitative comparison on cardiac-conduction anatomy from MMMU-Medical-test.

Explore similar work

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Multimodal Clinical Data Multimodal Large Language Models
LASA Team · DAMO Academy, Alibaba Group
OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
Jun 10, 2026Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6Medical Vision-Language Models Recent Vision-Language Models
York University · Vector Institute · University of British Columbia +5
MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5Multimodal Clinical Data Medical Visual Question Answering
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

Jun 8, 2025, cs.CLJ/K move · Enter open · S save

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied+14

LASA Team · DAMO Academy, Alibaba Group

Multimodal Clinical Data Multimodal Large Language Models Large Multimodal Models Wireless Foundation Models Reasoning Skills

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

Abstract

Explore similar work

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis