MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretraining

Yuetan Chu; Xinhua Ma; Xinran Jin; Gongning Luo; Xin Gao

cs.CVFeb 25, 2026

MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretraining

Authors: Yuetan Chu, Xinhua Ma, Xinran Jin, Gongning Luo, Xin Gao

Organizations: King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Faculty of Computing, Harbin, China.

Abstract

Medical vision-language pretraining increasingly relies on medical reports as large-scale supervisory signals; however, raw reports often exhibit substantial stylistic heterogeneity, variable length, and a considerable amount of image-irrelevant content. Although text normalization is frequently adopted as a preprocessing step in prior work, its design principles and empirical impact on vision-language pretraining remain insufficiently and systematically examined. In this study, we present MedTri, a deployable normalization framework for medical vision-language pretraining that converts free-text reports into a unified [Anatomical Entity: Radiologic Description + Diagnosis Category] triplet. This structured, anatomy-grounded normalization preserves essential morphological and spatial information while removing stylistic noise and image-irrelevant content, providing consistent and image-grounded textual supervision at scale. Across multiple datasets spanning both X-ray and computed tomography (CT) modalities, we demonstrate that structured, anatomy-grounded text normalization is an important factor in medical vision-language pretraining quality, yielding consistent improvements over raw reports and existing normalization baselines. In addition, we illustrate how this normalization can easily support modular text-level augmentation strategies, including knowledge enrichment and anatomy-grounded counterfactual supervision, which provide complementary gains in robustness and generalization without altering the core normalization process. Together, our results position structured text normalization as a critical and generalizable preprocessing component for medical vision-language learning, while MedTri provides this normalization platform. Code and data will be released at https://github.com/Arturia-Pendragon-Iris/MedTri.

Figures & tables

Figure 1: Illustration of normalization differences across raw reports, RadGraph, and MedTri. The raw clinical report contains irrelevant content and mixed structures. RadGraph extracts only diagnostic entities with limited imaging descriptions. MedTri produces anatomically anchored, image-grounded triplets that preserve morphological and spatial detail while removing stylistic and clinically irrelevant text.

Dataset	# Reports	Image Modality	Body Region
MIMIC-CXR [ 13 ]	65,000	X-ray	Chest
CT-RATE [ 10 ]	20,000	CT	Chest
CT-INSPECTION [ 11 ]	10,000	CT	Chest
AbdomenAtlas [ 17 ]	5,000	CT	Abdomen
Parrot [ 16 ]	2,658	CT, MRI, X-ray, etc	Head, abdomen, chest, etc

Table 1: Overview of the report dataset used to develop and evaluate the MedTri normalization platform.

Input:	IMPRESSION: 1. No evidence of pulmonary embolism. No acute intrathoracic pathology. 2. Mild bronchial thickening, nonspecific, but may reflect infectious or inflammatory process. 3. Main pulmonary artery is dilated measuring 39 mm, possibly reflecting background pulmonary hypertension.
Prompt:	You will receive one paragraph containing radiology findings (image-based description) and a final diagnosis. Your ONLY task is to convert this information into a structured radiology summary using the specified format. Each line must follow this exact format: [Anatomical Term]: [Radiologic Description + Diagnosis Category]. List only the anatomical regions mentioned in the findings or diagnosis. Maintain the anatomical hierarchy (e.g., lung before airway, heart before aorta) if multiple structures are present. For each anatomical term, include radiologic findings (location, size, attenuation, density, texture, etc.) and diagnosis category if explicitly stated. Use objective imaging language only. Use direct descriptive phrasing. Keep each anatomical line concise and single-sentence, using semicolons to separate multiple findings.
Output:	Lung: No evidence of pulmonary embolism; no acute intrathoracic pathology.
	Bronchi: Mild bronchial thickening; nonspecific; may reflect infectious or inflammatory process.
	Main pulmonary artery: Dilated; measures 39 mm; possibly reflecting background pulmonary hypertension.

Table 2: Example of the standardized ChatGPT-5.1 prompt used to generate structured triplets from free-text reports.

Figure 2: Illustration of MedTri’s optional augmentation modules. MedTri-K enriches each triplet with clinically validated radiologic signatures to improve visual interpretability. MedTri-C generates anatomically inconsistent counterfactuals through controlled perturbations to strengthen fine-grained visual–semantic discrimination.

Metric	ChatGPT-5.1	Qwen2.5	MedTri
Anatomical correctness (1-5)	4.888	3.363	4.838
Image groundedness (1-5)	4.775	2.938	4.763
VRAM usage (GB)	–	15.04	1.50
Time/report (s)	3.25±2.14	1.21±0.67	0.45±0.18
BERT score	–	0.734±0.073	0.906±0.072
BLEU	–	0.097	0.541

Table 3: Comparison of computational efficiency, expert evaluation, and normalization accuracy across different systems.

SwinT	MIMIC						NIH						Pneumonia
	1%		10%		100%		1%		10%		100%		1%		10%		100%
	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1
Raw report	0.612	0.218	0.701	0.263	0.781	0.302	0.655	0.221	0.722	0.266	0.762	0.293	0.812	0.673	0.833	0.699	0.848	0.722
RadGraph	0.584	0.205	0.673	0.251	0.748	0.284	0.628	0.207	0.703	0.249	0.747	0.265	0.804	0.665	0.822	0.690	0.840	0.715
MedTri	0.628	0.229	0.716	0.279	0.792	0.314	0.672	0.233	0.739	0.281	0.779	0.304	0.817	0.676	0.836	0.701	0.850	0.723
MedTri-K	0.645	0.241	0.732	0.286	0.784	0.305	0.689	0.242	0.746	0.284	0.771	0.292	0.823	0.681	0.840	0.703	0.851	0.722

Table 4: Downstream classification performance of Swin Transformer (SwinT) and ViT on different X-ray datasets under different text preprocessing strategies. Best performance is marked in bold.

	SwinT						ViT
	1%		10%		100%		1%		10%		100%
	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1
Raw report	0.719	0.493	0.741	0.518	0.775	0.525	0.696	0.473	0.733	0.511	0.745	0.518
RadGraph	0.714	0.480	0.735	0.513	0.744	0.523	0.694	0.471	0.722	0.508	0.716	0.509
MedTri	0.723	0.499	0.753	0.520	0.785	0.538	0.704	0.488	0.745	0.518	0.775	0.537
MedTri-K	0.732	0.510	0.757	0.519	0.789	0.536	0.718	0.498	0.743	0.519	0.775	0.530

Table 5: Downstream classification performance of Swin Transformer (SwinT) and ViT on CT-RATE datasets under different text preprocessing strategies. Best performance is marked in bold.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

Sep 28, 2026

Clinical Trajectory Alignment for Medical Vision-Language Pre-training

Medical vision-language pre-training largely follows a visit-level image-report matching paradigm, aligning paired images and reports at individual visits. While effective for static cross-modal correspondence, this paradigm provides limited supervision for longitudinal clinical change, such as whether abnormalities improve, remain stable, or worsen over time. Learning such change is challenging because temporal semantics are implicit in free-text reports, and different abnormalities within the same patient may evolve asynchronously or even in opposite directions. We propose MedCTA, which reframes medical vision-language pre-training from visit-level cross-modal matching to learning clinical change. Rather than compressing a patient history into a single temporal representation, MedCTA models clinical change at two complementary scopes. At the abnormality scope, clinically grounded queries construct abnormality-conditioned visual and textual trajectories to capture heterogeneous abnormality evolution. At the patient-course scope, global image and report sequences are modeled to capture overall clinical progression beyond any individual abnormality. Structured trend supervision is extracted from longitudinal reports by an offline LLM parser, removing the need for manual temporal annotations. Combined with static image-report alignment, MedCTA learns representations that preserve visit-level cross-modal correspondence while encoding longitudinal change semantics. Experiments on temporal image classification, image-text retrieval, and zero-shot classification show consistent gains over strong medical vision-language baselines.

Huimin Yan, Xian Yang, Zhi Wang +1

Institute of Intelligent Information Processing, Shanxi University, Taiyuan, China · Alliance Manchester Business School, The University of Manchester, Manchester, UK

Multimodal Pretraining Longitudinal Medical Image Analysis Medical VLMs Vision-Language Pretraining

Ask PDF

Mar 10, 2026

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.

Xiyao Wang, Xiaoyu Tan, Yang Dai +3

Shanghai University of Engineering Science, China · National University of Singapore, Singapore · Case Western Reserve University, United States

Medical Image Representation Learning Vision-Language Pretraining Medical Imaging Vision Transformer

Ask PDF

Feb 27, 2026

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.

Haowen Zhu, Ning Yin, Xiaogen Zhou

School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology

Medical VLMs Vision-Language Pretraining Medical Image Anomaly Detection Modality Gap in VLMs 3D Medical Imaging

Ask PDF

Abstract

Medical VLMs

Explore similar work

Clinical Trajectory Alignment for Medical Vision-Language Pre-training

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection