Clinically Structured Surrogate Rewards for Post-SFT Medical Image Captioning

Hyun Jun Kim; Heeseung Shin; Changwon Lim

cs.CVAug 19, 2026

Clinically Structured Surrogate Rewards for Post-SFT Medical Image Captioning

Authors: Hyun Jun Kim, Heeseung Shin, Changwon Lim

Organizations: Department of Smart Cities, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of AI in Medicine, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Statistics and Data Science, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea

Abstract

Medical image captioning requires translating heterogeneous visual evidence into concise clinical descriptions, where errors in findings, assertion states, or anatomical relations can alter clinical meaning despite surface-level fluency. Sequence-level policy optimization can directly optimize complete captions, but common rewards rely on global text similarity, direct image-caption compatibility, or unordered concept overlap, leaving visual neighborhoods and clinical-claim structure implicit. We propose a clinically structured surrogate reward framework for post-SFT medical image captioning. The framework combines biomedical semantic and short-range lexical fidelity with two structured rewards: distributional image-neighborhood alignment, which matches the medical-image-bank distributions induced by reference and generated captions, and clinical graph consistency, which applies maximum-weight one-to-one matching to entities, assertion states, and typed relations. The four rewards are independently normalized within each rollout group, combined with fixed relative weights, and optimized with GDPO. Across organizer-evaluated hidden test sets for the Standard and Synthetical ImageCLEFmedical Caption tracks and three vision-language backbones, the method improves Overall, Relevance, and Factuality over matched SFT baselines in all six backbone-track combinations, with average relative gains of 3.4%, 2.1%, and 5.8%, respectively. Ablations and paired diagnostics indicate that the structured rewards provide complementary signals, reducing image-neighborhood divergence and improving entity-assertion-relation consistency.

Figures & tables

Figure 1: Overview of the proposed SFT–GDPO framework. The selected SFT checkpoint initializes the rollout policy and is retained as the frozen KL reference. For each input,

K

sampled captions are scored by four reward branches, independently standardized within the rollout group, combined using fixed weights, and converted into a caption-level advantage for GDPO.

Track	Backbone	Stage	Aggregate Scores			Relevance Metrics				Factuality Metrics		$\Delta$
Track	Backbone	Stage	Ovr.	Rel.	Fact.	BERT	R-1	Sim.	BLT	MedCAT	Align	$\Delta$
Std.	MedGemma-4B	SFT	0.3669	0.5494	0.1843	0.6081	0.2900	0.9671	0.3325	0.2067	0.1619	–
	MedGemma-4B	GDPO	0.3798	0.5664	0.1932	0.6205	0.3223	0.9827	0.3402	0.2189	0.1676	+3.54%
	Qwen3.5-MedVL	SFT	0.3675	0.5537	0.1812	0.6160	0.2939	0.9715	0.3335	0.2054	0.1570	–
	Qwen3.5-MedVL	GDPO	0.3898	0.5790	0.2007	0.6266	0.3318	1.0052	0.3525	0.2314	0.1700	+6.09%
	Qwen3-VL-4B	SFT	0.3685	0.5475	0.1895	0.6104	0.2901	0.9587	0.3307	0.2083	0.1707	–

Table 1: Organizer-evaluated hidden test performance. Bold indicates the better result within each SFT–GDPO pair. R-1, Sim., and BLT denote ROUGE-1, scaled MedImageInsight image–caption similarity, and BLEURT, respectively.

\Delta

denotes the relative improvement in Overall score over SFT.

Configuration	Step	Rel.	Fact.	Ovr.	$\Delta$
Full	1000	0.5722	0.1834	0.3778	–
Semantic only	750	0.5664	0.1861	0.3763	-0.0016
Lexical only	250	0.5557	0.1751	0.3654	-0.0124
Neighborhood only	250	0.5475	0.1500	0.3487	-0.0291
Graph only	1250	0.5403	0.1751	0.3577	-0.0201
w/o Semantic	1750	0.5721	0.1819	0.3770	-0.0009

Table 2: Reward ablation on the Standard validation set using Qwen3.5-MedVL. Neighborhood denotes the distributional image-neighborhood reward

r_{\mathrm{nbr}}

\Delta

denotes the signed Overall difference relative to the full objective and is computed from unrounded scores.

Figure 2: Clinical graph consistency, where GDPO corrects the finding and recovers its localization relation. The supplementary material provides additional successful, failed, and ambiguous cases.;CC-BY [Muacevic et al(2024)]

Metric	SFT $\rightarrow$ GDPO	$\Delta$	95% CI
Nbr. KL $\downarrow$	$0.2323\rightarrow 0.2176$	$-0.0147$	$[-0.0158,-0.0135]$
Entity $F_{E}$ $\uparrow$	$0.3629\rightarrow 0.3904$	$+0.0275$	$[0.0247,0.0303]$
Graph reward $\uparrow$	$0.3010\rightarrow 0.3244$	$+0.0234$	$[0.0208,0.0259]$
Relation $F_{R}$ $\uparrow$	$0.1688\rightarrow 0.1838$	$+0.0150$	$[0.0123,0.0177]$

Table 3: Paired post-hoc diagnostics comparing the selected SFT and GDPO checkpoints under greedy decoding on the Standard validation set. Nbr. KL denotes

D_{\mathrm{KL}}(p^{*}\|q)

. Entries report mean per-example scores, with

\Delta=\mathrm{GDPO}-\mathrm{SFT}

; 95% confidence intervals are obtained from 10,000 paired bootstrap resamples of the per-example differences. Entity and graph metrics are computed on references with at least one extracted entity (16,699/19,240; 86.8%), and relation consistency on references with at least one extracted relation (15,196/19,240; 79.0%).

Image Captioning

Image captioning generates natural-language descriptions of visual content, with research focused on accuracy, coverage, grounding, and evaluation.

See all 96 papers

Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment
Aug 20, 2026Yunseo Lee, Hyun Jun Kim, Heeseung Shin +1Medical VLMs Vision-Language Alignment
Department of Statistics and Data Science, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Smart Cities, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Applied Statistics, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea
CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning
Feb 25, 2026Zhijiang Tang, Linhua Wang, Jiaxin Qi +4Image Captioning RL for VLMs
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China · LLM Team, Shopee Pte. Ltd., Shanghai, China
ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison
May 19, 2026Tianle Li, Xuyang Shen, Yan Ma +7VLM Evaluation Reward Modeling
1The Chinese University of Hong Kong · 2MiniMax

Aug 20, 2026J/K move · Enter open · S save

Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment

Yunseo Lee, Hyun Jun Kim, Heeseung Shin, Changwon Lim

Department of Statistics and Data Science, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Smart Cities, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea · Department of Applied Statistics, Chung-Ang University, 84 Heukseok-ro, Dongjak-gu, Seoul 06974, Republic of Korea

Medical VLMs Vision-Language Alignment Faithfulness in VLMs Inference-Time Language Model Alignment

Open paper AskPDF arXiv

Medical image captioning is a technique that accelerates early-stage diagnostic workflows and enhances the interpretability of medical diagnostic AI systems. However, unlike general image captioning, clinically reliable captioning remains challenging due to grayscale-based modalities, subtle anatomical cues, specialized medical phrasing, and variations in data quality. Despite recent advances in large vision-language models, fluent outputs do not necessarily guarantee sufficient alignment with clinical concept spaces or evaluation criteria. To address this issue, we propose a framework that strengthens clinical alignment by separating and enhancing training-time alignment and inference-time alignment. We build a medical image captioning pipeline that integrates single/dual vision encoders based on BioMedCLIP and SigLIP2, a Q-Former, and a LLaMA-based decoder, and examine the contribution of auxiliary learning for UMLS concept/type prediction. At inference, we apply single-embedding-based reranking to select the best caption among candidates, while at training we introduce MedPAIR-SCST, which combines clinically relevant rewards to shift the generative distribution toward improved clinical alignment. Our experiments show that complementary visual representations with a multi-encoder design and concept-level auxiliary learning help preserve clinically meaningful information. Furthermore, inference-time reranking provides a practical way to improve semantic and clinical alignment without additional training, whereas MedPAIR-SCST goes beyond selection by directly improving the model's distribution to generate more consistent and clinically grounded captions. These findings suggest that jointly leveraging selection-based alignment and reinforcement-learning-based alignment can promote more trustworthy medical image captioning even in data-constrained settings.

Abstract

Image Captioning

Explore similar work

Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment

CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning

ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison