LineupRL: Verifiable Reinforcement Learning for Time Series Captioning via Caption-to-Series Identification

Haochen Zhang; Laura Yao; Zachary Plotkin; Gengwei Zhang; Tianlong Chen

cs.AIOct 1, 2026

LineupRL: Verifiable Reinforcement Learning for Time Series Captioning via Caption-to-Series Identification

Authors: Haochen Zhang, Laura Yao, Zachary Plotkin, Gengwei Zhang, Tianlong Chen

Organizations: University of North Carolina at Chapel Hill · Foci Labs

Abstract

Time series captioning is a fundamental step in time series understanding and can also serve as the bridge between signal and natural language. Supervised fine-tuning (SFT) relies on a larger model's captions and cannot exceed their quality. Reinforcement learning (RL) can, but its rewards were designed for other modalities and other tasks, and they transfer poorly to open-ended generation in the time series domain. We address this by proposing LineupRL, a reinforcement learning with verifiable rewards (RLVR) pipeline whose reward is caption-to-series identification. The reward model is a frozen large language model (LLM) verifier that reads the generated caption and the candidate time series as raw values, never the chart, and must pick the described time series from multiple distractors. Matching is a far lighter demand on the verifier than writing questions or judging a caption, so an off-the-shelf LLM can supply the reward. Across two captioning benchmarks, and on forecasting and reconstruction where the predictor sees only the caption, LineupRL outperforms SFT and RL baselines on every metric. The 3B vision language model (VLM) trained by LineupRL also outperforms, at 1/24 of the parameters, the 72B VLM whose captions the SFT baseline is distilled from. Our case study shows that LineupRL resists reward hacking, and that the captioner it trains both traces the trend and names the values at key points.

Figures & tables

Figure 1: (a) LLM-as-judge is easily hacked. (b) Answerability needs strong time series understanding. (c) LineupRL (ours).

Figure 2: Overview of LineupRL. (a) The VLM captioner $\pi_{\theta}$ reads the chart $\rho(x)$ of a series $x$ and samples a caption $c$ . (b) RLVR training. For each series a group of $G$ captions is sampled. Each caption is scored by a frozen LLM verifier that receives the caption and the candidate set $\mathcal{C}(x)$ printed as raw numbers and must locate $x$ ; the true series is placed at each of the $K$ positions in turn, and the fraction of correct answers gives the reward $r(c,x)=\lambda\,\mathrm{acc}(c,x)$ . The rewards of the group drive the RLOO policy-gradient update of $\theta$ . (c) Negative selection. The $K-1$ distractors $\mathcal{N}(x)$ are drawn from a pool of other real series of the same length by matching summary statistics and limiting overlap with $x$ , and $\mathcal{C}(x)=\{x\}\cup\mathcal{N}(x)$ .

		BEDTime			CaTS-Bench
Type	Captioner	Entailment	c $\to$ s	s $\to$ c	Entailment	c $\to$ s	s $\to$ c
VLM	Qwen2.5-VL-3B	0.296	0.550	0.495	0.352	0.751	0.721
	Qwen2.5-VL-72B	0.320	0.605	0.484	0.440	0.803	0.643
	Qwen3-VL-8B	0.320	0.669	0.498	0.407	0.817	0.626
	InternVL3-14B	0.357	0.562	0.512	0.506	0.766	0.701
LLM	Phi-3.5-mini	0.180	0.909	0.807	0.318	0.980	0.890

Table 1: Entailment and identification on two captioning benchmarks. c

\to

s is short for caption-to-series ( cap

\to

ser ) identification. s

\to

c is short for series-to-caption ( ser

\to

cap ) identification. Each identification cell is the mean over two frozen verifiers that took no part in training, GLM-4-9B and Phi-4-14B; random chance is

\frac{1}{K}=0.25

. CaTS-Bench entailment is scored against rewritten references (Appendix B.1 ). Bold marks the best of the four models fine-tuned from the same initialization.

	Forecasting				Reconstruction
Captioner	ETTh2	ETTm2	Saug.	Elec.	ETTh2	ETTm2	Saug.	Elec.
Naive baseline	0.3047	0.0596	1.2955	0.9078	0.9501	0.8335	1.1748	0.8634
Empty caption	0.8446 [-2.5pt] $\pm$ 0.0101	0.8001 [-2.5pt] $\pm$ 0.0042	1.0601 [-2.5pt] $\pm$ 0.0016	0.9129 [-2.5pt] $\pm$ 0.0041	0.9284 [-2.5pt] $\pm$ 0.0306	0.8274 [-2.5pt] $\pm$ 0.0029	1.1764 [-2.5pt] $\pm$ 0.0008	0.8667 [-2.5pt] $\pm$ 0.0021
Shuffled caption	0.8437 [-2.5pt] $\pm$ 0.0087	0.8040 [-2.5pt] $\pm$ 0.0046	1.0619 [-2.5pt] $\pm$ 0.0042	0.9124 [-2.5pt] $\pm$ 0.0043	0.9347 [-2.5pt] $\pm$ 0.0121	0.8270 [-2.5pt] $\pm$ 0.0015	1.1774 [-2.5pt] $\pm$ 0.0018	0.8663 [-2.5pt] $\pm$ 0.0010
Qwen2.5-VL-3B	0.3062 [-2.5pt] $\pm$ 0.0086	0.1012 [-2.5pt] $\pm$ 0.0047	0.9419 [-2.5pt] $\pm$ 0.0119	0.5751 [-2.5pt] $\pm$ 0.0351	0.2445 [-2.5pt] $\pm$ 0.0052	0.0633 [-2.5pt] $\pm$ 0.0029	0.6852 [-2.5pt] $\pm$ 0.0173	0.4352 [-2.5pt] $\pm$ 0.0156
Qwen2.5-VL-72B	0.2149 [-2.5pt] $\pm$ 0.0094	0.0451 [-2.5pt] $\pm$ 0.0009	0.8521 [-2.5pt] $\pm$ 0.0061	0.3089 [-2.5pt] $\pm$ 0.0094	0.1134 [-2.5pt] $\pm$ 0.0065	0.0288 [-2.5pt] $\pm$ 0.0022	0.4444 [-2.5pt] $\pm$ 0.0113	0.1993 [-2.5pt] $\pm$ 0.0077

Table 2: Information bottleneck on time series tasks: Mean and standard deviation of test MSE over five training seeds of the predictor are reported. The naive baseline is last-value persistence for forecasting and mean training window for reconstruction. Bold marks every fine-tuned model not significantly worse than the best in its column; significance tests are deferred to Appendix B.5 .

Figure 3: Ablation on the reward’s three knobs. (a) three reward models and (b) $K\in\{4,6,8\}$ , where $K$ is the cardinality of the candidate set in each identification question and $K-1$ the number of distractors; the identification bar is ser $\to$ cap on CaTS-Bench under the two evaluation verifiers. (c) re-scores reference captions of three domains, written by no captioner, under every combination of reward model size (down) and $K$ (across); each cell is accuracy above chance, $(\mathrm{acc}-1/K)/(1-1/K)$ , and darker is higher. Every number behind the three panels is in Appendix C .

	Nearest	Random
Distractors	( ours )
Reward acc. over chance	0.872	0.982
Entailment	0.506	0.354
Ident. c $\to$ s	0.797	0.704
Ident. s $\to$ c	0.753	0.744
Caption length (chars)	667	738

Table 3: Nearest against random distractors.

Figure 4: Case study on captions from three RL-fine-tuned models for the series at the top left. Spans are classed as green , a fact pinned to a time and a magnitude the series supports (within

0.35\sigma

of the series over the interval it points at); blue , a position with no magnitude; red , a claim the series contradicts; gray , content irrelevant to the series. Underline marks a span covering one of the series’ four events (opening maximum, fall, trough at

t{=}6

, rebound), which all three captions visit in order.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	LineupRL	RL, answerability	RL, LLM-as-judge	SFT
Training signal	identification among $K=4$ candidates	accuracy on the chart’s verified questions	grade $1$ – $10$ of the caption	cross-entropy on 72B captions
Frozen model in the loop	Qwen2.5-14B-Instruct verifier	Qwen2.5-14B-Instruct answerer	Qwen2.5-14B-Instruct judge	Qwen2.5-VL-72B-Instruct teacher
Training charts	$9{,}500$	$7{,}243$ ( $23{,}436$ questions)	$9{,}500$	$9{,}500$ pairs
Passes over the data	$4$	$8$	$2$	$5$
Learning rate	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-5}$ , cosine
Checkpoint every	$50$ steps	$25$ steps	$50$ steps	$50$ steps

Appendix

Table 4: Configuration of the four fine-tuned models. Each model is selected on the

500

validation charts under its own training signal, so no model is selected on a test metric. The answerability model trains on fewer charts because a chart enters its training set only if at least two of its questions survive verification.

	SFT	Answ.	Judge	72B	3B init
Forecasting
ETTh2	0.001	$<$ 0.001	0.002	0.002	$<$ 0.001
ETTm2	0.016	$<$ 0.001	0.035	$<$ 0.001 †	$<$ 0.001
Saug.	0.007	$<$ 0.001	0.030	0.005	$<$ 0.001
Elec.	$<$ 0.001	$<$ 0.001	$<$ 0.001	$<$ 0.001	$<$ 0.001
Reconstruction

Appendix

Table 5: Paired

t

-test

p

-values, LineupRL against every other captioner. Two-sided, paired across the 5 predictor seeds. LineupRL has the lower mean everywhere except the cells marked

\dagger

, where the other captioner is ahead.

	Forecasting (predict the next window)
Captioner	ETTh2	ETTm2	Saug.	Elec.
Naive baseline	0.3047	0.0596	1.2955	0.9078
Empty caption	0.8446 $\pm$ 0.0101	0.8001 $\pm$ 0.0042	1.0601 $\pm$ 0.0016	0.9129 $\pm$ 0.0041
Shuffled caption	0.8437 $\pm$ 0.0087	0.8040 $\pm$ 0.0046	1.0619 $\pm$ 0.0042	0.9124 $\pm$ 0.0043
Qwen2.5-VL-3B	0.3062 $\pm$ 0.0086	0.1012 $\pm$ 0.0047	0.9419 $\pm$ 0.0119	0.5751 $\pm$ 0.0351
Qwen2.5-VL-72B	0.2149 $\pm$ 0.0094	0.0451 $\pm$ 0.0009	0.8521 $\pm$ 0.0061	0.3089 $\pm$ 0.0094

Appendix

Table 6: Criterion 3 with across-seed spread. The cells of Table 2 , laid out one study per block so the two panels can be read separately. Bolding and baselines are as in that table. Only the predictor’s training loop changes between seeds, so the spread comes from the predictor; each captioner is a single RL or SFT run.

	Qwen2.5-14B		GLM-4-9B		Llama-3.1-8B		Mistral-Nemo-12B		Phi-4-14B
Captioner	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c
Qwen2.5-VL-3B	0.664	0.566	0.559	0.443	0.472	0.328	0.518	0.349	0.541	0.548
Qwen2.5-VL-72B	0.742	0.625	0.610	0.455	0.582	0.338	0.574	0.334	0.600	0.513
Qwen3-VL-8B	0.752	0.604	0.639	0.446	0.648	0.372	0.620	0.351	0.700	0.549
InternVL3-14B	0.709	0.600	0.585	0.467	0.542	0.360	0.541	0.324	0.539	0.557
Phi-3.5-mini	0.936	0.903	0.893	0.828	0.884	0.668	0.837	0.701	0.925	0.787

Appendix

Table 7: Identification on BEDTime under five frozen verifiers. Every captioner’s c

\to

s and s

\to

c accuracy on the candidate sets of Table 1 , re-scored by each verifier under the same four-rotation protocol (chance

0.25

). Qwen2.5-14B is the reward verifier; GLM-4-9B and Phi-4-14B are the two evaluation verifiers whose mean Table 1 reports. Bold marks the best of the four models fine-tuned from the same initialization, as in Table 1 .

	Qwen2.5-14B		GLM-4-9B		Llama-3.1-8B		Mistral-Nemo-12B		Phi-4-14B
Captioner	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c	c $\to$ s	s $\to$ c
Qwen2.5-VL-3B	0.873	0.803	0.674	0.680	0.675	0.429	0.591	0.449	0.827	0.763
Qwen2.5-VL-72B	0.951	0.806	0.711	0.631	0.763	0.372	0.648	0.434	0.896	0.655
Qwen3-VL-8B	0.933	0.775	0.716	0.587	0.802	0.379	0.650	0.413	0.919	0.665
InternVL3-14B	0.919	0.798	0.684	0.689	0.734	0.384	0.619	0.426	0.849	0.713
Phi-3.5-mini	0.989	0.988	0.979	0.953	0.978	0.819	0.867	0.849	0.981	0.826

Appendix

Table 8: Identification on CaTS-Bench under five frozen verifiers. Every captioner’s c

\to

s and s

\to

c accuracy on the candidate sets of Table 1 , re-scored by each verifier under the same four-rotation protocol (chance

0.25

Domain	Verifier	$K=4$	$K=6$	$K=8$	$K=10$	$K=12$
TSFragment val.	Qwen2.5-14B	0.701	0.634	0.586	0.527	0.496
	Qwen2.5-7B	0.641	0.553	0.491	0.443	0.401
	Qwen2.5-3B	0.455	0.353	0.284	0.186	0.233
CaTS-Bench	Qwen2.5-14B	0.421	0.384	0.357	0.336	0.324
	Qwen2.5-7B	0.257	0.199	0.170	0.151	0.147
	Qwen2.5-3B	0.112	0.100	0.095	0.080	0.105

Appendix

Table 9: Reward discrimination on reference captions, the cells of Figure 3 (c): cap

\to

ser accuracy above chance,

(\mathrm{acc}-1/K)/(1-1/K)

, for three verifiers and five candidate counts on the reference captions of three domains.

	BEDTime				CaTS-Bench
Reward model	Ent.	c $\to$ s	s $\to$ c	Len.	Ent.	c $\to$ s	s $\to$ c	Len.
Untuned Qwen2.5-VL-3B	0.296	0.550	0.495	1084	0.352	0.751	0.721	1287
Qwen2.5-14B ( ours )	0.400	0.709	0.638	621	0.611	0.886	0.868	713
Qwen2.5-7B	0.287	0.710	0.644	1331	0.298	0.850	0.799	1725
Qwen2.5-3B	0.270	0.715	0.710	898	0.381	0.830	0.831	959

Appendix

Table 10: Ablation on the reward model: caption metrics. Effect of the size of the frozen verifier that turns a caption into the reward, with everything else held at the LineupRL recipe. Columns are those of Table 1 : Ent. is the fraction of reference captions the generated one entails, c

\to

s and s

\to

c are the identification accuracies averaged over the two frozen evaluation verifiers, GLM-4-9B and Phi-4-14B (chance

0.25

). Len. is the mean caption length in characters, printed because both entailment columns move with it. Bold marks the best of the three trained settings; the untuned initialization is shown for reference. Figure 3 plots the CaTS-Bench s

\to

c column.

	Forecasting					Reconstruction
Reward model	ETTh2	ETTm2	Saug.	Elec.	Rel.	ETTh2	ETTm2	Saug.	Elec.	Rel.
Untuned Qwen2.5-VL-3B	0.3114	0.1037	0.9234	0.5476	1.850	0.2469	0.0654	0.6698	0.4124	2.283
Qwen2.5-14B ( ours )	0.1687	0.0523	0.8225	0.1922	1.000	0.0974	0.0286	0.4472	0.1317	1.000
Qwen2.5-7B	0.2535	0.0758	0.8709	0.2926	1.369	0.1441	0.0435	0.5318	0.1847	1.391
Qwen2.5-3B	0.2312	0.0783	0.8459	0.3205	1.370	0.1693	0.0534	0.6552	0.2385	1.713

Appendix

Table 11: Ablation on the reward model: Criterion 3. Test MSE (lower is better) of the caption-only predictor of Table 2 , which never sees the series. Every setting keeps

K=4

candidates and the rest of the LineupRL recipe; only the verifier that scores the reward changes. Rel. is the geometric mean of the four per-corpus ratios to the LineupRL row, so

>1

means the caption carries less of what the predictor needs. Bold marks the best of the three trained settings. Figure 3 plots the Rel. columns alone.

	BEDTime				CaTS-Bench
Candidates	Ent.	c $\to$ s	s $\to$ c	Len.	Ent.	c $\to$ s	s $\to$ c	Len.
Untuned Qwen2.5-VL-3B	0.296	0.550	0.495	1084	0.352	0.751	0.721	1287
$K=4$ ( ours )	0.400	0.709	0.638	621	0.611	0.886	0.868	713
$K=6$	0.343	0.664	0.644	482	0.470	0.851	0.869	561
$K=8$	0.340	0.790	0.796	555	0.391	0.852	0.876	620

Appendix

Table 12: Ablation on the number of candidates: caption metrics. Effect of the number of candidate series the verifier must choose among, with everything else held at the LineupRL recipe. Columns are those of Table 1 : Ent. is the fraction of reference captions the generated one entails, c

\to

s and s

\to

c are the identification accuracies averaged over the two frozen evaluation verifiers, GLM-4-9B and Phi-4-14B (chance

0.25

\to

c column.

	Forecasting					Reconstruction
Candidates	ETTh2	ETTm2	Saug.	Elec.	Rel.	ETTh2	ETTm2	Saug.	Elec.	Rel.
Untuned Qwen2.5-VL-3B	0.3114	0.1037	0.9234	0.5476	1.850	0.2469	0.0654	0.6698	0.4124	2.283
$K=4$ ( ours )	0.1687	0.0523	0.8225	0.1922	1.000	0.0974	0.0286	0.4472	0.1317	1.000
$K=6$	0.2046	0.0442	0.8067	0.1925	1.002	0.0975	0.0288	0.4179	0.1234	0.969
$K=8$	0.2206	0.0529	0.8421	0.2120	1.106	0.1006	0.0271	0.4287	0.1382	0.996

Appendix

Table 13: Ablation on the number of candidates: Criterion 3. Test MSE (lower is better) of the caption-only predictor of Table 2 , which never sees the series. Every setting keeps the same frozen Qwen2.5-14B verifier and the rest of the LineupRL recipe; only the number of candidates it must choose among changes. The

K-1

distractors of a smaller

K

are a prefix of those of a larger one, so raising

K

adds candidates rather than exchanging them. Rel. is the geometric mean of the four per-corpus ratios to the LineupRL row, so

>1

means the caption carries less of what the predictor needs. Bold marks the best of the three trained settings. Figure 3 plots the Rel. columns alone.

	BEDTime				CaTS-Bench
Distractors	Ent.	c $\to$ s	s $\to$ c	Len.	Ent.	c $\to$ s	s $\to$ c	Len.
Untuned Qwen2.5-VL-3B	0.296	0.550	0.495	1084	0.352	0.751	0.721	1287
Nearest neighbor ( ours )	0.400	0.709	0.638	621	0.611	0.886	0.868	713
Random	0.324	0.598	0.636	723	0.384	0.810	0.853	753

Appendix

Table 14: Ablation on how the distractors are chosen: caption metrics. Effect of the rule that picks the distractors an item is scored against, with everything else held at the LineupRL recipe. Columns are those of Table 1 : Ent. is the fraction of reference captions the generated one entails, c

\to

s and s

\to

c are the identification accuracies averaged over the two frozen evaluation verifiers, GLM-4-9B and Phi-4-14B (chance

0.25

). Len. is the mean caption length in characters, printed because both entailment columns move with it. Bold marks the best of the two trained settings; the untuned initialization is shown for reference. Table 3 aggregates these columns.

	Forecasting					Reconstruction
Distractors	ETTh2	ETTm2	Saug.	Elec.	Rel.	ETTh2	ETTm2	Saug.	Elec.	Rel.
Untuned Qwen2.5-VL-3B	0.3114	0.1037	0.9234	0.5476	1.850	0.2469	0.0654	0.6698	0.4124	2.283
Nearest neighbor ( ours )	0.1687	0.0523	0.8225	0.1922	1.000	0.0974	0.0286	0.4472	0.1317	1.000
Random	0.2314	0.0643	0.8138	0.2484	1.212	0.1397	0.0268	0.4513	0.1652	1.142

Appendix

Table 15: Ablation on how the distractors are chosen: Criterion 3. Test MSE (lower is better) of the caption-only predictor of Table 2 , which never sees the series. Both settings keep the same frozen Qwen2.5-14B verifier,

K=4

candidates and the rest of the LineupRL recipe; only the rule that picks the

K-1

distractors of a training item changes. Rel. is the geometric mean of the four per-corpus ratios to the LineupRL row, so

>1

means the caption carries less of what the predictor needs. Bold marks the best of the two trained settings. Table 3 aggregates these columns.

Explore similar work

Jun 8, 2026cs.CV

CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning

Image and video captioning are fundamental tasks that bridge the visual and linguistic domains, playing a critical role in pre-training Large Vision-Language Models (LVLMs). Current state-of-the-art captioning models are typically trained with Supervised Fine-Tuning (SFT), a paradigm that relies on expensive, non-scalable annotations and often causes models to memorize specific ground-truth answers, limiting their generality and ability to generate diverse, creative descriptions. To overcome these limitations, we propose applying Reinforcement Learning with Verifiable Rewards (RLVR) to the open-ended task of multimodal captioning. We introduce Captioning Reinforcement Learning++ (CapRL++), a novel reference-free training framework that redefines caption quality through its utility: a high-quality caption should enable a non-visual language model to accurately answer questions about the corresponding visual content. CapRL++ employs a decoupled two-stage pipeline where an LVLM generates a caption, and the objective reward is derived from the accuracy of a separate, vision-free LLM answering Multiple-Choice Questions based solely on that caption. Evaluations on more than 20 image and video benchmarks show that CapRL++ improves dense caption quality and strengthens caption-based pretraining across tasks such as spatial and temporal understanding. Pretraining on scalable image and video caption datasets annotated by CapRL++ yields substantial downstream gains. Furthermore, within the Prism Framework for caption quality evaluation, compact models trained with CapRL++ achieve dense captioning performance comparable to substantially larger models such as Qwen2.5-VL-72B and Qwen3-VL-235B-A22B. These results validate that CapRL++ effectively trains models to produce generalizable, high-fidelity descriptions, establishing a robust foundation beyond the limitations of traditional SFT.

Penghui Yang, Long Xing, Xiaoyi Dong +10

Tsinghua University · University of Science and Technology of China · Microsoft +4

Video Captioning Video Dataset Verifiable Rewards Offline Reinforcement Learning

Ask PDF

May 27, 2026cs.CV

VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning

Visual captioning requires models to capture visual content faithfully while minimizing both omission and hallucination. As the dominant paradigm for captioning, MLLMs have achieved strong performance through scaling and high-quality data. Recently, RL has emerged as a key route to driving MLLMs toward higher precision and broader coverage, however, existing reward designs for captioning fail to provide fine-grained and reliable signals for factual verification, limiting their effectiveness. To address this, we propose VCap, a Witness-Adjudicator reward that pairs the reference caption (a witness) with the visual signal (an adjudicator). By explicitly verifying factual consistency between the reference and policy-generated captions grounded in the visual signal, VCap delivers a reward signal with hypergeometric-distribution-level precision for caption quality verification. This design enables effective learning even from imperfect references, facilitating weak-to-strong generalization in RL training. In our experiments, an 8B model trained with VCap outperforms open- and closed-source SOTA models on multiple image and video captioning benchmarks. Human evaluation further confirms its strong alignment with factual correctness. Additionally, VCap improves MLLM perceptual capability, generalizes across tasks, and surpasses best-of-N distillation, challenging prior assumptions about RLVR.

Xingyu Lu, Jinpeng Wang, Yi-Fan Zhang +13

1Tsinghua Shenzhen International Graduate School · 2Harbin Institute of Technology, Shenzhen · 3Chinese Academy of Sciences +1

Video Captioning Multimodal Large Language Models Witnesses

Ask PDF

May 8, 2026cs.CV

BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

Image captioning is one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era of multimodal large language models (MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned to reinforcement learning (RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing trade-offs across core dimensions of captioning. For example, utility-oriented objectives can encourage noisy, hallucinated, or overlong captions that improve downstream question answering while harming fluency, whereas arena-style objectives can favor fluent but generic descriptions with limited usefulness. To address this, we propose a more balanced RL framework that jointly optimizes utility-aware correctness, reference coverage, and linguistic quality. In order to effectively optimize the resulting continuous multi-objective reward formulation, we apply GDPO-style reward-decoupled normalization to continuous-valued captioning rewards and show that it improves performance over vanilla GRPO. Additionally, we introduce length-conditional reward masking, yielding a more suitable length penalty for captioning. Across LLaVA-1.5-7B and Qwen2.5-VL 3B and 7B base models, our method consistently improves caption quality, with peak gains of +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena across different models.

Shaokai Ye, Vasileios Saveris, Yihao Qian +3

Apple

Image Difference Captioning Video Captioning Multimodal Large Language Models Video Multimodal Large Language Models Balance

Ask PDF