Learning to Rank Caption Chains for Video-Text Alignment

Ansel Blume; Burak Uzkent; Shalini Chaudhuri; Garin Kessler

cs.CVMar 26, 2026

Learning to Rank Caption Chains for Video-Text Alignment

Authors: Ansel Blume, Burak Uzkent, Shalini Chaudhuri, Garin Kessler

Organizations: University of Illinois Urbana-Champaign · Amazon Prime Video

Abstract

Direct preference optimization (DPO) is an effective technique to train language models to generate preferred over dispreferred responses. However, this binary "winner-takes-all" approach is suboptimal for vision-language models whose response quality is highly dependent on visual content. In particular, a response may still be faithful to the visual inputs even if it is less preferable than an alternative. The standard Bradley-Terry DPO formulation lacks this nuance, upweighting winning responses without sufficient regard for whether the "losing" response still maintains high visual fidelity. In this work, we investigate ranking optimization as an alternative that more precisely situates responses' faithfulness to visual inputs. We focus on video-text alignment using detailed video captions, proposing a method to generate challenging, totally ordered caption chains at scale through repeated caption degradation. Our results show ranking optimization outperforms binary DPO for long-form content generation and assessment, and importantly, we find that these approaches require finetuning of the vision encoder to be effective, challenging the view of DPO as purely a language-reweighting process.

Figures & tables

Figure 1: Existing models struggle to differentiate between visual details in long-form and detailed captions. We propose caption ranking to capture fine-grained differences for video-language alignment.

Figure 2: Our proposed framework. Given a set of high-quality video captions, an LLM generates a totally ordered caption chain by repeatedly introducing visually-grounded errors into the previous caption. These caption chains are then used to train VLMs by having them rank the generated captions using ranking-based DPO. The captions’ similarity forces the models to pay attention to fine, visually-grounded details.

Figure 3: Error types used to generate caption chains through repeated caption mutation.

Method	On-Policy	Rel.	Descr.	Temp C.	Flu.
Error Injection (MPO)	✗	4.90	4.62	5.38	8.46
Ours - Caption Degradation (Rank)	✗	5.27	4.92	5.75	8.47
Response Improvement	$\checkmark$	5.07	4.92	5.32	7.61
Response Comparison	$\checkmark$	5.12	4.71	5.47	8.60
RLAIF-V	$\checkmark$	5.13	4.83	5.52	8.57

Table 1: Comparison of our proposed Caption Degradation method to other on- and off-policy DPO data generation methods with the PLM model. Captions are evaluated by LLM-as-a-Judge along Relevance ( Rel. ), Descriptiveness ( Descr. ), Temporal Consistency ( Temp C. ), and Fluency ( Flu. ) on PE-Video.

	Rel.	Descr.	Temp C.	Flu.	Rel.	Descr.	Temp C.	Flu.	Rel.	Descr.	Temp C.	Flu.	Rel.	Descr.	Temp C.	Flu.
Method	PerceptionLM								Qwen2.5-VL
	MSR-VTT				PE-Video				MSR-VTT				PE-Video
Pretrained	6.29	5.49	6.18	7.19	6.72	6.04	5.82	6.98	6.73	6.14	6.61	8.69	6.74	6.12	6.63	8.69
SFT	6.44	6.37	6.98	8.54	6.35	5.60	6.07	8.15	7.13	6.83	7.76	9.14	7.31	7.24	7.64	9.35
DPO	7.29	7.34	8.02	9.33	7.57	7.59	7.78	9.43	7.12	6.86	7.77	9.13	7.33	7.25	7.60	9.36
MPO	7.31	7.38	8.08	9.32	7.57	7.59	7.80	9.42	7.11	6.89	7.84	9.15	7.35	7.27	7.66	9.34

Table 2: Captioning results on PE-Video and MSR-VTT splits of Rcc . Models trained with the ranking objective perform best across the board, with DPO-based methods outperforming SFT. We note that MPO and DPO benefit from captions generated with our Caption Degradation method as they are fine-tuned on it.

	Cap. Chains		Err. Inj.
	Rank.	Other	Rank.	Other
vs. MPO	53%	47%	60%	40%
vs. DPO	56%	44%	61%	39%

Table 3: Head-to-head human evaluation of PLM on Rcc -PVD. Ranking objective combined with ordered caption chains outperform DPO and MPO 20% and 21%. On the other hand, our ordered caption chain data improves the MPO and DPO baselines by 7% and 5%.

	Score	Accuracy	Cost-H $(\downarrow)$	Cost-O $(\downarrow)$	Score	Accuracy	Cost-H $(\downarrow)$	Cost-O $(\downarrow)$
PerceptionLM					Qwen2.5-VL
Method	VDC		Argus		VDC		Argus
Pretrained	2.01	.382	.679	.895	2.30	.443	.664	.820
SFT	1.95	.373	.712	.875	2.33	.453	.703	.786
DPO	2.26	.436	.570	.806	2.34	.454	.683	.786
MPO	2.27	.438	.572	.801	2.35	.455	.684	.772

Table 4: Results on the VDC and ARGUS video captioning benchmarks. Ranking-based models show strong performance from both families. MPO and DPO benefit from our data generation approach (ordered captions) as they are fine-tuned on it.

Table 8

	✗	$\checkmark$	✗	$\checkmark$	✗	$\checkmark$
Metric	SFT		DPO		Rank
Relevance	7.31	7.34	7.33	7.35	7.34	7.46
Descriptiveness	7.24	7.25	7.23	7.25	7.25	7.40
Temporal Consistency	7.64	7.62	7.61	7.60	7.64	7.75
Fluency	9.35	9.35	9.34	9.36	9.36	9.40

Table 7: Performance of Qwen2.5-VL with a frozen (✗) or fine-tuned (

\checkmark

) vision encoder on PE-Video. Fine-tuning the Vision Encoder is critical for our Ranking objective.

Figure 4: Performance of Qwen2.5-VL and PLM w.r.t number of captions utilized in the chain. Model performance generally increases with chain length.

Hinge	7.58	7.72	7.88	9.47	7.34	7.36	7.79	9.38
Loss	PerceptionLM				Qwen2.5-VL
	Rel.	Descr.	Temp C.	Flu.	Rel.	Descr.	Temp C.	Flu.
Plackett-Luce DPO	7.67	7.72	7.93	9.49	7.46	7.40	7.75	9.40

Table 8: Comparison between models trained with different losses on Rcc -PVD.

Panel of Figure 5 — Figure 5: Qualitative comparison of our Ranking model vs. an SFT baseline. Green highlights video-grounded descriptions. The Ranking model captures finer details better than SFT. See Appendix for more in-depth comparison and examples.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	Rel.	Descr.	Temp C.	Flu.	Rel.	Descr.	Temp C.	Flu.
Method	MSR-VTT				PE-Video
PerceptionLM
Pretrained	3.16	2.83	3.32	6.89	3.65	3.09	2.83	6.62
SFT	3.68	3.55	4.41	7.95	3.81	3.14	3.53	8.06
DPO	4.45	4.32	5.60	8.56	5.03	4.60	5.30	8.69
MPO	4.45	4.36	5.56	8.57	5.06	4.61	5.29	8.70

Appendix

Table 1: Captioning results on PE-Video and MSR-VTT splits of Rcc using GPT-OSS as a judge to ensure that evaluations are not biased by Claude. Captions are evaluated by Relevance ( Rel. ), Descriptiveness ( Descr. ), Temporal Consistency ( Temp C. ), and Fluency ( Flu. ). Our ranking method achieves the best or tied-best results across nearly all metrics and datasets.

Method	METEOR	ROUGE-L
Pretrained	.152	.234
SFT	.124	.201
DPO	.196	.292
MPO	.196	.293
Ranking	.199	.293

Appendix

Table 2: Non-LLM-as-judge metrics for PLM on Rcc -PVD. We note that DPO and MPO benefit from the data generated by our caption degradation method as they are fine-tuned on it. The Ranking provides further gains compared to MPO given that both are fine-tuned on the dataset generated by our method.

Model	Error
Condition.	Rcc -PVD
		Relevance	Descriptiveness	Temporal C.	Flu.
PerceptionLM	✗	7.48	7.41	7.62	8.80
PerceptionLM	$\checkmark$	7.67	7.72	7.93	9.49

Appendix

Table 3: Comparison between PLM trained on chains generated with error-conditioning vs. PLM trained on chains with freely introduced errors.

Model	Relevance	Descriptiveness	Temporal Consist.	Fluency
MPO (Indep. Errs)	4.90	4.62	5.38	8.46
Ranking	5.27	4.92	5.75	8.47

Appendix

Table 4: Comparison between progressively degraded caption chains vs. sets of captions where each caption contains a single, independent error based on the original caption. Captions are generated and evaluations performed by GPT-OSS 20B. Progressive degradation with ranking outperforms independent errors across all metrics.

Method	On-Policy	PE-Video
		Rel.	Descr.	Temp C.	Flu.
PerceptionLM
Pretrained	✗	3.65	3.09	2.83	6.62
Rank	✗	5.27	4.92	5.75	8.47
Response Comparison	$\checkmark$	5.03	4.79	5.51	8.45
With Pre-SFT	$\checkmark$	5.12	4.71	5.47	8.60

Appendix

Table 5: Comparison between on- and off-policy data generation methods training PLM on Rcc -PVD. Despite having model-tailored data generation, the on-policy baselines do not outperform our ranking approach on video detailed captioning. Data generated and evaluated with GPT-OSS 20B.

Method	Relevance	Descriptiveness	Temporal C.	Fluency
RankNet [ 4 ]	4.91	4.67	5.42	8.52
Hinge	4.86	4.63	5.36	8.43
Plackett-Luce	5.27	4.92	5.75	8.47

Appendix

Table 6: Comparison of ranking losses with PLM on Rcc -PVD. The Plackett-Luce loss achieves the best results on three out of four metrics.

Figure 2: An example of model caption predictions on MSR-VTT with GPT-OSS 20B as a judge.

Figure 3: Examples from our Long-Form Multiple Choice Question Answering benchmark. Green represents the correct answer.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 361 papers

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
Mar 23, 2026Kaili Huang, Hongming Zhang, Rui Shen +4Vision-Language Alignment Preference Optimization
SenseTime Research
Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning
Sep 29, 2026Yanan Wang, Tingsong Li, Kaixun Jiang +3Direct Preference Optimization Video-Language Model Evaluation
Zhejiang University · Alibaba Token Hub, Alibaba Group · University of Science and Technology of China +2
Visual Preference Optimization with Rubric Rewards
Apr 14, 2026Ya-Qi Yu, Fangyu Hong, Xiangyang Qu +15Preference Optimization Multimodal Reward Modeling
Huawei Technologies Co., Ltd.

Mar 23, 2026J/K move · Enter open · S save

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

Kaili Huang, Hongming Zhang, Rui Shen, Linjun Dai+3

SenseTime Research

Vision-Language Alignment Preference Optimization Direct Preference Optimization VLM Hallucination

Open paper AskPDF arXiv

While Direct Preference Optimization (DPO) has become the de facto approach for aligning Large Vision-Language Models (LVLMs), it suffers from Likelihood Displacement, where the probability of both chosen and rejected responses collapses. This optimization flaw is especially detrimental in multimodal settings: the erosion of chosen likelihoods -- a failure we term Visual Anchor Collapse -- causes models to abandon visual evidence for strong language priors, precipitating significant hallucinations. To address this, we propose Asymmetric Constrained Preference Optimization (ACPO), a modality-agnostic alignment mechanism that applies dynamic, target-oriented scaling to preference optimization. ACPO derives a complexity-aware scaling coefficient applied exclusively to the rejected reward, asymmetrically suppressing the gradient flow on the rejected term while preserving the chosen distribution as a gradient-stable reference. While fundamentally a general-purpose objective, breaking this gradient symmetry is crucial for multimodal tasks, as it mitigates the suppression of visual tokens by language priors. Experiments on InternVL models demonstrate that ACPO effectively reverses the chosen-reward degradation of standard DPO. By halting Visual Anchor Collapse, ACPO generally outperforms baselines on hallucination benchmarks (HallusionBench, MM-IFEval) and general leaderboards (MMBench, MMStar, OCRBenchV2) while driving concurrent improvements in general capabilities.

Abstract

Vision-Language Alignment

Explore similar work

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning

Visual Preference Optimization with Rubric Rewards