cs.CVMar 8, 2026

VIVECaption: A Split Approach to Caption Quality Improvement

Authors: Varun Ananth, Baqiao Liu, Haoran Cai

Organizations: Adobe Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Caption quality has emerged as a critical bottleneck in training high-quality text-to-image (T2I) and text-to-video (T2V) generative models. While visual language models (VLMs) are commonly deployed to generate captions from visual data, they suffer from hallucinations, poor compositional reasoning, and limited fine-grained understanding, resulting in misaligned image-caption pairs that degrade downstream model performance. This technical report introduces VIVECaption, a systematic two-sided approach to caption quality improvement. We first establish a comprehensive taxonomy of caption evaluation metrics, distinguishing between "universal" and "instance-grounded" metrics, with the ultimate goal of showcasing the use-cases and tradeoffs between different caption quality metrics. We then use this language to describe our two-sided approach to caption quality improvement: (1) a gold-standard dataset creation methodology using stratified sampling and (2) a model alignment strategy encompassing context alignment and parameter-level finetuning using SFT. We demonstrate our methodology on open-source models, focusing on structured caption formats that enable better parsing and downstream utilization. We ultimately show that using a finetuned character detection model in an image captioning pipeline significantly improves holistic image-caption alignment quality. Our work addresses the growing need for high-quality "vegan" training data in enterprise AI development, providing practical solutions for teams seeking to improve caption-image alignment without relying on potentially copyright-protected web-scraped content.

Figures & tables

Figure 1: As seen in [ 3 ], the character in the image above is Ellie not Victoria. However, the caption claims Victoria to be the focus of the image. This datapoint, if used to train a downstream T2I model, would actively harm the model’s performance.
Figure 2: Diagram illustrating the large categories of caption quality metrics: universal and instance-grounded. The two sub-categories of universal metrics are also displayed, with examples for all types.
Figure 3: Characters from “Sprite Fright”. These images were used for in-context alignment as a reference to both the character detection model and image captioning model. Providing images allows the VLM to “understand” the language particular to this short film. Although obviously preferable, the images do not need to be “uniform” in character posture or angle.
Figure 4: (Left) UMAP projection onto two dimensions of CLIP embeddings of all sampled frames. Color distinguishes each of the 310 “clusters” as located by HDBSCAN. Samples from each cluster were used to make a gold-standard dataset. (Right) Pie chart showing the occurrence of each character in the gold-standard dataset after stratified sampling on HDBSCAN clusters. Ellie’s over-representation is a consequence of her being the main character and is essentially unavoidable. The total number of samples in the gold standard dataset is 310, which is 14.35% of the entire 2161 frame dataset.
Figure 5: (Left) Plot showing relative Qwen2.5-VL parameter sizes and the effect on the average Macro F1 score on the test set pre-and-post SFT. The greatest jump in performance for both the baseline and SFT comes between the 3B and 7B parameter models. (Right) Plot showing relative Qwen2.5-VL parameter sizes and the effect on the average # of mistakes (per example) on the test set pre-and-post SFT. The greatest jump in performance for both the baseline and SFT comes between the 3B and 7B parameter models.
MetricModel Parameters
 3 B7 B3 2B
BaselineSFTBaselineSFTBaselineSFT
Precision ↑\uparrow0.460.910.710.920.750.91
Recall ↑\uparrow0.520.870.640.930.750.94
Macro F1 ↑\uparrow0.440.880.660.920.740.92
# Mistakes ↓\downarrow2.050.340.870.210.500.23
Table 1: Table showing the values on the test set for the following instance-grounded metrics: precision, recall, MacroF1, and # of mistakes. In bold are the best values for each metric across all model sizes and baseline vs. SFT. Italicized values indicate the best performance on a metric within a model size. SFT always outperforms the baseline.
MetricModel Parameters
 N o Character Detection7 B3 2B
BaselineBaselineSFTBaselineSFT
Overall Score (/10)5.896.367.356.827.26
Salient Objects Score (/10)5.446.457.526.817.29
Characters Score (/10)3.894.485.444.955.42
Background Score (/10)9.449.119.529.299.45
Table 2: Table showing the values for the following VLM-as-eval metrics on 300 randomly sampled images not in the train/test sets: salient objects, characters, background, and scene. Overall score is a simple mean of all the aforementioned values. In bold are the best values for each metric across all model sizes and baseline vs. SFT. Italicized values indicate the best performance on a metric within a model size. SFT always outperforms the baseline.
Figure 6: Spider charts showing the VLM-as-eval metric improvements for the two-stage captioning pipeline when a Qwen2.5-VL 7B or 32B model is employed for character detection. Also shown are the evaluation scores for single-pass captioning without a character detection stage and all characters passed as context. 300 random images not in the train/test set for finetuning were selected to be captioned and evaluated. The caption quality improvements when the model is fine tuned with SFT carry beyond just the “character” section of the structured caption.
MetricModel Parameters
 7 B3 2B
p-value
Overall Score6.72e-154.48e-05
Salient Objects Score1.82e-101.25e-03
Characters Score2.71e-101.12e-03
Background Score6.27e-076.44e-03
Table 3: Results of a right-tailed paired t-test on the VLM-as-eval scores from 300 randomly sampled frames not in the train/test sets. Only compares statistical significance of the difference between the non-finetuned model and the finetuned model (no character detection is not in consideration). Even after applying conservative Bonferroni correction, we arrive at a p-value threshold of 0.055=0.01\frac{0.05}{5}=0.01 . Given this threshold, all differences between the quality of caption subsections with/without a finetuned character detection model are significant.
Figure 10
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Training/Evaluation loss and mean token accuracy curves for finetuning Qwen2.5-VL-3B. Model trained using PEFT (LoRA) for 5 epochs and a learning rate of 1e-4 with the AdamWFused optimizer.
Figure 8: Training/Evaluation loss and mean token accuracy curves for finetuning Qwen2.5-VL-7B. Model trained using PEFT (LoRA) for 5 epochs and a learning rate of 1e-4 with the AdamWFused optimizer.
Figure 9: Training/Evaluation loss and mean token accuracy curves for finetuning Qwen2.5-VL-32B. Model trained using PEFT (LoRA) for 5 epochs and a learning rate of 1e-4 with the AdamWFused optimizer.
Figure 14
Figure 15
Figure 16
More in this topic

Image Captioning Evaluation

See all 16 papers

Explore similar work

CardsList
Aug 3, 2026

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang

University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Image CaptioningImage Captioning EvaluationT2I GenerationMultimodal Understanding
AskPDF
Feb 25, 2026

CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning

Image captioning remains a fundamental task for vision language understanding, yet ground-truth supervision still relies predominantly on human-annotated references. Because human annotations reflect subjective preferences and expertise, ground-truth captions are often incomplete or even incorrect, which in turn limits caption models. We argue that caption quality should be assessed by two objective aspects: completeness (does the caption cover all salient visual facts?) and correctness (are the descriptions true with respect to the image?). To this end, we introduce CCCaption: a dual-reward reinforcement learning framework with a dedicated fine-tuning corpus that explicitly optimizes these properties to generate Complete and Correct Captions. For completeness, we use diverse LVLMs to disentangle the image into a set of visual queries, and reward captions that answer more of these queries, with a dynamic query sampling strategy to improve training efficiency. For correctness, we penalize captions that contain hallucinations by validating the authenticity of sub-caption queries, which are derived from the caption decomposition. Our symmetric dual-reward optimization jointly maximizes completeness and correctness, guiding models toward captions that better satisfy these objective criteria. Extensive experiments across standard captioning benchmarks show consistent improvements, offering a principled path to training caption models beyond human-annotation imitation.

Zhijiang Tang, Linhua Wang, Jiaxin Qi +4

Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China · LLM Team, Shopee Pte. Ltd., Shanghai, China

Image CaptioningRL for VLMs
AskPDF
Sep 9, 2026

Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.

Zizhen Wang, Bo Feng, Zhengfeng Lai +5

Apple

Video Caption EvaluationVideo Captioning
AskPDF