Detailed image captioning demands both factual grounding and fine-grained coverage, yet existing methods have struggled to achieve them simultaneously. We address this tension with Reflective Note-Guided Captioning (ReflectCAP), where a multi-agent pipeline analyzes what the target large vision-language model (LVLM) consistently hallucinates and what it systematically overlooks, distilling these patterns into reusable guidelines called Structured Reflection Notes. At inference time, these notes steer the captioning model along both axes -- what to avoid and what to attend to -- yielding detailed captions that jointly improve factuality and coverage. Applying this method to 8 LVLMs spanning the GPT-4.1 family, Qwen series, and InternVL variants, ReflectCAP reaches the Pareto frontier of the trade-off between factuality and coverage, and delivers substantial gains on CapArena-Auto, where generated captions are judged head-to-head against strong reference models. Moreover, ReflectCAP offers a more favorable trade-off between caption quality and compute cost than model scaling or existing multi-agent pipelines, which incur 21--36% greater overhead. This makes high-quality detailed captioning viable under real-world cost and latency constraints.
Figures & tables
Figure 1: Overview of ReflectCAP. In the offline phase, a multi-agent reflective learning pipeline distills a target LVLM’s recurring captioning errors and omissions into Structured Reflection Notes. In the online phase, these notes guide caption generation for new images, producing captions that better balance factuality and coverage.
Figure 2: ReflectCAP framework. In the offline phase , a multi-agent pipeline analyzes a small exemplar set to distill recurring errors and omissions of the target LVLM into Structured Reflection Notes . In the online phase , these notes guide caption generation: Avoid Notes suppress hallucinations, Include Notes encourage missing details, and a final merge integrates grounded and detail-focused captions into the final output.
Figure 3: Solid and dash-dotted lines denote improvements from zero-shot to ReflectCAP and CapMAS, respectively. ReflectCAP achieves higher F1 scores while requiring 21–36% less compute than CapMAS. Light dashed lines denote performance gains from model parameter scaling. Compared to simply increasing model size, ReflectCAP achieves comparable quality at up to 8× lower compute cost, enabling high-quality, detailed captioning more practical under real-world cost and latency constraints.
Model
Method
Fact.
Cov.
F1
InternVL3.5-4B
Zero-shot
64.1
54.9
59.1
SFT w/ Human-authored
58.2
56.5
57.4
SFT w/ ReflectCAP-generated
66.5
61.6
63.9
Qwen2.5-VL-7B
Zero-shot
68.3
57.9
62.7
SFT w/ Human-authored
57.1
58.0
57.6
SFT w/ ReflectCAP-generated
69.9
63.5
66.5
Table 3: Impact of SFT on detailed captioning factuality.
Figure 4: Factuality comparison between Zero-shot and Grounded Base Caption across all models. Models with stronger instruction-following capabilities show larger gains.
Model
Method
Fact.
Cov.
F1
GPT-4.1-mini
Separate-Merge
84.8
72.8
78.3
Combined
84.8
70.7
77.1
GPT-4.1-nano
Separate-Merge
78.1
68.6
73.1
Combined
78.4
67.0
72.2
Qwen3-VL-8B
Separate-Merge
77.6
72.4
74.9
Combined
74.4
71.2
72.7
Table 4: Separate vs. Combined note injection. Separate-Merge applies Avoid and Include notes in separate passes with merging; Combined injects both into a single prompt.
Figure 5: Ablation on note construction parameters. (a) F1 vs. the number of exemplar images N . Performance saturates at N≈30 , indicating that systematic error patterns can be surfaced from a modest exemplar set. (b) F1 vs. the maximum number of note items per category K . Even K=1 already yields strong gains, with performance improving slightly further at K=5 .
Figure 6: Case study of our pipeline. Top : Zero-shot Caption Middle : ReflectCAP-Base suppresses hallucinations via Avoid notes. Bottom : ReflectCAP-Full recovers embossed text details guided by Include notes. Red denotes hallucinated expressions, blue denotes hallucination-corrected descriptions, and green denotes recovered fine-grained details.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Target Model
Note Source
Fact.
Cov.
F1
Δ F1
Qwen3-VL-8B
Self-generated
77.6
72.4
74.9
—
GPT-4.1 mini
76.7
73.3
74.9
± 0.0
Qwen3-VL-32B
Self-generated
77.9
75.7
76.8
—
GPT-4.1 mini
81.0
75.4
78.1
+1.3
Qwen2.5-VL-7B
Self-generated
68.1
63.5
65.7
—
GPT-4.1 mini
67.4
64.8
66.0
+0.3
Appendix
Table 7: Self-generated error notes vs. notes written by GPT-4.1 mini as a proxy writer. Both note types describe the target model’s error patterns; only the authorship differs. Bold indicates the better score per target model.
Figure 7: Success case. Error notes correct zero-shot hallucinations ( red → green ), and the extract-merge step successfully adds verifiable details ( blue , ✓ ).
Figure 8: Limitation case. Error notes correct zero-shot hallucinations ( red → green ), but the extract-merge step introduces a new spatial error ( blue , ✗ ) when following a missing-detail note that exceeds the target model’s perceptual competence.
Scene Level Captioning (DOCCI)
Product Level Captioning (Fashion-Gen)
Avoid Notes Navoid
Avoid specifying materials, object types, or inferred object roles without clear visible evidence.
Match exact color, pattern, and fabric details precisely from reference or visible image.
Do not add or alter visible object features, colors, or counts without confirmation.
Do not add clothing fit, style, or garment length details not explicitly stated or clearly visible.
Avoid inferring lighting direction, time of day, or environment unless clearly visible.
Avoid subjective or interpretive descriptions not supported by reference or image.
Do not add unsupported details to signs, logos, or symbols.
Do not add accessories, footwear, or personal attributes unless clearly visible or mentioned.
Avoid assumptions about scene context or climate not explicitly shown.
Avoid describing visible details that contradict the reference ( e.g. , visible buttons when concealed).
Appendix
Table 8: Structured Reflection Notes: Scene-level vs. Product-level captioning. Both note sets are generated by the same offline pipeline with GPT-4.1-mini. The notes automatically adapt to domain-specific visual characteristics and description conventions.
Figure 9: Fashion domain qualitative examples. Zero-shot captions produce generic descriptions ( e.g. , “multiple zippers and buttons,” “a classic lapel”), while ReflectCAP generates domain-appropriate captions with precise garment construction vocabulary. Green denotes fashion-specific details recovered by the Structured Reflection Notes.
Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.
Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies. This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT). In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement. Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re3Cap), a retrieval-guided reasoning strategy that enhances image captioning without requiring additional annotations. Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions. Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning. Especially, Re3Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.
Haonan Jia, Shichao Dong, Zenghui Sun +7
Taobao & Tmall Group of Alibaba · The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Artificial Intelligence Laboratory
Dense image captioning is critical for vision-language pretraining and for text-to-image generation, but scaling expert-quality annotations is prohibitively expensive. While synthesizing captions from strong vision-language models (VLMs) is a practical alternative, supervised distillation often yields limited output diversity and weak generalization. Reinforcement learning (RL) could overcome these, but its successes have so far been concentrated in verifiable domains that rely on deterministic checkers---a luxury not available in open-ended captioning. We address this verification bottleneck with RubiCap, an RL framework that derives fine-grained, sample-specific rewards from LLM-written rubrics. For each image, an LLM rubric writer compares captions from a diverse committee of VLMs to identify consensus strengths and diagnose the current policy's deficiencies. These findings are converted into explicit evaluation criteria, enabling an LLM judge to decompose quality assessment and replace coarse scalar rewards with structured, multi-faceted assessments. Across extensive benchmarks, RubiCap achieves the highest win rates on CapArena, outperforming supervised distillation, prior RL methods, human-expert annotations, and GPT-4V-augmented outputs. On CaptionQA, it shows superior word efficiency: our 7B model matches Qwen2.5-VL-32B-Instruct, and our 3B model surpasses its 7B counterpart. Remarkably, using a compact RubiCap-3B as a captioner produces stronger pretrained VLMs than those trained on captions from proprietary models.
Tzu-Heng Huang, Sirajul Salekin, Javier Movellan +2
Work done during an internship at Apple · University of Wisconsin—Madison · Apple