OACap: Caption-Guided Active Observation for Detailed Video Captioning

Hao Shi; Hong Yan; Xiaoyang Zhang; Chao Huang; Qing Cui; Jun Zhou

cs.CVOct 21, 2026

OACap: Caption-Guided Active Observation for Detailed Video Captioning

Authors: Hao Shi, Hong Yan, Xiaoyang Zhang, Chao Huang, Qing Cui, Jun Zhou

Organizations: Ant Group · Zhejiang University · Peking University

Abstract

Unlike question answering, where a query specifies a perceptual target, open-ended detailed captioning must determine what evidence to acquire and how to integrate it into a coherent global description. We propose Observation-Adaptive Video Captioner (OACap), a caption-centered agent that turns unsupported or missing information into targeted observation requests, uses spatiotemporal tools to acquire evidence, and iteratively verifies and revises the caption. OACap captions consistently improve performance across general video-understanding benchmarks using a fixed text-only reader. To better diagnose the information they preserve, we introduce Factual-Semantic Captioning Bench (FSCap-Bench). Its Core-Support Factuality Score (CSFS) combines claim precision with importance-aware fact coverage and responds as expected to four controlled perturbation types on 93.0-98.3% of valid caption pairs. Its Cross-Modal Semantic Round-Trip Score (CSRS) measures semantic preservation through a video-caption-video-caption round trip by matching atomic facts from source and recovered captions. Together, they connect downstream textual-memory utility with factual and semantic preservation.

Figures & tables

Figure 1: OACap and FSCap-Bench at a glance. Left: OACap is a caption-centric video captioner that invokes tools to continually self-verify and revise its evolving caption; fig. 2 presents the complete caption-guided observation–revision workflow. Right: FSCap-Bench evaluates atomic claims against human-verified Core and Support facts. CSFS measures claim precision and importance-aware fact coverage, while CSRS measures visual-semantic recoverability through a fixed video–caption–video–caption round trip.

Figure 2: Caption-guided observation and revision. Caption gaps route targeted temporal sampling, cropping, OCR, or retrieval; returned evidence supports the next revision, which exposes the next gaps.

Figure 3: FSCap-Bench curation and composition. (a) Terminal filtering outcomes; (b) category composition of the final

498

videos; (c) duration distribution.

Figure 4: FSCap-Bench. Videos are curated and annotated as Core and Support facts. CSFS measures factual precision and importance-aware coverage; CSRS matches facts before and after $V\!\to\!C\!\to\!V^{\prime}\!\to\!C^{\prime}$ .

Model / Method	Video-MME			ActivityNet-QA $\uparrow$	VCapsBench
Model / Method	Short $\uparrow$	Medium $\uparrow$	Long $\uparrow$	ActivityNet-QA $\uparrow$	AR $\uparrow$	IR $\downarrow$	CR $\uparrow$
Qwen3.8-27B / Direct Video QA (oracle)	86.72 (overall)			62.10	72.94 (overall)
Qwen3.5-A3B / Single-pass Caption	70.22	62.00	57.33	49.04	61.63	22.52	79.55
Qwen3.5-27B / Single-pass Caption	78.33	69.89	65.11	49.22	66.16	20.84	83.57
Qwen3.5-122B / Single-pass Caption	76.03	67.89	64.99	49.07	64.93	21.48	82.69
Qwen3.8-27B / Single-pass Caption	76.17	65.44	63.52	48.86	69.21	21.51	88.19

Table 1: Caption-as-memory results (%). Video-MME and ActivityNet-QA report accuracy; VCapsBench reports answer recall (AR), inconsistency (IR), and coverage (CR). The grayed oracle row answers questions from the video itself rather than from a caption and is therefore excluded from comparisons; its overall scores also use a different input condition. Bold marks the best comparable result.

Evaluator / configuration	Clean	Controlled perturbation
Evaluator / configuration	Clean	+Correct $\uparrow$	+Hallucination $\downarrow$	$-$ Core $\downarrow$	$-$ Support $\downarrow$
Exact-provenance $D$ (Oracle)	0.9485	0.9974 $\bm{\uparrow}$	0.8696 $\bm{\downarrow}$	0.8376 $\bm{\downarrow}$	0.9168 $\bm{\downarrow}$
PAC-S	0.7744	0.7744	0.7748 ${\color[rgb]{1,0,0}\uparrow}$	0.7706 ${\color[rgb]{0.2031,0.6602,0.3242}\bm{\downarrow}}$	0.7756 ${\color[rgb]{1,0,0}\uparrow}$
EMScore	0.2736	0.2736	0.2736	0.2727 ${\color[rgb]{0.2031,0.6602,0.3242}\bm{\downarrow}}$	0.2743 ${\color[rgb]{1,0,0}\uparrow}$
FactVC	0.2261	0.2261	0.2261	0.2261	0.2266 ${\color[rgb]{1,0,0}\uparrow}$
BLEU-4	0.0268	0.0236 ${\color[rgb]{1,0,0}\downarrow}$	0.0226 ${\color[rgb]{0.2031,0.6602,0.3242}\bm{\downarrow}}$	0.0278 ${\color[rgb]{1,0,0}\uparrow}$	0.0283 ${\color[rgb]{1,0,0}\uparrow}$

Table 2: Controlled caption perturbations on 300 Video-MME short videos. Green arrows match the expected direction; red arrows do not.

Backbone	Direct Caption	OACap Pipeline	$\bm{\Delta}\uparrow$
Qwen3.5-27B	65.87	67.81	+1.94
Qwen3.5-122B	63.38	68.58	+5.20
Qwen3.8-27B	66.50	87.24	+20.75

Table 3: Factual comprehensiveness on FSCap-Bench. CSFS is reported on a

0

–

100

scale. Direct Caption uses a single whole-video caption generated by each backbone, while OACap Pipeline applies our caption-guided observation and revision framework with the same backbone.

\Delta

is the absolute improvement in CSFS points and is computed from unrounded scores.

T2V / Duration / Captioner	$R\uparrow$	$P\uparrow$	CSRS $\uparrow$	Loss $\downarrow$	Drift $\downarrow$
LTX-2.5 / 10s / Qwen3.8-27B	0.206	0.352	0.245	0.794	0.648
LTX-2.5 / 10s / Qwen3.8-Max	0.230	0.257	0.231	0.770	0.743
Wan2.2 / 10s / Qwen3.8-27B	0.168	0.296	0.206	0.832	0.704
Wan2.2 / 10s / Qwen3.8-Max	0.187	0.241	0.200	0.813	0.759
LTX-2.5 / 60s / Qwen3.8-27B	0.213	0.362	0.254	0.787	0.638
LTX-2.5 / 60s / Qwen3.8-Max	0.291	0.327	0.291	0.709	0.673

Table 4: Cross-modal round-trip results on 33 FSCap-Bench videos sourced from Video-MME, all shorter than two minutes (per-video macro average).

R

: source-fact recovery;

P

: recovered-fact precision; CSRS: fact-level

F_{1}

; Loss

=1-R

; Drift

=1-P

. LTX is evaluated at 10 and 60 seconds; Wan is evaluated at 10 seconds.

Video Caption Evaluation

See all 13 papers

CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning
May 26, 2026Zihan Lin, Songhe Deng, Shuwei He +6Video Caption Evaluation Video Captioning
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Sep 9, 2026Zizhen Wang, Bo Feng, Zhengfeng Lai +5Video Caption Evaluation Video Captioning
Apple
DualFact+: A Multimodal Fact Verification Framework for Procedural Video Understanding
Apr 28, 2026Cennet Oguz, Yasser Hamidullah, Josef van Genabith +1Multimodal Grounding Video Understanding
German Research Center for Artificial Intelligence (DFKI) · Saarland Informatics Campus

May 26, 2026J/K move · Enter open · S save

CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning

Zihan Lin, Songhe Deng, Shuwei He, Danxiang Zhu+5

ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University

Video Caption Evaluation Video Captioning Video-Language Models Dense Video Captioning

Open paper AskPDF arXiv

Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.

Abstract

Video Caption Evaluation

Explore similar work

CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning

Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

DualFact+: A Multimodal Fact Verification Framework for Procedural Video Understanding