Compressed-domain video captioning avoids full video decoding by operating directly on I-frames, motion vectors and residuals, trading a small amount of accuracy for a large gain in inference speed. CoCap established this pipeline using a CLIP vision encoder and a shallow BERT-style multimodal decoder. Both components predate substantially stronger alternatives. We ask a narrow, controlled question: how much of CoCap's accuracy is limited by these two components, and which of the two is the binding constraint? We replace the CLIP I-frame encoder with SigLIP2 and the BERT-style decoder with GPT-2, and evaluate three configurations (the original pairing, the encoder substitution alone, and both substitutions together) under identical data, sampling budget and optimisation schedule. All comparisons are made against our own reproduction of CoCap rather than its published numbers, because we train on a 4,999-clip subset of VATEX at a reduced sampling budget; absolute values are therefore not comparable with the original work. Our reproduction tracks the published result closely: CIDEr and METEOR run slightly above it (54.9 against 52.7; 23.4 against 23.2), BLEU-4 and ROUGE-L slightly below (29.7 against 31.4; 48.9 against 49.4). We attribute the differences to our evaluation subset rather than to any improvement in either direction. We find the two substitutions pull in opposite directions: SigLIP2 alone improves every metric (+4.5 CIDEr), while adding GPT-2 on top erodes that gain, because a pretrained decoder overfits 4,999 clips within two epochs. We additionally report inference latency for each configuration, since speed is the property that motivates compressed-domain captioning in the first place, and an accuracy gain purchased at a latency cost should be reported as such.
Figures & tables
Figure 1: The captioner with both substitutions applied. Highlighted blocks are the components we replace; the remainder follows CoCap [ 12 ] . Per Group of Pictures, one I-frame is encoded by SigLIP2 and M motion-vector/residual pairs by two lightweight, randomly initialised Vision Transformers. The action encoder fuses them, using the motion features as queries against the I-frame’s patch tokens. The resulting per-GOP context and action features form a visual prefix for a causal GPT-2 decoder.
This work
CoCap [ 12 ]
Training clips
4,999
25,991
Evaluation clips
1,000
full test split
GOPs per clip ( N )
5
8
Frames per GOP ( M )
16
59
Motion-vector channels
2
4
Epochs
12
20
Table 1: Deviations from the published configuration, applied uniformly across all three variants. Sections 4.2.1 and 4.2.2 justify the sampling reductions empirically.
B4
M
R
C
CoCap, published [ 12 ]
31.4
23.2
49.4
52.7
Our reproduction
29.7
23.4
48.9
54.9
Table 2: Our CoCap reproduction against the published VATEX result. The evaluation splits differ (Section 4.2 ), so this indicates fidelity rather than being a like-for-like comparison. Two of the four metrics run slightly above the published values and two slightly below, despite one fifth the training data; we attribute the differences to the evaluation subset rather than to any improvement.
Encoder
Decoder
B4
M
R
C
CLIP
BERT-style
29.67
23.40
48.92
54.85
SigLIP2
BERT-style
31.01
24.06
49.43
59.33
SigLIP2
GPT-2
29.71
23.63
48.75
56.56
Table 3: Attribution over the two substitutions, under identical data, sampling budget and schedule. Each row reports the epoch of highest validation CIDEr. Single seed throughout; see Section 6 .
Encoder
Decoder
Best C
Epoch
Final C
CLIP
BERT-style
54.85
11
54.85
SigLIP2
BERT-style
59.33
9
59.11
SigLIP2
GPT-2
56.56
2
45.30
Table 4: Training stability. The shallow decoder holds its peak through to the end of training; the pretrained decoder reaches its peak at epoch 2 and loses 11.3 CIDEr over the nine epochs that follow.
Encoder
Decoder
Visual
Decode
Total
CLIP
BERT-style
50.6
184.3
249.2
SigLIP2
BERT-style
50.6
173.4
222.9
SigLIP2
GPT-2
55.0
400.1
458.7
Table 5: Per-clip inference latency in milliseconds (median of 50 runs, batch size one, RTX 4070 Laptop), measured on the same hardware as the accuracy results. Visual encoding and caption decoding are reported separately because the two substitutions affect them differently.
Ground truth : a man repelling down a side of a mountain covered in snow
CLIP+BERT
a man is using a shovel to clean up a snow covered area
SigLIP2+BERT
a man is using a shovel to pull a large snow covered mountain
SigLIP2+GPT-2 (ep. 2)
A man is shown using a rope to climb up a large rock
SigLIP2+GPT-2 (ep. 11)
A man is holding onto a rope and floating in the snow
Ground truth : A group of men are standing on a theater stage and preparing to do an acting scene
CLIP+BERT
a man is talking to a group of people in a room
Table 6: Generated captions, each configuration at its best epoch, with the GPT-2 configuration also shown after overfitting. Detokenisation artefacts are removed for readability.
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
Zihan Lin, Songhe Deng, Shuwei He +6
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University
CLIP models learn transferable multi-modal features via image-text contrastive learning on internet-scale data. They are widely used in zero-shot classification, multi-modal retrieval, text-to-image diffusion, and as image encoders in large vision-language models. However, CLIP's pretraining is dominated by images paired with short captions, biasing the model toward encoding simple descriptions of salient objects and leading to coarse alignment on complex scenes and dense descriptions. While recent work mitigates this by fine-tuning on small-scale long-caption datasets, we identify an important common bias: both human- and LLM-generated long captions typically begin with a one-sentence summary followed by a detailed description. We show that this acts as a shortcut during training, concentrating attention on the opening sentence and early tokens and weakening alignment over the rest of the caption. To resolve this, we introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions. DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations. It is a drop-in replacement for Long-CLIP with no additional trainable parameters.
Marc-Antoine Lavoie, Anas Mahmoud, Aldo Zaimi +2
University of Toronto Robotics Institute · Mila - Quebec AI Institute
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.