Video captioning aims to preserve visual information through compact descriptions, predominantly using natural language. However, precisely expressing fine-grained spatial structures and temporal dynamics in natural language often leads to lengthy and potentially ambiguous descriptions. In this work, we explore Code as Video Caption, a paradigm where executable programs serve as a complementary representation of visual content through graphical primitives, layout rules, and functions of time. To examine whether code captions effectively preserve video information, we compare them with natural language captions on downstream question answering and video reconstruction tasks. Our results show that code captions achieve superior reasoning accuracy and reconstruction fidelity. Despite these advantages, realizing the potential of code captioning depends on whether current multimodal large language models (MLLMs) can generate faithful code captions. To systematically assess this capability, we introduce CodeCap-Bench, which comprises 323 videos spanning five categories of structured 2D content, along with fine-grained annotations and a multi-level evaluation framework. Extensive experiments reveal that while leading proprietary MLLMs demonstrate promising code captioning capabilities, they still struggle to faithfully capture complex temporal dynamics, with open-source models lagging substantially behind.
Figures & tables
Figure 1: Illustration of video captioning with executable code. The reference video shows a flight seat-selection interface. (a) and (b) present natural language and code captions of the same segment, respectively. The code caption encodes spatial structure, temporal dynamics, and visual state changes through executable rules and functions.
Figure 2: The data construction pipeline of CodeCap-Bench. The pipeline comprises three stages: multi-source video collection, content filtering for structured 2D videos, and fine-grained annotation with human review, covering static and dynamic focal points and object-level spatio-temporal information.
Figure 3: Statistics of CodeCap-Bench. (a) Video counts and proportions across five categories. (b) Video duration distribution. (c) Reference objects per video. (d) Static and dynamic focal points per video.
Model
Harness
Video Production Success Rate (%) ↑
Visual similarity ↑
Object-level Spatio-temporal IoU ↑
Fine-grained Visual Information Retention
Dynamic ↑
Static ↑
Overall ↑
Agentic
GPT-6-astra
Codex
99.4
0.9735
75.37
68.61
80.72
73.24
GPT-6-astra
Pi Agent
99.7
0.9771
74.45
67.66
81.35
72.87
Opus-5
Pi Agent
97.8
0.9449
61.45
55.49
65.10
58.84
Opus-5
Claude Code
99.4
0.9555
58.55
53.23
63.40
56.88
Table 1: Main results on CodeCap-Bench. All configurations use medium thinking effort. Bold values indicate the best results within the Agentic and Non-agentic groups, respectively.
Figure 4: Average scores of four models across video categories. Across all four models, overall visual information retention is higher for desktop and application interfaces and data and scientific visualization than for visual design and media animation and icons and abstract motion graphics.
Figure 5: Visual fidelity and generation cost across different thinking effort levels using the Pi Agent harness. (a) GPT-6-sol; (b) Qwen3.8-max. The left axis shows object-level spatio-temporal IoU and fine-grained visual information retention, while the right axis shows average generation cost.
Figure 6: Generation cost and efficiency with the Pi Agent harness.
Figure 7: Tool call composition for seven models with the Pi Agent harness.
Figure 8: Scores from different judge models and their correlation with human scores . The two judge configurations yield the same overall ranking for the two evaluated generation models, and GPT-6-sol judge scores correlate strongly with human scores (Pearson r=0.942).
Figure 9: Execution trace of GPT-6-astra reconstructing a bubble chart video with the Codex harness.
Caption model
Code-rendered video QA ↑
Natural language caption QA ↑
GPT-5.6-sol
76.42
73.58
Opus-5
83.02
72.64
Qwen3.8-max
81.13
78.30
Original videos
83.96
Table 2: Question-answering accuracy (%) from rendered videos and caption text.
Caption model
Method
AS ↑
MPS ↑
KSS ↑
GPT-5.6-sol
Code rendering
0.8907
0.5412
0.8068
Caption-to-video
0.6628
0.5156
0.5925
Opus-5
Code rendering
0.9148
0.5572
0.8165
Caption-to-video
0.6932
0.5150
0.5958
Qwen3.8-max
Code rendering
0.8773
0.5488
0.7817
Caption-to-video
0.6754
0.5185
0.5984
Table 3: Video reconstruction fidelity from code and natural language captions.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Configuration
Description
Agent harness
Pi Agent 1.1.0; Codex CLI 0.155.1; Claude Code 2.1.296
Versions used for code generation
Code generation time limit
3,600 seconds
Timed separately for each video; Qwen3.8-max at xhigh thinking effort has a 7,200-second limit
Video rendering time limit
900 seconds
Time limit for a single rendering process
Video rendering frame rate
24 fps
Call the page’s seek(seconds) interface at 1/24 -second intervals along the video timeline, capture screenshots, and encode them into a video
Appendix
Table 4: Runtime configuration for the agentic code generation experiments.
Figure 10: Video generation failures for seven models with Pi Agent. Sectors show the shares of failure causes, the center shows the failure count, and the label below shows the failure rate.
Figure 11: Workspace interface. The four columns correspond to 3.0, 5.0, 11.5, and 16.0 seconds, showing a calendar view, workspace navigation, and database template previews.
Figure 12: 2D game. The four columns correspond to 3.1, 5.0, 8.0, and 13.4 seconds, showing the launch, collision, and structural collapse.
Figure 13: Character and scene animation. The four columns correspond to 7.7, 9.6, 13.3, and 15.6 seconds, showing character appearance, scene composition, and transitions.
Figure 14: Scientific visualization. The four columns correspond to 0.5, 4.3, 26.5, and 27.3 seconds, showing energy conversion diagrams, process structure, and graphical transitions.
Figure 15: Icon collections. The four columns correspond to 3.0, 15.3, 17.5, and 23.0 seconds, showing the shapes, colors, and arrangement of different icon groups.
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
Zihan Lin, Songhe Deng, Shuwei He +6
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.