Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment. To bridge this gap, we propose EVA-Cap, a novel framework that decomposes captions into atomic audiovisual events and transforms unstructured text supervision into event-centric alignment. We train EVA-Cap via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on a self-curated dataset, EVA-Data, which is built with event-centric quality control. During the GRPO phase, the model is explicitly optimized against three event-centric alignment targets: global event completeness, intra-event attribution, and inter-event synchronization. Extensive experiments demonstrate that EVA-Cap significantly outperforms existing methods in event-centric evaluation (e.g., +9.0% average gain on ChronusAV), while achieving competitive performance on general audiovisual captioning benchmarks (e.g., +3.3% on WorldSense).
Figures & tables
Figure 1: Illustration of the event-centric representation of audiovisual captions, where aligned audiovisual events are composed into the predicted caption.
Figure 2: Illustration of the designed data annotation pipeline, comprising three stages: initialization, refinement and event-centric quality control. Raw videos are transformed into audiovisual captions with high-fidelity audiovisual synchronization and event attributions.
Figure 3: Statistics of EVA-Data. (a) Video category distribution and (b) Video duration distribution demonstrate the diversity of this dataset, while (c) Event count distribution shows comparable counts of audio and visual events, indicating no modality bias in annotated captions.
Figure 4: Illustration of the event-centric alignment with GRPO for EVA-Cap. We introduce event-centric rewards during GRPO training, focusing on three key dimensions: Global Completeness (event recall), Intra-Event Attribution (character matching), and Inter-Event Synchronization (cross-modal retrieval between audio and visual events).
Model
Size
video-SALMONN-2 testset
UGC-VideoCap
Miss.↓
Hall.↓
Total↓
Audio↑
Visual↑
Detail↑
Avg.↑
Gemini-2.5-Pro [ 8 ]
-
18.1
13.3
31.3
69.5
74.7
73.7
72.6
Gemini-2.5-Flash [ 8 ]
-
19.3
13.9
33.3
69.1
75.8
74.0
73.0
HumanOmniV2 [ 49 ]
7B
49.2
12.3
61.6
45.6
66.3
59.5
57.1
ARC-Hunyuan-Video [ 12 ]
7B
45.7
12.5
58.2
52.7
56.0
55.8
54.8
Qwen2.5-Omni [ 45 ]
7B
41.7
15.4
57.1
46.9
66.1
60.0
57.7
Table 1: Performance comparison on general audiovisual video captioning benchmarks. Following previous works [ 4 , 5 ] , we adopt GPT-4.1 [ 1 ] as the judge model for the video-SALMONN-2 testset, and GPT-4o [ 17 ] for UGC-VideoCap evaluation.
Model
Size
Daily- Omni
World- Sense
Gemini-2.5-Pro [ 8 ]
-
60.2
33.8
Gemini-2.5-Flash [ 8 ]
-
55.3
31.0
HumanOmniV2 [ 49 ]
7B
8.2
6.6
ARC-Hunyuan-Video [ 12 ]
7B
8.6
8.7
MiniCPM-o-2.6 [ 52 ]
8B
9.8
7.2
Qwen2.5-Omni [ 45 ]
7B
12.4
8.6
Table 2: Performance comparison on QA-cascaded benchmarks, with Gemini-2.5-Pro serving as the judge.
Model
Size
V2A
A2V
AV-Speaker
M
Acc
Score
M
Acc
Score
Acc
Gemini-2.5-Pro [ 8 ]
-
2.4
61.2
2.4
1.2
61.0
2.4
49.1
TimeChat [ 51 ]
7B
1.2
31.6
1.2
0.7
30.1
1.2
23.7
AVoCaDO [ 4 ]
7B
1.8
40.9
1.6
1.1
44.9
1.8
26.8
DiaDem [ 5 ]
7B
1.8
45.9
1.7
1.1
47.7
1.9
32.7
EVA-Cap
7B
2.4
55.4
2.2
1.2
56.2
2.2
40.1
Table 3: Performance comparison on event-centric evaluation.
Setting
Reward
V2A
A2V
AV-Speaker
Daily-Omni
Rcomp
Rsync
Rattr
M
Acc
Score
M
Acc
Score
Acc
Acc
Baseline
-
-
-
0.1
8.4
0.2
0.3
11.9
0.5
3.1
12.4
SFT
-
-
-
2.0
44.8
1.7
1.0
46.1
1.8
31.8
50.1
SFT → GRPO stage1
✓
-
-
2.2
53.3
2.0
1.2
52.9
2.0
33.9
53.8
SFT → GRPO stage1
✓
✓
-
2.3
53.8
2.1
1.2
55.0
2.1
37.0
55.0
SFT → GRPO stage1
✓
✓
✓
2.1
47.9
1.8
1.1
50.8
1.9
35.6
52.9
Table 4: Ablation study on the post-training stages and event-centric reward mechanisms of EVA-Cap. The two-stage GRPO training setting we adopted is marked in light gray . All GRPO stage1 settings are derived from the SFT model, whereas GRPO stage 2 settings build upon the marked stage1.
Figure 5: Case analysis of EVA-Cap and AVoCaDO, with features relevant to audiovisual synchronization in bold, and accurate event attribution underlined. Incorrect event-centric alignments are marked in red , and correct ones in green .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
V2A
A2V
AV-Speaker
Daily-Omni
M
Acc
Score
M
Acc
Score
Acc
Acc
ASID-1M (63K)
0.9
15.2
0.8
0.9
30.7
1.8
34.1
46.5
EVA-Data
1.2
20.5
1.2
1.1
35.6
2.1
39.6
50.1
Appendix
Table 5: Ablation experiments on different SFT dataset.
Setting
V2A
A2V
AV-Speaker
Daily-Omni
M
Acc
Score
M
Acc
Score
Acc
Acc
ASID baseline
1.1
16.8
1.0
0.9
30.4
1.8
38.4
48.5
ASID GRPO stage1
1.3
21.0
1.2
1.1
36.4
2.1
41.0
52.2
ASID GRPO stage2
1.3
22.2
1.3
1.1
39.0
2.1
42.2
53.3
Appendix
Table 6: Ablation experiments on event-centric reward framework with public SFT model and data.
Setting
Reward LLM
V2A
A2V
Daily-Omni
M
Acc
Score
M
Acc
Score
Acc
SFT
-
1.2
20.5
1.2
1.1
35.6
2.1
50.1
GRPO
Qwen3.5-27B
1.4
23.0
1.3
1.2
40.1
2.3
53.0
GRPO
Gemma-4-31B-it
1.4
24.2
1.4
1.2
39.9
2.3
52.2
Appendix
Table 7: Ablation experiments on different reward LLMs.
Model
[0, 700)
[700, 1000)
[1000, 2048)
DiaDem [ 5 ]
18.6
17.7
14.9
EVA-Cap
17.4
16.8
17.0
Appendix
Table 8: Hallucination comparison under different output length.
Reward
Avg tokens
Hall.↓
Hall./1k token↓
Miss.↓
Total↓
Daily-Omni↑
Recall
748.2
17.7
8.1
19.0
36.7
53.7
Recall + Precision
624.5
16.2
8.9
20.9
37.1
51.6
Appendix
Table 9: Performance comparison with event precision reward.
Figure 6: Failure cases of EVA-Cap, e.g., videos involving rapid scene transitions and multi-character scenes with off-screen speech. Incorrect descriptions are marked in red .
Figure 7: Prompt for caption refinement during annotation stage 2 with Gemini-2.5-Pro.
Figure 8: Prompt for timestamped event tuple extraction during annotation stage 3 with Gemini-2.5-Pro.
Figure 9: Prompt for event-centric data verification during annotation stage 3 with GPT-4.1.
Figure 10: Prompt for audiovisual event extraction with Qwen3.5.
Figure 11: Prompt for global event completeness reward judgement via Qwen3.5.
Figure 12: Prompt for intra-event attribution reward judgement via Qwen3.5.
Figure 13: Prompt for inter-event synchronization reward judgement via Qwen3.5.