AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment

Mingyang Wu; Kaituo Feng; Bohao Li; Kaixiong Gong; Zihao Yin; Xiangyu Yue

cs.CVOct 19, 2026

AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment

Authors: Mingyang Wu, Kaituo Feng, Bohao Li, Kaixiong Gong, Zihao Yin, Xiangyu Yue

Organizations: MMLab, CUHK · Peking University

Abstract

Dense temporal audio-visual captioning is important for multimodal video understanding and generation. However, prior work is constrained by three main limitations: (1) the lack of a dataset combining high temporal resolution, longer videos, and bidirectional alignment between time and events; (2) the reliance of most existing RL methods on checklists; and (3) the lack of a benchmark that bidirectionally evaluates event semantic correctness and temporal accuracy. To address these challenges, we propose: (1) AVTime-50K, a high-quality 50K-scale dataset with temporally precise, semantically accurate captions and longer videos; (2) AVTime, a state-of-the-art model optimized with a novel Bidirectional Temporal-Semantic Alignment (BiTSA) reward that imposes explicit bidirectional constraints between time and events; and (3) AVTime-Bench, a specialized benchmark that introduces a novel bidirectional evaluation of temporal accuracy and semantic correctness. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avtime.

Figures & tables

Figure 1: AVTime-50K construction combines adaptive audio-visual segmentation, clip-level dense captioning, and timeline assembly to produce full-video event annotations.

Figure 2: AVTime-50K statistics show (a) the source content distribution and (b) the relationship between video duration and event count.

Figure 3: BiTSA combines (a) Event-to-Time localization and (b) Time-to-Event semantic agreement, illustrated using solid outlines for reference events and dashed outlines for predictions.

Figure 4: AVTime-Bench evaluates T2E, E2T, and E2E through separate Audio and Visual branches using semantic judgments and temporal calculations on the same generated timeline.

Model	Size	A&V	TU	Omni-TVG				Omni-DVC			Omni-SC
Model	Size	A&V	TU	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	mIoU $\uparrow$	S $\uparrow$	C $\uparrow$	M $\uparrow$	B $\uparrow$	R $\uparrow$	C $\uparrow$	M $\uparrow$
Visual video LLMs
VideoChat	7B	$\times$	$\times$	2.20	0.90	0.40	3.00	0.70	0.20	0.90	0.50	9.60	0.00	8.20
VideoChatGPT	7B	$\times$	$\times$	4.90	2.00	0.90	5.00	0.70	0.10	0.90	0.40	14.00	0.90	5.90
Audio-visual omni-modal LLMs
VideoLLaMA	7B	$\checkmark$	$\times$	2.50	1.10	0.30	1.90	0.60	0.60	0.90	0.90	11.50	0.10	8.90

Table 1: Main results on LongVALE. Size denotes parameter count, A&V denotes audio-visual input, and TU marks models specialized for fine-grained temporal understanding; S, C, M, B, and R denote SODA-c, CIDEr, METEOR, BLEU-4, and ROUGE-L.

Model	Size	A&V	TU	Time-to-Event		Event-to-Time		Event-to-Event		Mean
Model	Size	A&V	TU	Audio $\uparrow$	Visual $\uparrow$	Audio $\uparrow$	Visual $\uparrow$	Audio $\uparrow$	Visual $\uparrow$	Mean
VideoChat	7B	$\times$	$\times$	0.44	1.37	0.19	0.14	3.60	2.25	1.33
PandaGPT	7B	$\checkmark$	$\times$	0.60	1.27	0.18	0.11	2.64	1.95	1.13
TimeChat	7B	$\times$	$\checkmark$	1.98	8.56	0.69	0.74	8.29	7.78	4.67
VTimeLLM	7B	$\times$	$\checkmark$	0.20	8.36	0.48	0.53	3.39	2.52	2.58
LongVALE-LLM	7B	$\checkmark$	$\checkmark$	1.74	5.35	0.61	0.62	4.57	4.27	2.86

Table 2: Main results on AVTime-Bench. We report Time-to-Event, Event-to-Time, and Event-to-Event performance under Audio and Visual branches. Size, A&V, and TU follow Table 1 . All scores are on a 0–100 scale; Mean is their arithmetic average.

Figure 5: Ablation study on two benchmarks. Panels (a)–(c) report LongVALE scores; (d) is the mean over audio/visual T2E, E2T, and E2E on AVTime-Bench. The SFT pair compares supervision without and with AVTime-50K. The three RL variants independently start from AVTime-SFT, using T2E only, E2T only, or both (BiTSA). Complete per-diagnostic results appear in Table 18 . Bars start at zero with panel-specific scales; higher is better.

Figure 6: BiTSA sensitivity. Mean paired reward change (

n=16

; shading:

\pm 1

SE). Solid: temporal shift. Dashed: unshifted paraphrase control. Rewards lie in

[0,1]

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
ASR minimum sentence pause / hard pause	0.24 / 0.42
Scene-change threshold / minimum scene length	24 / 12 frames
Acoustic window / hop	1.2 / 0.3
Acoustic onset / offset threshold	0.45 / 0.30
Minimum acoustic-event duration	0.6
Boundary merge tolerance	0.20

Appendix

Table 3: Adaptive segmentation settings. Times are in seconds unless otherwise indicated.

Setting	SFT	GSPO
Initialization	Qwen3-Omni-Instruct	AVTime-SFT
Training data	AVTime-50K + LongVALE	AVTime-50K
Global batch size	128	512
Learning rate	$1\times 10^{-4}$	$5\times 10^{-5}$
TP / EP / PP / CP	2 / 4 / 2 / 2	2 / 4 / 2 / 2
Numerical precision	bfloat16	bfloat16

Appendix

Table 4: AVTime-30B training configuration. TP, EP, PP, and CP denote tensor, expert, pipeline, and context parallelism.

1	Sample eight stratified time probes $T$ ; draw $\min(4,\|Y\|)$ reference events $E$ uniformly without replacement.
2	Parse each $\hat{Y}_{g}$ into event captions and normalized intervals.
3	For each valid candidate and $t\in T$ , collect active reference and predicted events.
4	Assemble caption-pair comparisons for these active sets and event-matching requests for $E$ .
5	Query the semantic judge for the candidate group; recover pair scores and matched prediction IDs.
6	For each invalid or empty candidate, set $r_{g}=-1$ .
7	For each valid candidate:

Appendix

Figure 7: BiTSA reward procedure. A nonempty probe has an event on at least one side. Missing semantic responses use the fallback described above.

Panel of Figure 8 — Figure 8: Frames from the first two clips: preparing food, placing a dish, and a decorated table with changing title text.

Event	Caption
1	A logo appears on a pink background with a red reel icon and company text below it.
2	A person with tattoos and a silver bracelet adds herbs to a plate of food in a modern kitchen, then carries the dish as introductory text appears on screen.
3	A person in a dark apron places a plate of food on a wooden table as the introductory text transitions to ’Cordially Invites You’.
4	The camera pans across a wooden table adorned with lit candles and a floral centerpiece while the text ’To Experience A New Kind Of’ appears.

Appendix

Table 5: The first four event annotations of the example video, with specific names omitted for presentation.

Event	Clip offset (s)	Local interval (s)	Global interval (s)	Buckets
1	0.00	0.00–1.70	0.00–1.70	00–01
2	0.00	1.70–8.56	1.70–8.56	01–10
3	8.56	0.00–3.00	8.56–11.56	10–13
4	8.56	3.00–6.48	11.56–15.04	13–17

Appendix

Table 6: Equivalent local and global event coordinates and serialized percentage buckets. Clip-local coordinates subtract the clip offset from the saved global timestamps.

Statistic	Minimum	25th pct.	Median	75th pct.	Maximum
Video duration (min)	0.92	2.08	2.71	4.44	11.52
Events per video	12	22	30	44	79
Events per minute	2.23	8.84	10.63	12.36	17.87

Appendix

Table 7: Video and event statistics of AVTime-Bench. Quartiles summarize the distribution across videos.

Modality	Facet	Events	Percentage
Audio	Speech	6,035	58.34%
	Human vocalization	684	6.61%
	Music	622	6.01%
	Sound events	626	6.05%
	Ambience	73	0.71%
Visual	Entities	10,074	97.39%

Appendix

Table 8: Reference-caption facets. Percentages use all 10,344 events as the denominator; multiple labels may apply to one event.

Query / relation	Audio	Visual
T2E	7,059	10,344
E2T	7,059	10,344
E2E	4,919	10,044
E2E: before	0	0
E2E: overlap	4,919	10,043
E2E: after	0	1

Appendix

Table 9: Query counts and E2E reference relations. The benchmark contains 49,769 queries in total.

Video duration $D$ (minutes)
Group	Model	$N$	T2E		E2T		E2E		Mean
			A	V	A	V	A	V
$D<2$	Base	61	1.33	14.06	3.74	4.42	12.81	21.28	9.61
	AVTime		13.85	43.46	31.63	34.01	62.02	66.25	41.87
$2\leq D<4$	Base	145	0.84	12.14	2.44	2.73	18.80	17.52	9.08
	AVTime		15.40	42.53	23.40	24.39	59.65	57.76	37.19

Appendix

Table 10: Stratified AVTime-Bench results.

N

is the number of videos; A/V denote Audio/Visual. All scores are on the same 0–100 scale as the main results.

Duration	$N_{A}$	$N_{V}$	Base		AVTime
			Audio	Visual	Audio	Visual
$t<5$ s	1,206	2,241	2.41	3.10	20.13	20.83
$5\leq t<10$ s	3,015	4,580	2.14	2.17	18.91	19.89
$t\geq 10$ s	2,838	3,523	2.14	2.40	15.11	16.98

Appendix

Table 11: E2T scores by reference-event duration.

N_{A}

and

N_{V}

are audio and visual query counts.

Query-level
Metric	$N$	Spearman $\rho\uparrow$	95% CI for $\rho$	MAE $\downarrow$	95% CI for MAE
T2E	333	0.6630	[0.6026, 0.7179]	0.1168	[0.0896, 0.1429]
E2T	333	0.7517	[0.6418, 0.8393]	0.0450	[0.0274, 0.0635]
E2E	333	0.6423	[0.5745, 0.7014]	0.1675	[0.1254, 0.2103]
Video-level (sampled)
T2E	50	0.8298	[0.6911, 0.9066]	0.0653	[0.0484, 0.0822]

Appendix

Table 12: Alignment of AVTime-Bench with human judgments. Audio and visual queries are pooled.

N

counts paired queries in the upper block and videos in the lower block. CI denotes a 95% confidence interval; MAE uses the

[0,1]

scale.

Panel of Figure 9 — Figure 9: Frames around the reference emergence event (29.33–37.33 seconds).

Source	Time (%)	Event description
Reference	11–13	A prairie dog emerges from a dirt burrow, its head raised and eyes wide with caution, while the narrator states that neighbors are on high alert.
AVTime-30B-RL	11–13	A prairie dog emerges from a burrow in a grassy field, its head and shoulders visible above the dirt mound as it looks around with its mouth slightly open.
LongVALE-SFT	07–09	A prairie dog, its fur a mix of brown and gray, stands alert in the grassy field, its gaze fixed on something in the distance as a man’s voice narrates, "Her neighbors are on high alert."

Appendix

Table 13: Localizing a visual action. Selected event descriptions and their time intervals.

Panel of Figure 10 — Figure 10: Frames during the reference interview question (75.89–86.73 seconds).

Source	Time (%)	Event description
Reference	42–47	A woman with long brown hair speaks into a large studio microphone while gesturing with her hands, as on-screen text identifies the show ’Terms of Service with Clare Duffy’ and she asks, ’So you are a practicing doctor but you are also a data scientist. Which of those interests came first for you?’
AVTime-30B-RL	42–44	The camera cuts to a close-up of the woman as she continues speaking, ’So you are a practicing doctor, but you are also a data scientist. Which of those interests came first for you?’
	44–47	A woman with long brown hair speaks into a large studio microphone against a dark background, gesturing with her hands as she asks, ’So you are a practicing doctor, but you are also a data scientist. Which of those interests came first for you?’
LongVALE-SFT	00–40	A woman with long, wavy hair, wearing a brown top, sits in front of a microphone, speaking directly to the camera about the importance of early disease diagnosis and the potential of AI in medical imaging.

Appendix

Table 14: Recovering a spoken event across fragments. Selected event descriptions and their time intervals.

Panel of Figure 11 — Figure 11: The reference dialogue occurs in a cave; AVTime assigns it to a later desert scene.

Source	Time (%)	Event description
Reference	04–08	A blue-skinned giant with gold jewelry emerges from swirling purple smoke in a cave, gesturing emphatically while correcting a man in a red vest: ’I am not a giant. I am a genie. There is a difference.’
AVTime-30B-RL	10–12	A blue genie with gold jewelry and a red sash stands in a desert, gesturing toward a person in a red vest and white shirt as he says, ’I am not a giant. I am a genie. There is a difference.’
LongVALE-SFT	04–05, 06–08	A man with a blue face and a gold necklace, illuminated by a bright light, stands in a dark, rocky landscape, his voice booming as he corrects the man from before, "I am not a giant. I am a genie. There is a difference."

Appendix

Table 15: Correct dialogue at an incorrect time. Selected event descriptions and their time intervals.

Figure 12: Visual context at the queried time in the subway sequence.

Source	Time (%)	Event description
Reference	19–28	A woman with curly blonde hair in a pink jacket stands in a dimly lit subway station, her expression shifting between concern and alarm as the loud rumble of an approaching train echoes through the cavernous space.
AVTime-30B-RL	22–25	The woman turns her head sharply to the left, her face contorting with alarm as she continues to stare off-screen, her expression shifting from shock to fear.

Appendix

Table 16: An audio omission in an otherwise detailed visual description.

Panel of Figure 13 — Figure 13: Frames at 39.5, 43.5, and 54.0 seconds in the telephone sequence.

Source	Time (%)	Event description
Reference A	19–21	A man in the driver’s seat of a car looks down at his phone, then lifts his head and says ’Hey’ into the device as he focuses on the screen.
Reference B	21–24	A woman on a rooftop adjusts her camera on a tripod while saying, ’I made the call to Gia, she immediately freaked out and called her partner in crime.’
AVTime A	20–23	A man in a car speaks on a phone, saying, ’Hey,’ as the camera cuts between his face and the woman on the rooftop who continues to aim her camera.
AVTime B	27–29	A woman with long blonde hair stands on a rooftop, holding a large camera with a telephoto lens while speaking urgently into a phone, her expression tense as she says, ’I made the call to Gia, she immediately freaked out and called her partner in crime. Have you seen Luke pick up?’

Appendix

Table 17: Preserved dialogue with an incorrect temporal relation. A and B identify the matched events.

(a) LongVALE: all 11 metrics
Variant	AVTime data	BiTSA reward		Omni-TVG				Omni-DVC			Omni-SC
Variant	AVTime data	T2E	E2T	[email protected]	[email protected]	[email protected]	mIoU	S	C	M	B	R	C	M
LongVALE-SFT	$\times$	$\times$	$\times$	15.42	9.18	4.15	11.26	2.85	6.92	3.90	5.12	16.85	17.40	8.15
AVTime-SFT	$\checkmark$	$\times$	$\times$	20.81	13.27	5.95	14.46	3.81	9.79	5.42	7.08	22.58	23.40	11.14
AVTime-T2E	$\checkmark$	$\checkmark$	$\times$	21.60	13.40	6.81	16.45	4.12	10.04	5.75	7.37	23.59	30.88	11.24
AVTime-E2T	$\checkmark$	$\times$	$\checkmark$	21.30	14.74	7.60	16.83	4.05	10.99	5.98	7.83	23.83	30.81	12.22

Appendix

Table 18: Complete ablation results on two benchmarks. All variants use Qwen3-Omni-30B-A3B-Instruct. S, C, M, B, and R denote SODA-c, CIDEr, METEOR, BLEU-4, and ROUGE-L; A/V denote Audio/Visual. All metrics are higher-is-better. Best and second-best scores are bold and underlined.

Figure 14: Complete ablation comparison. Panels (a)–(k) show all 11 LongVALE metrics; (l)–(q) show Audio/Visual T2E, E2T, and E2E on AVTime-Bench, and (r) reports their mean. T2E only, E2T only, and BiTSA (both) are independent RL runs initialized from AVTime-SFT. Bars follow the legend order from left to right in every panel. Bars start at zero with panel-specific scales. Higher is better; bold values mark the best score in each panel.

Dense Video Captioning

See all 9 papers

Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning
Jul 2, 2026Chen Zhao, Jiajun Ma, Qilong Huang +6Audio-Visual Understanding Video Captioning
Meituan, China · Nanjing University, State Key Laboratory for Novel Software Technology, China
AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward
Aug 7, 2026Mingyang Wu, Kaituo Feng, Bohao Li +3Audio-Visual Understanding Group Relative Policy Optimization
MMLab, CUHK · Peking University
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
Feb 9, 2026Linli Yao, Yuancheng Wei, Yaojie Zhang +12Audio-Visual Understanding Video Captioning
School of Computer Science, Peking University · South China University of Technology · University of Electronic Science and Technology of China +3

Jul 2, 2026J/K move · Enter open · S save

Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning

Chen Zhao, Jiajun Ma, Qilong Huang, Tiehan Fan+5

Meituan, China · Nanjing University, State Key Laboratory for Novel Software Technology, China

Audio-Visual Understanding Video Captioning Audio-Visual Alignment Temporal Video Understanding

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) have advanced video understanding, achieving precise temporal and cross-modal alignment in audiovisual video captioning remains a formidable challenge. Most existing approaches suffer from modality detachment and temporal incoherence, failing to accurately bind auditory events to visual entities or capture complex causal dynamics. To address these deficiencies, we propose TCA-Captioner, a framework specifically engineered to enhance Temporal and Cross-Modal Alignment for audiovisual video captioning. We first introduce the Observer-Checker-Corrector (OCC) framework, an iterative refinement strategy that generates high-fidelity, meticulously grounded training data. Leveraging a curated high-density human interaction dataset, TCA-Captioner is optimized to model sophisticated audiovisual interactions. Furthermore, we present TCA-Bench, a diagnostic benchmark utilizing a Decoupled Evaluation Protocol to isolate and quantify model proficiency in audiovisual binding and temporal relational reasoning. Extensive experiments demonstrate that TCA-Captioner sets a new standard for temporally-coherent and synchronized audiovisual narratives.

Abstract

Dense Video Captioning

Explore similar work

Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning

AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions