Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions

Subhashini Venugopalan; Jenny Tan; Taylor Roper; Jimmy Tobin; Anton Kast; Alicia Martin; Sam Sepah; Amy Pavel

cs.CVOct 29, 2026

Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions

Authors: Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel

Organizations: Google Research · UC Berkeley

Abstract

Audio descriptions (AD) are essential for making visual media accessible to blind and low-vision (BLV) users. While Multimodal Large Language Models (MLLMs) offer a scalable solution for generating audio descriptions on-demand, their performance on ``inline'' audio descriptions -- which must fit within existing silences in a video -- remains under-explored, particularly for diverse user-generated content (UGC). We present a comprehensive investigation into MLLM-generated inline audio descriptions. First, we introduce MTG-AD, a professionally annotated dataset of 36 hours of videos, each averaging $\sim$ 5 mins. and 8k+ descriptions in total, co-created by paired blind and sighted AD professionals to provide a high-quality benchmark for this task. Second, we propose MTG-AD-Eval, an evaluation framework based on audio description expert guidelines that provides fine-grained, reference-free actionable metrics for model improvement. Crucially, our framework evaluates the end-to-end task: it renders the generated scripts into audio tracks to assess their fit within the original video's natural silences, minimizing disruptive overlaps. Our analysis across frontier MLLMs reveals that while models accurately describe visual events and substantially reduce speech overlaps, gaps persist in narrative pacing and flow. Finally, we empirically validate that our automated and reference-free metrics correlate significantly with human quality ratings from sighted raters and 7 BLV participants, finding that while users narrowly prefer human-authored audio descriptions, they find MLLM-generated audio descriptions highly beneficial for comprehension. We provide actionable system design guidance and release MTG-AD at https://hf.co/datasets/google/mind-the-gap-inline-ad and MTG-AD-Eval at https://github.com/google/mtg-ad.

Figures & tables

Figure 1: Temporal Constraints of Inline Audio Descriptions. Inline AD must fit precisely into natural silences within a video’s narration (purple). Both the professional “Gold” human AD from MTG-AD (green) and our model-generated “AutoAD” (orange) fit within these tight temporal gaps without overlapping speech. While AutoAD generates vivid, coherent descriptions with timing-aware prompts, the professional AD achieves stronger narrative flow by capturing camera perspectives ( e.g. , “drone footage”).

Dataset	Video Domain	Total Duration	Avg. Clip / Video Len.	Source of AD	# AD Segments	Speech / Audio	Eval. Scope
M-VAD [ 27 ]	Movies	84.6 hrs (92 movies)	6.2 s	Professionals (DVS)	48,986	Mixed	Segment (gaps)
MAD [ 29 ]	Movies (CLIP feats.)	1,207.3 hrs (650 movies)	4.1 s	Professionals (AudioVault)	384,600	Mixed	Segment (gaps)
MAD-v2 [ 8 ]	Movies (CLIP feats.)	892 hrs (488 movies)	${\sim}4.1$ s	Professionals (AudioVault)	264,000	Mixed	Segment (gaps)
Movie101v2 [ 35 ]	Movies (ZH/EN)	353 hrs (203 movies)	11–20 s	Professionals	71,000	Non-dialogue	Segment (gaps)
CMD-AD [ 9 ]	Movie Clips	477 hrs (1,432 movies)	${\sim}2$ min	Professionals (AudioVault)	${\sim}101,000$	Mixed	Segment (gaps)
TV-AD [ 34 ]	TV Series	251 hrs (426 episodes)	2.1 s	Professionals (AudioVault)	34,013	Mixed	Segment (gaps)

Table 1: Comparison of Audio Description Datasets. Prior movie and TV benchmarks evaluate isolated silent intervals using pre-trimmed clips or pre-extracted visual features, while prior user-generated or instructional datasets rely on volunteer narrations, short clips with ignored audio, or synthetic ASR adaptations. MTG-AD provides full-length user-generated videos co-annotated by paired sighted and blind AD professionals and evaluates end-to-end inline AD across each entire video.

Figure 2: Dataset Statistics. MTG-AD comprises 433 professionally annotated videos ( ${\sim}36$ hours): (a) video durations center around 5 minutes; (b) human-annotated AD segments per minute; (c) vlogs form the largest category alongside travel, nature, and education; both (d) speech density and (e) silence-gap distribution illustrate the tight pauses into which inline AD must fit.

Figure 3: Prompt strategy comparison on content and temporal metrics ( Sec. 4 ) for Gemini-2.5 Flash. Content (first two panels; higher is better): guideline prompts improve visual prioritization and on-screen text recall over Simple AD . Temporal precision (last three panels; lower is better): Simple AD leaves long silences, whereas Guideline only overfills gaps and overlaps speech; Gd. + Timing ASR , which adds the timestamped transcript and per-gap word budgets, best balances content against timing.

Metric (guideline)	Question it answers $\rightarrow$ computation	Scorer	Ref-free	Better
Content metrics
Prioritization (G1.1)	Is AD spent on salient visual events? $\rightarrow$ Judge scores each segment as $c_{i}$ : covered (1), partly (0.5), or not (0) for support in $\mathcal{D}$ ; mean $\bar{c}$ .	LLM	✓	$\uparrow$ $[0,1]$
No Redundancy (G1.2)	Does AD avoid restating audible cues? $\rightarrow$ Judge scores each segment’s overlap with nearby transcript or explicit sound cues, $r_{i}\in[0,1]$ ; $1-\bar{r}$ .	LLM	✓	$\uparrow$ $[0,1]$
On-Screen Text (G1.3)	Is on-screen text conveyed? $\rightarrow$ Fraction of on-screen text words from $\mathcal{D}$ in the combined AD and ASR vocabulary.	Prog.	✓	$\uparrow$ $[0,1]$
Temporal metrics
Timeliness (G2)	Is each description heard near its visual event? $\rightarrow$ Judge matches each AD segment to an event at $t_{i}$ in $\mathcal{D}$ ; mean $\|s_{i}-t_{i}\|$ on matches.	LLM	✓	$\downarrow$ sec.

Table 2: Evaluation metrics at a glance. Metrics are computed per video and averaged across videos. Scorer : Prog. = programmatic or word arithmetic; LLM = text-only LLM judge; MLLM = video + text judge (prompts in Appx. J ). Ref-free : runs without human AD.

\mathcal{D}

= dense visual information; TTS = AD rendered at

1.2\times

Figure 4: Frontier model comparison on content (G1.1, G1.3; higher is better) and temporal metrics (G2, G7, Avg. Silence; lower is better), all with the Gd. + Timing ASR prompt. Bars show means with 95% CIs over the 101 test videos. Models marked † : G1.1 and G2 scored by a Gemini-3.5 Flash judge (others: Gemini-2.5 Flash). Tabs. 5 and 6 show detailed scores.

Figure 5: Qualitative alignment ( VideoA11y-Ref ) ratings on the same videos as Fig. 4 . Because the judge compares against the human AD, Accurate here measures agreement with that reference (Reference Alignment). Models marked † were scored by a Gemini-3.5 Flash judge and the others by Gemini-2.5 Flash, so scores should be compared only within each group.

Panel of Figure 6 — Figure 6: Judge Alignment and Bias. (a, left) Inter-judge Pearson correlation across 101 test videos. Each score is the mean across five metrics Prioritization (G1.1) and VideoA11y-Ref rubrics; GPT-4.1 mini and Qwen-2-VL-Max align most closely. (b, right) Deviation from model-specific mean scores. Shows systematic bias of each judge by calculating the mean deviation from the global average score per video. Positive values (green) indicate "lenient" judging, while negative values (pink) indicate "harsh" judging relative to the consensus.

Panel of Figure 7 — Figure 7: YouDescribe generalization. (a): Mean values (with 95% CI) for Prioritization, On-Screen Text coverage, Timeliness, and Avg. Silence across 2 available human AD sources and our automated AD on the YouDescribe dataset. (b) Distribution of available silence gaps (right) shows significantly longer gaps compared to our dataset due to the source videos being largely silent or lacking pre-existing dialogue.

Figure 8: Sighted rater model comparisons . Mean Likert scores across five (of 11) key dimensions on videos evaluated by sighted raters ( $\uparrow$ indicates higher is better). Gemini-2.5 Flash approaches Ground Truth on Accuracy, Timeliness and script Quality, while Claude 4 has fewer Audio Issues due to lower speech coverage.

Category	Automated Metric	Cohort	Target Human Dimension	Spearman $\rho$	$p$ -value
Reference-Free Guideline Metrics (G1, G2, G7, G8)	Audio Overlaps (G7)	Sighted ( $n=42$ )	Audio Issues (No Overlaps)	$\mathbf{-0.467}$	$\mathbf{0.0018^{**}}$
	Avg. Silence (G8)	Sighted ( $n=42$ )	Gaps of Silence (Few Quiet Gaps)	$\mathbf{-0.415}$	$\mathbf{0.0063^{**}}$
	Timeliness (G2)	Sighted ( $n=42$ )	Timeliness	$-0.228$	$0.1470$
	Prioritization (G1.1)	Sighted ( $n=42$ )	Coverage	$+0.246$	$0.1160$
	No Redundancy (G1.2)	BLV ( $n=28$ )	Consistency / Non-Redundancy	$\mathbf{+0.386}$	$\mathbf{0.0427^{*}}$
	Audio Overlaps (G7)	BLV ( $n=28$ )	Audio Quality (truncated in UI)	$+0.132$	$0.5024$

Table 3: Empirical validation of automated metrics against Sighted (

n=42

) and BLV (

n=28

) human judgments. Spearman rank correlations (

\rho

) and

p

-values across our Reference-Free Guideline Metrics ( G1–G8 ), Reference-Based VideoA11y-Ref , and Reference-Free Quality-RF ( video + ASR ). Full qualitative interpretations and 4-variant ablations are in Appx. E .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Simple AD	Guideline only	Guideline + ASR	Gd. + Timing ASR	Human AD
Prioritization, G1.1 ( $\uparrow$ )	0.75 $\pm$ 0.02	0.79 $\pm$ 0.02	0.79 $\pm$ 0.02	0.76 $\pm$ 0.02	0.73 $\pm$ 0.02
No Redundancy, G1.2 ( $\uparrow$ )	0.77 $\pm$ 0.03	0.72 $\pm$ 0.02	0.71 $\pm$ 0.02	0.69 $\pm$ 0.03	0.72 $\pm$ 0.02
On-Screen Text, G1.3 ( $\uparrow$ )	0.47 $\pm$ 0.03	0.69 $\pm$ 0.03	0.65 $\pm$ 0.03	0.63 $\pm$ 0.03	0.61 $\pm$ 0.03
Timeliness, G2 (s) ( $\downarrow$ )	6.46 $\pm$ 3.03	3.27 $\pm$ 1.29	5.11 $\pm$ 2.07	4.43 $\pm$ 1.17	2.90 $\pm$ 0.48
Audio Overlaps, G7 (s) ( $\downarrow$ )	15.07 $\pm$ 3.01	59.19 $\pm$ 7.28	21.34 $\pm$ 4.64	12.77 $\pm$ 3.88	7.51 $\pm$ 1.42
Audio Coverage, G8 (ratio, $\leq$ 1)	0.41 $\pm$ 0.06	1.23 $\pm$ 0.11	0.83 $\pm$ 0.08	0.78 $\pm$ 0.05	0.61 $\pm$ 0.03

Appendix

Table 4: Prompt Comparisons . AD generated by Gemini-2.5 Flash with each prompt, and the professional Human AD, on validation videos with output from every prompt. Showing the mean and the 95% confidence interval,

\uparrow

(higher is better);

\downarrow

(lower is better). Cells are shaded by rank: First and Second .

Metric	Gemini-2.5 Flash	Qwen-2-VL-Max	GPT-4.1 mini	Claude 4	Claude 4.5	GPT-5.2
Prioritization, G1.1 ( $\uparrow$ )	0.76 $\pm$ 0.04	0.59 $\pm$ 0.05	0.52 $\pm$ 0.05	0.51 $\pm$ 0.05	0.59 $\pm$ 0.05	0.55 $\pm$ 0.05
No Redundancy, G1.2 ( $\uparrow$ )	0.70 $\pm$ 0.05	0.71 $\pm$ 0.04	0.70 $\pm$ 0.05	0.69 $\pm$ 0.05	0.70 $\pm$ 0.04	0.73 $\pm$ 0.04
On-Screen Text, G1.3 ( $\uparrow$ )	0.64 $\pm$ 0.06	0.59 $\pm$ 0.06	0.58 $\pm$ 0.06	0.56 $\pm$ 0.05	0.61 $\pm$ 0.05	0.63 $\pm$ 0.05
Timeliness, G2 (s) ( $\downarrow$ )	5.00 $\pm$ 3.07	6.92 $\pm$ 3.91	65.28 $\pm$ 18.59	19.83 $\pm$ 4.39	20.50 $\pm$ 4.30	22.56 $\pm$ 4.76
Audio Overlaps, G7 (s) ( $\downarrow$ )	8.92 $\pm$ 5.62	46.53 $\pm$ 11.16	21.46 $\pm$ 4.77	7.45 $\pm$ 2.14	12.86 $\pm$ 3.31	11.14 $\pm$ 2.74
Audio Coverage, G8 (ratio, $\leq$ 1)	0.74 $\pm$ 0.07	0.94 $\pm$ 0.15	1.48 $\pm$ 0.46	0.62 $\pm$ 0.06	0.70 $\pm$ 0.07	0.65 $\pm$ 0.07

Appendix

Table 5: Model Comparisons (Initial Frontier Models) . Comparing frontier models under the Gd. + Timing ASR prompt on the 101 test videos. Showing the mean and the 95% confidence interval,

\uparrow

(higher is better);

\downarrow

(lower is better). Cells are shaded by rank: First and Second .

Metric	Gemini-2.5 Flash	Gemini-3.5 Flash †	Qwen-2-VL-Max	Qwen-3.5-Omni-Flash †	Qwen-3.5-Omni-Plus †
Prioritization, G1.1 ( $\uparrow$ )	0.76 $\pm$ 0.04	0.93 $\pm$ 0.03	0.59 $\pm$ 0.05	0.94 $\pm$ 0.02	0.79 $\pm$ 0.06
No Redundancy, G1.2 ( $\uparrow$ )	0.70 $\pm$ 0.05	0.90 $\pm$ 0.03	0.71 $\pm$ 0.04	0.88 $\pm$ 0.05	0.88 $\pm$ 0.04
On-Screen Text, G1.3 ( $\uparrow$ )	0.64 $\pm$ 0.06	0.57 $\pm$ 0.05	0.59 $\pm$ 0.06	0.61 $\pm$ 0.06	0.58 $\pm$ 0.06
Timeliness, G2 (s) ( $\downarrow$ )	5.00 $\pm$ 3.07	2.99 $\pm$ 0.91	6.92 $\pm$ 3.91	4.91 $\pm$ 3.64	2.67 $\pm$ 1.04
Audio Overlaps, G7 (s) ( $\downarrow$ )	8.92 $\pm$ 5.62	0.26 $\pm$ 0.13	46.53 $\pm$ 11.16	52.14 $\pm$ 15.89	30.45 $\pm$ 8.69
Audio Coverage, G8 (ratio, $\leq$ 1)	0.74 $\pm$ 0.07	0.45 $\pm$ 0.03	0.94 $\pm$ 0.15	0.94 $\pm$ 0.20	0.72 $\pm$ 0.14

Appendix

Table 6: Next-Generation Omni-Modal Model Comparisons . Comparing Gemini-2.5/3.5 Flash and Qwen-2-VL-Max vs. Qwen-3.5-Omni-Flash/Plus under the Gd. + Timing ASR prompt. Each model is scored on all test videos it has output for: 101 for Gemini-2.5/3.5 Flash and Qwen-2-VL-Max, 99 for Qwen-3.5-Omni-Plus, and 57 for Qwen-3.5-Omni-Flash. Showing the mean and 95% confidence interval,

\uparrow

(higher is better);

\downarrow

(lower is better). For models marked with † , all LLM-judged metrics (G1.1, G1.2, G2, and VideoA11y-Ref ) used Gemini-3.5 Flash as the judge; the other models used Gemini-2.5 Flash. The segment-level metrics (G1.1, G1.2, G2) appear robust to this change, whereas the holistic VideoA11y-Ref ratings are more sensitive to it; programmatic metrics (G1.3, G7, G8) do not depend on the judge. Cells are shaded by rank: First and Second .

Metric	Krippendorff’s $\alpha$	Avg Std Dev
Timeliness	0.721	0.581
Audio Issues	0.700	0.545
Objective	0.665	0.437
Gaps of Silence	0.647	0.637
Overall Quality	0.581	0.681
Descriptive	0.579	0.699

Appendix

Table 7: Inter-Annotator Agreement (Krippendorff’s

\alpha

) and Average Standard Deviation per Metric. Overall (all metrics): 0.654

Figure 10: Sighted rater prompt comparisons . Mean Likert scores across five (of 11) key dimensions ( $\uparrow$ indicates higher is better). Sighted raters evaluated videos played with AD generated from different prompts.

Figure 12: BLV Study. In a blind study, BLV participants listening to the same video with human generated AD and AutoAD (Gemini-2.5 Flash with Gd. + Timing ASR) when asked to pick which AD version they liked better, marginally preferred the video version with human generated AD.

Automated Metric	Metric Type	Target Human Dimension	Spearman $\rho$	$p$ -value	Interpretation
Audio Overlaps (G7)	Programmatic (TTS + ASR)	Audio Issues (No Overlaps)	$\mathbf{-0.467}$	$\mathbf{0.0018^{**}}$	Moderate alignment (fewer overlap seconds $\rightarrow$ higher human audio rating)
Avg. Silence Duration (G8)	Programmatic (TTS + ASR)	Gaps of Silence (Few Quiet Gaps)	$\mathbf{-0.415}$	$\mathbf{0.0063^{**}}$	Moderate alignment (shorter silent gaps $\rightarrow$ higher human rating)
VideoA11y-Ref Clear	LLM Judge	Clarity	$\mathbf{+0.453}$	$\mathbf{0.0026^{**}}$	Moderate positive alignment
VideoA11y-Ref Descriptive	LLM Judge	Descriptive	$\mathbf{+0.328}$	$\mathbf{0.0338^{*}}$	Statistically significant positive alignment
Prioritization (G1.1)	LLM Judge (Visual Info)	Coverage	$+0.246$	$0.1160$	Positive trend
Timeliness (G2)	LLM Judge (Visual Info)	Timeliness	$-0.228$	$0.1470$	Expected negative trend (lower offset seconds $\rightarrow$ higher human rating)

Appendix

Table 8: Automated Metrics vs. Sighted Rater Judgments (

n=42

videos). Video-level Spearman rank correlations (

\rho

) and

p

-values alongside qualitative interpretations.

Automated Metric	Metric Type	Target BLV Dimension	Spearman $\rho$	$p$ -value	Interpretation / Note
VideoA11y-Ref Clear	LLM Judge (Ref-based)	Clarity	$\mathbf{+0.388}$	$\mathbf{0.0411^{*}}$	Statistically significant positive alignment
No Redundancy (G1.2)	LLM Judge (Ref-free)	Consistency / Non-Redundancy	$\mathbf{+0.386}$	$\mathbf{0.0427^{*}}$	Statistically significant positive alignment
VideoA11y-Ref Descriptive	LLM Judge (Ref-based)	Descriptive	$+0.340$	$0.0771$	Marginal positive alignment ( $p<0.10$ )
VideoA11y-Ref Objective	LLM Judge (Ref-based)	Objective	$+0.335$	$0.0811$	Marginal positive alignment ( $p<0.10$ )
Audio Overlaps (G7)	Programmatic (Ref-free)	Audio Quality	$+0.132$	$0.5024$	Low correlation because the study player truncated AD at speech onset
VideoA11y-Ref Accurate	LLM Judge (Ref-based)	Perceived Accuracy	$-0.264$	$0.1743$	Negative correlation due to reference-matching bias (resolved by Quality-RF )

Appendix

Table 9: Automated Metrics vs. BLV Listener Judgments (

n=28

video evaluations). Video-level Spearman rank correlations (

\rho

) and

p

-values.

VideoA11y Dimension	Ref-Based $\rho$	Ref-Based $p$	Ref-Free $\rho$ (Video + ASR)	Ref-Free $p$	Key Finding
Descriptive	$+0.340$	$0.0771$	$\mathbf{+0.583}$	$\mathbf{0.0011^{**}}$	+0.243 $\rho$ gain; highly significant
Clear	$\mathbf{+0.388}$	$\mathbf{0.0411^{*}}$	$\mathbf{+0.415}$	$\mathbf{0.0281^{*}}$	+0.027 $\rho$ gain; statistically significant
Objective	$+0.335$	$0.0811$	$+0.365$	$0.0561$	+0.030 $\rho$ gain; strong marginal trend
Accurate	$-0.264$	$0.1743$	$+0.244$	$0.2109$	Reverses negative correlation (+0.508 swing)

Appendix

Table 10: Reference-Free ( Quality-RF ) vs. Reference-Based ( VideoA11y-Ref ) Correlation with BLV Human Ratings (

n=28

). Replacing the human reference text with raw video and ASR transcript reverses the negative accuracy correlation and strengthens alignment across all four rubric dimensions.

Figure 13: Example demonstrating more detailed descriptions from human annotators relative to model generated description despite the tight timing constraints of the video.

Figure 14: Example demonstrating human annotators describing the perspective of the video in addition to vivid descriptions going with the flow of the narration.

Figure 15: Dataset Examples. Model Generated AD (created with the Simple AD and Gd. + Timing ASR prompts) and corresponding professionally annotated AD for a video segment. Here, we illustrate each video segment with manually selected video frames. Top example demonstrates narrative flow in the professionally annotated AD (describes woman’s story rather than just the actions), and the example below shows a missed event in the Model Generated ADs (the truck rapidly breaks for the elephant).

Panel of Figure 16 — Figure 16: Prompt used to generate detailed visual information in the video. The responses are used when evaluating the VLM and human generated descriptions.

Panel of Figure 17 — Figure 17: Most basic baseline prompt used for generating audio description.

Panel of Figure 21 — Figure 21: Prompt used to evaluate the prioritization of important events in the audio description based on detailed visual information.

Panel of Figure 22 — Figure 22: Initial version of the On-Screen Text (G1.3) metric, which used an LLM judge to count unsaid on-screen words. All reported G1.3 results instead use a programmatic word-level recall ( Sec. 4.1 ): the fraction of on-screen-text words listed in the dense visual information that appear anywhere in the AD or the ASR transcript; no LLM judge is involved.

Panel of Figure 23 — Figure 23: Prompt used to evaluate the redundancy of audio descriptions with the original transcript.

Panel of Figure 24 — Figure 24: Prompt for evaluating the timeliness of audio descriptions by comparing them to pre-extracted visual information.

Audio Description Generation

See all 10 papers

Jun 22, 2026

REFRAMED: Towards Realistic Audio Description Generation for Movies

Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.

Igor Sterner, Mirella Lapata, Alex Lascarides +1

School of Informatics University of Edinburgh United Kingdom

Audio Description Generation Video Captioning

Ask PDF

Sep 1, 2026

From Visual Cues to Spoken Narration: Rethinking Audio Description

Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The problem requires determining both what (which visual event) and when (position for inserting the AD) to narrate, to achieve the best user experience. Prior work has largely reduced the problem to video captioning of pre-segmented video clips, i.e., what is largely predefined and when is ignored entirely. We propose Cue2Narrate, a two-stage pipeline that jointly predicts what and when to narrate in longer untrimmed movie clips. A dual-head audio-visual localizer predicts two temporally distinct windows per AD utterance: a visual cue window and a spoken narration window. A LoRA-adapted VLM then generates concise ADs from the predicted visual evidence, trained with a Description Ranking Loss that ranks captions (negative samples) of the same frames lower than the GT AD. To benchmark this new problem statement, we introduce the LongLSMDC benchmark with up to 8-min movie clips (~6.5min on average). On LongLSMDC, Cue2Narrate outperforms video-only and audio-only localization baselines by 5--12 points in avg. mAP. Under both predicted- and GT-window evaluation, Cue2Narrate improves AD generation over the corresponding fine-tuned base VLM. These results establish the first benchmark for multi-segment AD generation on long-form clips. Data & Code: https://github.com/multimodal-ai-lab/Cue2Narrate

Akshita Gupta, Aditya Arora, Federico Tombari +2

TU Darmstadt & hessian.AI, Darmstadt, Germany · Google Research · TU Munich

Audio-Visual Understanding Audio Description Generation Assistive Technology for Blind and Low-Vision People

Ask PDF

Abstract

Audio Description Generation

Explore similar work

READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations

REFRAMED: Towards Realistic Audio Description Generation for Movies

From Visual Cues to Spoken Narration: Rethinking Audio Description