Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video

Masoud Jalayer; Changyi Li; Yu Xiao

cs.CVAug 23, 2026

Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video

Authors: Masoud Jalayer, Changyi Li, Yu Xiao

Organizations: Dept. of Information and Communications Engineering, Aalto University, 02150 Espoo, Finland · Dept. of Electrical Engineering and Automation, Aalto University, 02150 Espoo, Finland

Abstract

Automatically analyzing hours-long egocentric video is increasingly essential for progress monitoring, quality control, and safety in logistics, construction, and manufacturing. Yet current pipelines that process short, fixed-size windows with a vision-language model (VLM) are prohibitively expensive because cost scales with the number of model calls. To reduce this cost, prior work proposes triage policies to select which windows merit a VLM invocation. However, these policies either sample uniformly or rank windows using visual features, which ironically requires the video decoding that the budget constraints are meant to avoid. We propose audio-first triage: select windows using the lightest modality, scored before any video frame is decoded, so the approach composes naturally with token compression or quantization. The novelty lies in the objective, not the representation: rather than a per-frame sound-event detector, we train the selector to trigger once per action. This objective shift improves action coverage by 4.0-10.8 percentage points across all evaluated call rates, using frozen AudioSet-pretrained features without domain-specific sound-event labels. Using fewer than half of the available calls, the triage cuts 9-20% of VLM calls at matched coverage on EPIC-KITCHENS-100 (EK-100), surpasses uniform sampling through the mid-range on Ego4D over 247 clips, and outperforms two recent visual keyframe selectors. Code, the reference implementation and every results file this manuscript reads are at https://github.com/masjalayer/PreDecoding-AcousticTriage.

Figures & tables

Figure 1: The evaluated pipeline: a frozen extractor encodes the audio once, a span-level gate emits episodes, and only those episodes reach the VLM.

Figure 2: Sample frames from the two evaluation datasets: EK-100 kitchens, and Ego4D clips annotated as manual or workplace activity.

System	What it scores	Ref.
Rules with no content score
Full scan	every window	—
Uniform	evenly spaced windows	—
Random spans	matched length statistics	—
Acoustic frontlines, thresholded
Short-term energy	frame RMS	—

Table 1: Systems compared, and the signal each one ranks windows by.

Figure 3: An example of sixty seconds of stream at the deployed operating point: decoded frames, the audio, the gate score against $\tau$ , the annotated actions, and the frames that survive selection, one call each.

Dataset	Content	Recordings	Hours	Occupied (%)
EPIC-KITCHENS-100	real daily activities	24	5.5	85.1
STARSS23	real indoor scenes	168	7.2	96.6
TAU-NIGENS 2020	synthesised scenes	800	12.1	99.3
DCASE 2022 synth	synthesised scenes	1200	20.0	99.1

Table 2: Occupancy of each dataset: the fraction of four-second windows holding at least one annotated event.

Baseline	Mean $\Delta$ (pp)	Median $\Delta$	W/L/T	$p$
EK-100 , 24 recordings
Uniform sampling	+5.8	+5.2	22/2/0	${<}10^{-4}$
Rank only (no spread)	+5.7	+6.3	20/3/1	${<}10^{-4}$
AKS [ 1 ]	+5.6	+5.6	20/1/3	${<}10^{-4}$
BOLT [ 2 ]	+3.2	+2.3	16/7/1	$0.019$
Ego4D , 247 clips

Table 3: Paired comparison at

25\%

of calls, on both datasets. W/L/T counts recordings or clips won, lost and tied.

Feature extractor (our span-level head)	Recall (%)	Reduction (%)	Seed range
PANNs CNN14 [ 12 ]	94.1 $\pm$ 1.2	43.2 $\pm$ 4.7	36–51
BEATs AudioSet-Strong [ 3 ]	95.6 $\pm$ 1.0	51.6 $\pm$ 1.8	50–55
ATST-Frame [ 16 ]	92.5 $\pm$ 2.5	56.3 $\pm$ 7.1	47–66
fPaSST [ 14 ]	93.8 $\pm$ 1.5	52.6 $\pm$ 1.8	50–55
BEATs-SlowFast (this work)	93.8 $\pm$ 0.7	53.0 $\pm$ 2.6	50–57

Table 4: The same span-level head over five frozen extractors, at a

93\%

development recall target.

Figure 4: EK-100, in VLM calls, below half the grid. (a) Training objective. (b) Every frontline and selection rule on one plane. (c) The boxed span at scale. (d) Calls saved against uniform sampling at equal coverage.

Figure 5: Ego4D, in VLM calls, on the axes of Fig. 4 . (a) The selection rule isolated: score alone, spread alone, and both. (b) Every rule on one plane. (c) The boxed span at scale. (d) Calls saved against uniform sampling at equal coverage; the tightest rate is omitted, where our coverage falls below anything uniform sampling reaches on this grid.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Calls	vs.	Gain	Median	W/L/T	$p$
50%	Uniform	$-4.4$	$-3.4$	2/20/2	$2.9\times 10^{-4}$
	Rank-only	$-0.7$	$-0.9$	10/13/1	$0.693$
	AKS	$-0.5$	$-0.7$	11/13/0	$0.812$
	BOLT	$-2.9$	$-2.5$	4/16/4	$0.011$
33.5%	Uniform	$+5.2$	$+4.6$	19/3/2	$4.3\times 10^{-4}$
	Rank-only	$+5.8$	$+5.9$	21/3/0	$<10^{-4}$

Appendix

Table 5: EK-100, paired over 24 recordings. Gain is our coverage minus theirs, in points. W/L/T counts recordings won, lost and tied.

Calls	vs.	Gain	Median	W/L/T	$p$
50%	Uniform	$-3.2$	$+0.0$	56/95/96	$0.001$
	Rank-only	$-0.2$	$+0.0$	83/81/83	$0.917$
	AKS	$-1.4$	$+0.0$	63/77/107	$0.125$
	BOLT	$+0.5$	$+0.0$	82/76/89	$0.777$
33.5%	Uniform	$+1.8$	$+0.0$	94/82/71	$0.211$
	Rank-only	$+3.5$	$+0.0$	93/56/98	$8.5\times 10^{-4}$

Appendix

Table 6: Ego4D, paired over 247 clips, against question-referenced acoustic moments rather than action intervals.

Recording	Set	Act.	Dur. (s)	Mean (s)	Unif.	Ours	$\Delta$
P01_14	primary	354	1353	3.4	43.5	48.6	$+5.1$
P02_12	primary	371	1315	3.1	42.0	46.1	$+4.0$
P03_24	primary	136	822	3.9	39.0	61.0	$+22.1$
P04_31	primary	113	825	5.7	50.4	54.9	$+4.4$
P05_07	primary	111	802	3.6	49.5	55.9	$+6.3$
P08_09	primary	147	621	3.4	42.2	44.2	$+2.0$

Appendix

Table 7: All 24 recordings at

25\%

of calls. Act. is the annotated action count, Mean the mean action length.

			Calls (% of grid)
$w$ (s)	Windows	Occ. (%)	$50$	$33.5$	$25$	$15$	$10$
0.5	17402	78.9	$-8.1$	$-15.2$	$-17.0$	$-15.4$	$-12.0$
1	8703	83.0	$-11.7$	$-11.7$	$-11.5$	$-5.9$	$-3.9$
2	4354	88.0	$-9.3$	$-5.8$	$-1.9$	$-1.3$	$-0.4$
4	2181	92.1	$-1.8$	$-0.8$	$+1.5$	$+2.6$	$+1.3$
8	1094	95.5	$-0.5$	$+2.4$	$+2.4$	$+2.8$	$+1.6$

Appendix

Table 8: Window length against ranking’s coverage minus uniform sampling’s, in points, on the primary EK-100 set. The deployed setting is in bold.

Quantity	Value	Chosen by
Windowing and cost
Call window $w$	$4$ s	fixed, swept in Table 8
Gate frame rate	$25$ Hz	extractor
Chunk / hop	$10$ s / $5$ s	fixed
Span-level objective
Pooling $\tau$	$4.0$	development recordings

Appendix

Table 9: Operating points and training configuration.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

Audio Token Attention Is Predictable Before the Language Model Runs
Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token Compression Audio Understanding
The University of Texas at Austin
Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
Sep 22, 2025Geewook Kim, Minjoon SeoVLM Evaluation Audio-Visual Understanding
NAVER Cloud AI, South Korea · KAIST AI, South Korea
TRIAGE: Efficient Vision-Language-Action Inference Needs Both Reuse and Pruning
Oct 25, 2026Sooyoung Kwon, Carlee Joe-Wong, Kyunghan LeeEfficient VLA Model Inference Visual Token Pruning
Dept. of ECE, Seoul National University. · Dept. of ECE, Carnegie Mellon University.

Sep 30, 2026J/K move · Enter open · S save

Audio Token Attention Is Predictable Before the Language Model Runs

Kyoungjun Park, Yunzhe Li, Lili Qiu

The University of Texas at Austin

Audio Token Compression Audio Understanding Audio-Language Models Token Pruning LLM Inference Acceleration Token Compression

Open paper AskPDF arXiv

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at

ρ\geq .69

on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io

Abstract

Egocentric Video Understanding

Explore similar work

Audio Token Attention Is Predictable Before the Language Model Runs

Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy

TRIAGE: Efficient Vision-Language-Action Inference Needs Both Reuse and Pruning