Revisiting Frame-Wise Saliency for Audio Moment Retrieval

Tatsuya Komatsu; Hokuto Munakata

eess.ASOct 5, 2026

Revisiting Frame-Wise Saliency for Audio Moment Retrieval

Authors: Tatsuya Komatsu, Hokuto Munakata

Organizations: LY Corporation, Tokyo, Japan

Abstract

This paper revisits frame-wise saliency for audio moment retrieval (AMR). We show that the frame-wise saliency sequence, conventionally used only as an auxiliary output in DETR-based AMR models, can itself serve as an effective source of moment predictions. We convert the saliency sequence into ranked moments using a simple SED-inspired segmentation rule with no learned parameters, enabling moment retrieval directly from frame-wise temporal information. On the CASTELLA dataset, saliency-based prediction consistently outperforms decoder-based prediction from the same model across all 18 runs of QD-DETR and CG-DETR. For QD-DETR, simply replacing the inference output improves R1@0.7 from 21.0 to 36.1. The advantage remains 7-15 points when the two outputs are evaluated at their independently selected best epochs. The same tendency extends to TaskWeave and UVCOM, whereas TR-DETR shows the opposite behavior, suggesting that how saliency construction may matter. The performance gap is especially pronounced for short moments: for queries whose annotated moments average at most 2 s, R1@0.7 improves from 6.3 to 27.8 with QD-DETR. Decoder supervision nevertheless benefits saliency-based prediction, indicating that its role during training differs from the utility of its inference output.

Figures & tables

		R1@0.5		R1@0.7		mAP
Model	$\lambda$	sal.	dec.	sal.	dec.	sal.	dec.
QD-DETR	1	50.1 $\pm$ 1.1	40.8 $\pm$ 1.6	36.1 $\pm$ 0.8	21.0 $\pm$ 1.3	30.7 $\pm$ 0.5	17.5 $\pm$ 1.0
	4	50.3 $\pm$ 0.6	39.1 $\pm$ 2.5	36.1 $\pm$ 0.8	20.2 $\pm$ 2.7	30.5 $\pm$ 0.2	17.4 $\pm$ 1.5
	8	51.3 $\pm$ 0.5	41.9 $\pm$ 1.5	37.3 $\pm$ 1.0	21.1 $\pm$ 1.3	31.4 $\pm$ 0.4	18.5 $\pm$ 0.5
CG-DETR	1	51.2 $\pm$ 1.3	46.3 $\pm$ 2.9	36.7 $\pm$ 0.5	25.9 $\pm$ 3.7	29.7 $\pm$ 0.5	21.3 $\pm$ 2.4
	4	52.0 $\pm$ 1.0	48.2 $\pm$ 4.3	38.3 $\pm$ 0.9	27.4 $\pm$ 5.7	30.6 $\pm$ 0.4	22.4 $\pm$ 3.3

Table 1: Saliency- vs. decoder-based predictions on CASTELLA test (%, mean

\pm

SD over three seeds).

\lambda

: saliency-loss weight. sal./dec.: saliency-/decoder-based predictions.

$\lambda$	QD-DETR	CG-DETR
1	21.4	29.8
4	23.2	30.2
8	24.3	28.2

Table 2: Decoder-based test R1@0.7 (%, mean over three seeds) at epochs selected by decoder-based validation R1@0.7. Compare with the saliency-selected results in Table 1 .

	QD-DETR		CG-DETR
	$\lambda{=}1$	$\lambda{=}8$	$\lambda{=}1$	$\lambda{=}8$
Peak-relative threshold (Ours)	36.1	37.3	36.7	38.5
- w/o gap tolerance and overlap removal	30.2	32.1	31.3	33.4
Recording-wise threshold	32.5	33.0	32.7	35.4
Global absolute threshold	21.2	24.1	25.1	24.3
Fixed 5 s duration	6.1	6.2	6.0	6.5

Table 3: Saliency-based test R1@0.7 (%) under different segmentation rules, with decoder-based prediction shown for reference (mean over three seeds). Simple recording-wise thresholding retains a substantial advantage over decoder-based prediction.

	R1@0.5		R1@0.7		mAP
Model	sal.	dec.	sal.	dec.	sal.	dec.
TaskWeave (HD $\to$ MR)	51.9	37.6	36.5	19.2	29.9	17.0
UVCOM	50.5	47.6	35.9	28.2	30.2	24.2
TR-DETR	41.1	52.2	22.7	32.9	18.2	26.7

Table 4: Saliency- and decoder-based predictions for three additional DETR-based models on CASTELLA test (%, mean over three seeds). Saliency-based prediction is higher for TaskWeave and UVCOM, whereas TR-DETR shows the opposite behavior. sal./dec.: saliency-/decoder-based predictions.

R1@0.7
		1 moment	2–3	$\geq$ 4
Model	Output	( $n$ =573)	( $n$ =404)	( $n$ =370)
QD-DETR ( $\lambda=1$ )	dec.	21.8	21.3	19.3
	sal.	34.3	36.1	38.9
CG-DETR ( $\lambda=8$ )	dec.	29.7	26.7	28.5
	sal.	36.9	38.4	41.0

Table 5: Test performance (%) by the number of annotated moments (mean over three seeds;

n

= number of test queries). Both metrics use IoU 0.7; moment recall uses up to ten predictions and is averaged over queries. sal./dec.: saliency-/decoder-based predictions.

		$\leq$ 2 s	$(2,5]$ s	$>$ 5 s
Model	Output	( $n$ =290)	( $n$ =358)	( $n$ =699)
QD-DETR ( $\lambda=1$ )	dec.	6.3	15.7	29.7
	sal.	27.8	30.7	42.3
CG-DETR ( $\lambda=8$ )	dec.	12.3	19.5	39.7
	sal.	31.6	30.7	45.4

Table 6: Test R1@0.7 (%) by the mean duration of the query’s annotated moments (mean over three seeds;

n

= number of test queries). sal./dec.: saliency-/decoder-based predictions.

	R1@0.7		mAP
Model	joint	no dec.	joint	no dec.
QD-DETR	37.3 $\pm$ 1.0	34.3 $\pm$ 3.2	31.4 $\pm$ 0.4	29.0 $\pm$ 1.8
CG-DETR	38.5 $\pm$ 0.6	36.4 $\pm$ 1.9	31.4 $\pm$ 0.1	29.4 $\pm$ 1.4

Table 7: Saliency-based prediction with joint training and without decoder supervision (no dec.), retaining all other losses (

\lambda=8

; test, %, mean

\pm

SD over three seeds).

Figure 1: Example from CG-DETR (

\lambda=8

, seed 2023) on CASTELLA test, cropped to 129–168 s. Top: raw saliency scores

r_{j}

and annotated moments. Bottom: the five highest-ranked predictions of each output.

Explore similar work

Overview and Meta-Analysis of DCASE 2026 Challenge Task 6: Audio Moment Retrieval from Long Audio
Sep 11, 2026Hokuto Munakata, Tatsuya Komatsu, Keisuke Imoto +3Unsupervised Detection
LY Corporation, Japan · Kyoto University, Japan · Sony Interactive Entertainment, Japan +2
GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval
May 30, 2026Shihang Zhang, Mingjin Kuai, Ye Wei +2Detection Transformer Transformer Encoder
Shihang Zhang and Ye Wei are with the College of Electronics and Information Engineering, Sichuan University, Chengdu 610065, China · Mingjin Kuai, Zhen Zhang, and Wei Ji are with the School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China
AudioJev: Direct Audio Decisions with Order-Calibrated Probabilities
Oct 1, 2026Sihan Lv, Zhen Li, Zhiqi Cao +4Audio Understanding Audio Understanding
Zhejiang University, Hangzhou, China · China Academy of Space Technology, Beijing, China · Innovation and Management Center of the School of Software (Ningbo), Zhejiang University, Ningbo, China +2

Sep 11, 2026, eess.ASJ/K move · Enter open · S save

Overview and Meta-Analysis of DCASE 2026 Challenge Task 6: Audio Moment Retrieval from Long Audio

Hokuto Munakata, Tatsuya Komatsu, Keisuke Imoto, Taichi Nishimura+2

LY Corporation, Japan · Kyoto University, Japan · Sony Interactive Entertainment, Japan+2

Unsupervised Detection

Open paper AskPDF arXiv

This paper presents an overview of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2026 Challenge Task 6, Audio Moment Retrieval (AMR) from Long Audio. Given a several-minute-long audio recording and a free-form text query, AMR aims to retrieve temporal moments in the recording that match the query, where each moment is represented by a pair of start and end timestamps. This task requires effective cross-modal alignment and long-range temporal modeling. We describe the task definition, the evaluation metrics, the development and evaluation datasets, and a baseline system that combines a pre-trained MS-CLAP feature extractor with a Detection Transformer (DETR)-based moment-detection network. On the development data, the baseline trained on a manually annotated dataset and a synthetic dataset achieved Recall1@0.7 of 13.56%, indicating that AMR in long audio remains a challenging problem. The challenge attracted 21 teams, which submitted 59 systems in total. The three best systems achieved Recall1@0.7 of 48.59%, roughly 3.5 times the baseline score. The results show that strengthening the audio-text feature extractor and the moment-detection network led to substantial performance improvements. Furthermore, the top three teams boosted performance by applying confidence score calibration or ensembling across different temporal resolutions of features.

Abstract

Explore similar work

Overview and Meta-Analysis of DCASE 2026 Challenge Task 6: Audio Moment Retrieval from Long Audio

GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval

AudioJev: Direct Audio Decisions with Order-Calibrated Probabilities