ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

Ashutosh Chaubey; Jinxi Guo; Minh Tran; Michael Andersland; Ozlem Kalinli

cs.SDOct 12, 2026

ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

Authors: Ashutosh Chaubey, Jinxi Guo, Minh Tran, Michael Andersland, Ozlem Kalinli

Organizations: Meta Superintelligence Labs · University of Southern California

Abstract

Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.

Figures & tables

Panel of Figure 1 — Figure 1: EgoSCAPE benchmark overview. (Left) The three benchmark tasks: audio timeline generation, dense audio captioning (DAC) with sound sources, and temporal audio grounding (TAG). (Right) Samples per task and sound-source distribution.

Figure 2: EgoSCAPE benchmark creation pipeline. To ensure high-quality annotations, we first chunk the audios that are automatically annotated using an omni-LLM with additional supervision from the video and narrations available for Ego4D. All the generated samples for the benchmark are manually verified and corrected at the end.

Figure 3: Hallucination for difft. audio durations in EgoSCAPE (details - § C.4 ).

Proprietary Models
Model	EgoSCAPE Dense Audio Captioning							AudioSet-SL
Model	[email protected]	[email protected]	[email protected]	EvtF1	SegF1	Hal% $\downarrow$	SrcF1	[email protected]	[email protected]	[email protected]	EvtF1	SegF1	Hal% $\downarrow$
Gemini 3.1 Pro	0.235	0.213	0.201	0.084	0.276	31.1	0.371	0.207	0.192	0.181	0.079	0.126	22.7
Gemini 3.6 Flash	0.242	0.215	0.200	0.091	0.197	38.9	0.351	0.217	0.200	0.172	0.037	0.066	30.3
GPT 4o Audio	0.226	0.208	0.198	0.059	0.186	48.4	0.359	0.208	0.184	0.167	0.051	0.058	50.9
Captioning Models

Table 1: Dense audio captioning results on EgoSCAPE and AudioSet-SL datasets comparing ATTEST to other audio and omni LLM baselines. [email protected] and SegF1 are micro. Hal% and [email protected] are macro. * denotes sliding-window evaluation described in § C.3 . TimeAudio and AudioFlamingo 2 did not adhere to the instructions to produce segment-wise sound sources so we have omitted SrcF1 for them.

Proprietary Models
Model	Clotho-Moment			UnAV-100			CASTELLA
Model	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	mIoU
Gemini 3.1 Pro	0.385	0.296	0.344	0.340	0.240	0.294	0.080	0.041	0.081
Gemini 3.6 Flash	0.581	0.431	0.536	0.570	0.410	0.524	0.233	0.124	0.223
GPT 4o Audio	0.106	0.031	0.185	0.200	0.060	0.216	0.042	0.018	0.062
Finetuned Temporal Grounding Models

Table 2: Zero-shot temporal audio grounding results on Clotho-Moment, UnAV-100 and CASTELLA datasets comparing ATTEST to other audio and omni LLM baselines.

Model	TA	$R_{\text{CAG}}$	$R_{\text{AD}}$	$R_{\text{DAC}}$	EgoSCAPE Dense Captioning				Temporal Grounding
					Overall		Long (>120s)		Clotho-Moment	CASTELLA
					[email protected]	Hal% $\downarrow$	[email protected]	Hal% $\downarrow$	[email protected]	[email protected]
Qwen2.5 Omni	-	-	-	-	0.177	48.2	0.169	55.2	0.409	0.092
+ SFT	-	-	-	-	0.367	47.2	0.321	50.1	0.875	0.224
+ Vanilla GRPO	✗	✗	✗	✓	0.395	38.2	0.379	41.2	0.879	0.236
+ ATTEST	✓	✓	✓	✓	0.427	30.5	0.414	34.2	0.909	0.297

Table 3: Ablation study to show the effect of different techniques and rewards used in ATTEST framework. TA: Token-wise fine-grained advantage.

Model	MMAR		LongAudioBench
Model	All	Temporal	All	Temporal
Qwen2.5 Omni	63.2	56.0	61.5	60.1
+ TimePro-RL	61.0	55.4	58.3	59.5
+ ATTEST	64.4	58.2	62.8	62.2
+ ATTEST w/ markers	62.2	56.9	60.5	61.2
AudioFlamingo 3	60.4	53.2	57.7	53.9

Table 4: LongAudioBench and MMAR results.

Figure 4: Perf. drop on intervened clips.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Type	# unique audios	# samples	Avg. Dur.	Max. Dur.	Dense Captions	Source Annot.	Audio Timeline	Temporal Grounding
FTAR-test	Real	1224	1738	9.9	10.0	✓	✗	✗	✓
LAT-Bench	Real	247	2722	600.0	1790.2	✗	✗	✓	✓
Clotho-Moment	Synthetic	7569	7569	60.0	60.0	✗	✗	✗	✓
SpotSound-Bench	Synthetic	300	300	52.9	60.0	✗	✗	✗	✓
CASTELLA	Real	640	1302	206.0	300.0	✗	✗	✗	✓
EgoSCAPE	Real	700	4200	67.6	300.0	✓	✓	✓	✓

Appendix

Table 5: Comparison of EgoSCAPE with existing temporal audio grounding benchmarks.

Panel of Figure 5 — Figure 5: Statistics for the EgoSCAPE benchmark tasks.

Figure 6: Dashboard used for performing human verification.

Task	# audios	# entities	# annotators	Raw Agree.	Fleiss’ $\kappa$
Dense Audio Captioning	700	37803	3	90.7%	0.789
Audio Timeline Generation	700	4587	3	88.6%	0.764
Temporal Audio Grounding	700	2800	3	91.0%	0.797

Appendix

Table 6: Inter-annotator agreement for the human verification of EgoSCAPE benchmark. The agreement is computed over the fine-grained entities for each task: each time-stamped segment in dense captioning and audio timeline generation, and each query in temporal audio grounding.

Proprietary Models
Model	EgoSCAPE Dense Audio Captioning							AudioSet-SL
Model	[email protected]	[email protected]	[email protected]	EvtF1	SegF1	Hal% $\downarrow$	SrcF1	[email protected]	[email protected]	[email protected]	EvtF1	SegF1	Hal% $\downarrow$
Gemini 3.1 Pro	0.235	0.213	0.201	0.084	0.276	31.1	0.371	0.207	0.192	0.181	0.079	0.126	22.7
Gemini 3.5 Flash	0.226	0.208	0.198	0.066	0.180	38.5	0.327	0.208	0.192	0.176	0.043	0.074	23.7
Gemini 3.6 Flash	0.242	0.215	0.200	0.091	0.197	38.9	0.351	0.217	0.200	0.172	0.037	0.066	30.3
Gemini 3.1 Flash Lite	0.228	0.211	0.199	0.064	0.149	34.7	0.400	0.202	0.185	0.171	0.032	0.064	25.1

Appendix

Table 7: Dense audio captioning results (expanded from Tab. 1 ) on EgoSCAPE and AudioSet-SL datasets comparing ATTEST to other audio and omni LLM baselines. [email protected] and SegF1 are micro. Hal% and [email protected] are macro.

Proprietary Models
Model	[email protected]	[email protected]	[email protected]
Gemini 3.1 Pro	0.234	0.197	0.172
Gemini 3.5 Flash	0.213	0.185	0.166
Gemini 3.6 Flash	0.218	0.183	0.164
Gemini 3.1 Flash Lite	0.245	0.205	0.174
GPT RealTime 2.1	0.209	0.184	0.167

Appendix

Table 8: Audio timeline generation results on the EgoSCAPE benchmark.

Proprietary Models
Model	Clotho-Moment			UnAV-100			CASTELLA
Model	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	mIoU
Gemini 3.1 Pro	0.385	0.296	0.344	0.340	0.240	0.294	0.080	0.041	0.081
Gemini 3.5 Flash	0.189	0.079	0.223	0.430	0.220	0.373	0.082	0.039	0.098
Gemini 3.6 Flash	0.581	0.431	0.536	0.570	0.410	0.524	0.233	0.124	0.223
Gemini 3.1 Flash Lite	0.235	0.134	0.302	0.540	0.340	0.487	0.042	0.021	0.050

Appendix

Table 9: Zero-shot temporal audio grounding results on Clotho-Moment, UnAV-100 and CASTELLA datasets comparing ATTEST to other audio and omni LLM baselines.

Proprietary Models
Model	Coarse			Fine
Model	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	mIoU
Gemini 3.1 Pro	0.173	0.110	0.191	0.060	0.042	0.077
Gemini 3.5 Flash	0.159	0.077	0.192	0.048	0.028	0.065
Gemini 3.6 Flash	0.197	0.106	0.236	0.077	0.046	0.094
Gemini 3.1 Flash Lite	0.184	0.117	0.217	0.074	0.048	0.090

Appendix

Table 10: Temporal audio grounding results on EgoSCAPE benchmark across coarse and fine-grained audio-related queries.

Model	MMAR		LongAudioBench
Model	All	Temporal	All	Temporal
Qwen2.5 Omni	63.2	56.0	61.5	60.1
+ TimePro-RL	61.0	55.4	58.3	59.5
+ ATTEST (Ours)	64.4	58.2	62.8	62.2
+ ATTEST w/ markers	62.2	56.9	60.5	61.2
Qwen3 Omni	73.4	67.1	65.9	64.8

Appendix

Table 11: Results of the proposed approach on LongAudioBench and MMAR.

			EgoSCAPE Dense Captioning				Temporal Grounding
			Overall		Long (>120s)		Clotho-Moment	CASTELLA
$\eta_{1}(R_{\text{CAG}})$	$\eta_{2}(R_{\text{AD}})$	$\eta_{3}(R_{\text{DAC}})$	[email protected]	Hal%	[email protected]	Hal%	[email protected]	[email protected]
1	0.2	1	0.427	30.5	0.414	34.2	0.909	0.297
0.25	0.2	1	0.415	35.5	0.389	37.3	0.901	0.256
0.5	0.2	1	0.424	32.3	0.406	35.1	0.905	0.285
2	0.2	1	0.418	30.8	0.401	34.5	0.906	0.267

Appendix

Table 12: Effect of different reward weights on the performance of ATTEST framework.

		EgoSCAPE Dense Captioning				Temporal Grounding
		Overall		Long (>120s)		Clotho-Moment	CASTELLA
Corruption Operator (M)	Corruption Fraction (p)	[email protected]	Hal%	[email protected]	Hal%	[email protected]	[email protected]
silence	0.3	0.427	30.5	0.414	34.2	0.909	0.297
pink noise	0.3	0.419	31.2	0.404	35.6	0.902	0.265
time shuffle	0.3	0.421	32.1	0.402	36.0	0.895	0.268
audio swap	0.3	0.411	31.0	0.398	35.2	0.897	0.259

Appendix

Table 13: Effect of varying corruption operator

M(\cdot)

and fraction

p

used in

R_{\text{CAG}}

Layers used for $R_{\text{AD}}$	EgoSCAPE Dense Captioning				Temporal Grounding
	Overall		Long (>120s)		Clotho-Moment	CASTELLA
	[email protected]	Hal%	[email protected]	Hal%	[email protected]	[email protected]
Last	0.427	30.5	0.414	34.2	0.909	0.297
First	0.419	31.6	0.408	34.9	0.905	0.289
Middle	0.426	30.4	0.409	34.5	0.905	0.281
First + Middle + Last	0.427	30.8	0.412	34.4	0.909	0.293

Appendix

Table 14: Effect of using different layers in

R_{\text{AD}}

. Peak gap represents using the layer where the SFT checkpoint has the most attention density gap between the best and the worst FENSE scoring samples as mentioned in § D.6 .

Figure 7: Percentage of audios in EgoSCAPE where at least one dense captioning segment was predicted in a silenced region.

Panel of Figure 8 — Figure 8: Attention density over the relevant audio tokens across the layers for the best performing and the worst performing samples in EgoSCAPE after performing SFT on different models.

Panel of Figure 9 — Figure 9: Attention density across the layers over the relevant audio tokens before (only SFT) and after tuning the model with the ATTEST framework averaged over all the samples of EgoSCAPE dataset.

Training Data	EgoSCAPE ( $>$ 120s)		AudioSet-SL (Joined-Long)
Training Data	[email protected]	Hal%	[email protected]	Hal%
Real	0.425	33.2	0.398	36.2
Synthetic	0.361	43.8	0.404	34.6
Real + Synthetic	0.416	34.9	0.399	35.3

Appendix

Table 15: Dense captioning results on using Real vs Synthetic long audio training data on EgoSCAPE as well as AudioSet-SL (by joining multiple clips together).

Task. You are annotating a short audio clip (up to 30 seconds) from an egocentric recording. For each candidate sound event detected in the clip, decide keep / drop / relabel by listening to the audio and watching the video.

Inputs. You are given: 1. a short video clip (typically up to 30 seconds) with synchronized audio — watch and listen to it; 2. a list of candidate sound events, each with an idx, a class name, and onset/offset times in seconds (relative to the clip start at 0.0); 3. sometimes a NARRATION section (see below).

Guidelines. Decisions — for each candidate choose one: • keep — the event is present in the audio at roughly the indicated time, and its class label is correct. You can optionally refine the sound event class to make it more accurate. • drop — the event is not actually present. • relabel — a sound is audible at this time but the class name is wrong; the visual frames at the same timestamp support a different class. Use relabel whenever the class is wrong but a sound is present — it is usually a better answer than drop in that case. How to decide: • The candidate list comes from a high-recall audio model. Most events are real, but some are false positives, wrong-class, or duplicates. • Listen to the audio at [onset, offset] before deciding. Faint, brief, distant, or background sounds count as present — do NOT drop just because a sound is subtle, as long as you can hear something matching the class. • Drop when: (a) the audio at [onset, offset] is genuinely silent or contains no sound matching the class; (b) the candidate is an obvious duplicate of another candidate of the same class with near-identical timing AND the audio clearly contains only one such event. • Relabel when: the audio has a clear sound at [onset, offset], but it’s not what the class says — and the video frames disambiguate the true class. • Keep when: you can hear a sound matching the class, even if faint or partially obscured. Rules: • Do not add new events that were not in the candidate list. • Do not change onset or offset. • Every candidate (every idx in the input) must appear in decisions exactly once. • If two candidates of the same class overlap heavily and both look correct, keep both.

Using the narration. A NARRATION section may list what people DID and SAID during the clip (window-relative seconds), split into the WEARER (#C = the FIRST PERSON / camera wearer) and OTHER people (#O/#X, sometimes named). Use it as a STRONG HINT: • #C or #O actions that make sound (handling objects, tools, walking, pouring) support KEEPING a co-occurring candidate of the matching class. • When a candidate’s class conflicts with what the narration says is happening at that time, prefer RELABEL (if the video supports the true class) or DROP. • Narration is sparse and may miss events or be slightly off in time; OVERRIDE it when what you actually hear/see clearly conflicts. When no narration is provided, judge from audio+video alone.

Output format. Return STRICT JSON, in a single json fenced block, with this exact schema: {"decisions": [{"idx": <int>, "action": "keep", "refined_class": "<refined class label>", "reason": "<short reason>"}, {"idx": <int>, "action": "drop", "reason": "<short reason>"}, {"idx": <int>, "action": "relabel", "new_class": "<short class label>", "reason": "<short reason>"}]} Reasons should be one short sentence, no more than 120 characters. No prose outside the JSON block.

User-turn template (the clip’s video+audio is attached): Clip: {chunk_label} ({duration_sec:.2f} s, video + audio) Candidate events (sorted by onset), as a json block — one entry per SED candidate: [{"idx": <int>, "class": "<class name>", "onset": <float>, "offset": <float>}, …] NARRATION (what people did/said; window-relative seconds; #C = the wearer/first person, others are #O/#X): WEARER / first person (#C) — actions & speech: {t:.1f} {text} OTHER people (#O/#X, may be named) — actions & speech: {t:.1f} [#{tag}] {text} Return the JSON block only.

Appendix

Table 16: Prompt used to verify and refine the sound events detected by the sound event detector model as described in § 3 .

Task. You are annotating a short audio clip (up to 30 seconds) from an egocentric recording. Write a SMALL NUMBER of coarse, scene-level captions with timestamps that summarize what is happening in the soundscape; together they should cover almost the entire clip except stretches of genuine silence. You HEAR the audio and can SEE the video, but the video is only for disambiguating what you hear.

Inputs. You are given: 1. the audio clip (video available as a disambiguation aid) — watch and listen to it; 2. a list of verified sound events with timestamps (from an SED model, model-audited; sorted by onset); 3. any transcribed speech (with lang and, when non-English, a text_en translation); 4. sometimes a NARRATION section (see below).

Guidelines. Scene grouping is the primary task: • Treat overlapping or interleaved fine-grained events as ONE layered scene. Name the dominant sound source; mention background/co-occurring sounds inside the SAME caption. • Repeated instances of the same sound within a short window (repeated clinks, taps, footsteps) belong in ONE caption describing the pattern, NOT one caption per instance. • A new caption is only warranted when the scene MEANINGFULLY changes: the dominant sound source changes, a new sonic environment begins, or a distinct isolated event punctuates the ambience. Worked example (what “scene grouping” means): given events on 0–10 s (coughs at 1.1–2.3 and 4.1–4.7; dishwasher running 0.0–5.6; dish clanks at 0.8–1.2 and 4.5–5.0), WRONG is one caption per event; CORRECT is a single caption: “0.0 - 5.6, Dishes clank inside a running dishwasher while a man coughs briefly in the foreground.”

Audio is the subject; video is only a disambiguation aid (strict): • EVERY caption MUST describe a sound that is actually AUDIBLE. If something is visible but produces NO audible sound, DO NOT mention it at all. • The video is allowed only to decide the SOURCE of a sound you actually hear (e.g. a metallic clang + a visible dishwasher rack

\rightarrow

“dishes clank”). You may NOT add silent actions. • Examples of what NOT to write: “The person applies yellow wood glue from a bottle” (visual, no distinctive sound); “A woman gestures toward the camera” (silent); “The person walks toward the kitchen” (unless footsteps are audible). Speech content must be captured (when English or when translated): • If English speech is audible (or the transcript’s lang is en), include a short verbatim quote (<= 15 words, in double quotes) OR a one-sentence summary, inside the caption covering that timespan. Do NOT just write “a person speaks”. • If the speech is non-English, name the language and, if text_en is provided, include a short summary of the translated content; otherwise describe manner and prosody. • The speech description belongs INSIDE the same layered caption, not as its own separate caption (unless speech is the ONLY thing happening for a long stretch). Timestamps must be audio-anchored, not grid-snapped: • Anchor caption boundaries to the ACTUAL audible transitions; the verified event list gives the dense onset/offset timestamps — USE THEM. • Do NOT snap boundaries to multiples of 5 seconds. Prefer real audio-anchored values like 3.2, 11.7, 24.6. Timestamps are float seconds relative to clip start (0.0), one decimal. Content style and coverage: • Describe what the sound ACTUALLY SOUNDS LIKE (pitch, texture, rhythm, loudness, duration, timbre, spatial impression) — not just the activity. Distinguish foreground from background inside a single caption. Aim for 15–30 words per caption. • The union of your caption spans should cover the audible portion of the clip; consecutive spans should be contiguous when scenes flow into each other; overlapping spans only when two independent layers cannot be merged.

Using the narration. A NARRATION section may list what people DID and SAID (window-relative seconds), split into the WEARER (#C = the FIRST PERSON) and OTHER people (#O/#X, sometimes named). Use it as a STRONG HINT for WHO is doing/saying what: • #C actions (chopping, pouring, washing, typing) and #O actions help you name the correct sound SOURCE inside a caption; use the names to keep distinct people apart. • When #C or an #O person is narrated talking around a speech segment, attribute the speech accordingly and capture its content in the caption covering that span. • Narration describes VISIBLE actions/speech, NOT sounds — mention a narrated action only if it is actually AUDIBLE. OVERRIDE it when what you hear/see conflicts; when none is provided, judge from audio+video alone.

Output format. STRICT JSON in a single json fenced block: {"captions": [{"start": 0.0, "end": 8.0, "text": "..."}]} . No prose outside the JSON block.

User-turn template (video attached as disambiguation aid): Clip: {clip_label} ({duration_sec:.2f} s, audio with video available as disambiguation aid) Verified sound events (from an SED model, model-audited; sorted by onset), as a json block: [{"class": "<name>", "onset": <float>, "offset": <float>}, …] Speech transcription (included only when speech is present), as a json block: [{"start": <float>, "end": <float>, "lang": "<lang>", "text_en": "<english>"}, …] NARRATION (what people did/said; window-relative seconds; #C = the wearer/first person, others are #O/#X): WEARER / first person (#C) — actions & speech: {t:.1f} {text} OTHER people (#O/#X, may be named) — actions & speech: {t:.1f} [#{tag}] {text} Return the JSON block only.

Appendix

Table 17: Prompt used to generate the audio timeline events using the dense audio caption sound events as described in § 3 .

Task. You classify the SOURCE of sounds in an egocentric (head-worn camera) video clip. The person wearing the camera is the ‘wearer’ / FIRST PERSON — you never see their face; you see the scene from their eyes and their hands. For EACH segment, watch what happens during its time window and listen, then assign its PRIMARY source class from the taxonomy (you decide how many apply — usually exactly one).

Inputs. You are given: 1. the CLIP itself (video with its audio) — watch and listen to it; 2. a numbered list of SOUND SEGMENTS already detected in the clip (id, [start - end] seconds, and a short description); 3. sometimes a NARRATION section (see below).

Guidelines. Taxonomy (injected): {taxonomy} Decide with this procedure for every segment: 1. Is the sound SPEECH / a VOICE? – Yes, and it is the WEARER’S voice (usually the LOUDEST, closest, most ‘present’ voice, with NO person visible in frame whose mouth moves in sync)

\rightarrow

self_speech. – Yes, and it is ANOTHER co-present person

\rightarrow

other_speech. – Yes, but it comes from a screen/speaker/recording

\rightarrow

electronic_media. 2. Not speech? Then: – the wearer’s own BODY (breathing, footsteps, coughing, chewing)

\rightarrow

self_body. – the wearer ACTING on objects/environment (handling, tools, tap, typing, pouring, washing)

\rightarrow

first_person_interaction. – a device or media

\rightarrow

electronic_media. – anything else

\rightarrow

environment_ambient. The same description can recur at several segments; judge each on its own window. Use ‘uncertain’ only when you truly cannot decide.

Using the narration. A NARRATION section may list what people DID and SAID during the clip (window-relative seconds), split into the WEARER (#C = the FIRST PERSON / camera wearer — that is YOU/the viewpoint) and OTHER people (#O/#X, sometimes named like ‘woman H’). Narration describes VISIBLE actions/speech, NOT sounds — use it as a STRONG HINT for WHO is doing/saying what: • When the WEARER (#C) is narrated talking around a speech segment

\rightarrow

that speech is self_speech. When an OTHER person is narrated talking

\rightarrow

other_speech (use the names to keep distinct people apart). • #C actions (kneading, picking, chopping, pouring, washing) support first_person_interaction for co-occurring object sounds; the wearer walking/breathing

\rightarrow

self_body. • #O actions (another person handling things, walking) support environment_ambient for their non-speech sounds; another person’s speech

\rightarrow

other_speech. • Narration is sparse and may miss events or be slightly off in time; OVERRIDE it when what you actually hear/see clearly conflicts. When no narration is provided, judge by the audio/video alone.

Output format. STRICT JSON only, in one json fenced block, with one object for EVERY segment id given (same ids, no more, no fewer): {{ "segments": [ {{ "seg_id": "<the exact id>", "source_primary": ["<taxonomy key(s)>"], "source_labels": ["<same as primary, plus any extra overlapping source>"], "confidence": <0.0-1.0>, "rationale": "<one short clause; cite the narrated actor if you used narration>" }} ] }}

User-turn template (the clip’s video+audio is attached): CLIP DURATION: {duration_sec:.1f} seconds (the video with audio is attached). SOUND SEGMENTS ( { n } total) — classify the source of each: {seg_id} [{start:.1f} - {end:.1f}] {description} …one line per segment … NARRATION (what people did/said; window-relative seconds; #C = the wearer/first person, others are #O/#X): WEARER / first person (#C) — actions & speech: {t:.1f} {text} OTHER people (#O/#X, may be named) — actions & speech: {t:.1f} [#{tag}] {text} Watch and listen to the attached clip, then return the JSON with one object per seg_id above. Return only the JSON block.

Appendix

Table 18: Prompt used to tag different acoustic events in dense audio captioning to their sound sources as mentioned in § 3 . The injected {taxonomy} defines seven source keys: self_body , self_speech , first_person_interaction , environment_ambient , electronic_media , other_speech , uncertain .

Task. You are cleaning up a list of audio captions that has been assembled from one or more 30-second source clips into a single ordered timeline. The listener will read your output start-to-finish as a single audio sample. Remove or rephrase any wording that assumes a different frame of reference (the original 30 s clip’s start/end, or a per-clip event count).

Inputs. You are given: 1. The ordered caption list of one sample (text only; no audio, no source-chunk metadata), each caption with start, end, and text.

Guidelines. Strict rules: 1. Do NOT change timestamps. Every caption’s start and end must be identical to the input, to at least three decimal places. If you feel you need to change a timestamp, LEAVE THE CAPTION UNCHANGED. 2. Do NOT change the acoustic description (sound source, character, texture, layering) beyond removing stale positional wording. 3. Do NOT change speech content: preserve verbatim quotes (in double quotes) and one-sentence summaries of what was said. 4. Do NOT invent new information, sounds, or speakers. If a phrase is genuinely descriptive (e.g. “the tapping resumes after a brief pause”), keep it. 5. MERGE two captions if their context is seemingly the same and update their start and end times. What to rewrite — remove or rephrase: • Position-relative to a chunk boundary: “towards the end”, “at the very end”, “just before the clip ends”, “at the very beginning”, “at the start of the clip”. • Cardinal/ordinal counters that referenced the source clip’s event count: “for the first time”, “for the second time”, “the third instance”, “for the last time”, “once again”, “yet again”, “again” (when it counts events not sounds), “previously in this clip”, “earlier in the clip”, “later in the clip”. • Ordinal digits or short forms that count within the clip: “1st”, “2nd”, “3rd”, “the third X”. Rephrase strategy: replace with a neutral equivalent when possible (“for the first time”

\rightarrow

drop or “a”; “towards the end”

\rightarrow

drop or a specific timestamp already carried in the timestamp field). If the phrase is only cosmetic, drop it and keep the sentence natural. If removing it would break the sentence, do the minimum rewrite needed to keep the acoustic meaning. If a caption has NONE of these patterns, return it byte-for-byte identical.

Output format. STRICT JSON in a single json fenced block, one item per input caption in the same order, with fields start, end, text (unchanged start/end): {"captions": [{"start": 0.0, "end": 12.4, "text": "..."}]} . No prose outside the JSON block. No trailing commentary.

User-turn template (text-only): Sample id: {sample_id} Total sample duration: {duration:.2f} s Number of captions: {n} Captions (ordered, timeline-local seconds), as a json block: [{"start": <float>, "end": <float>, "text": "<caption text>"}, …] Rewrite each caption per the rules above. Preserve timestamps EXACTLY. Return the JSON block only.

Appendix

Table 19: Prompt used to rewrite the audio timeline when merging multiple individually generated chunk audio timelines.

Task. You are annotating temporal audio grounding (TAG) data. Given the audio and a text annotation of sound events, produce {K} temporal audio grounding QAs that locate specific sounds in time, or abstain if the annotation clearly mismatches the audio.

Inputs. You are given: 1. the audio recording — listen carefully; 2. a text annotation of DETECTED EVENTS (label

\rightarrow

list of intervals) produced by an earlier automated model, which may be imperfect.

Guidelines. Procedure: (1) LISTEN to the audio; (2) READ the annotation; (3) VERIFY whether it plausibly describes what you hear; (4) if clearly wrong or mismatched, ABSTAIN by returning an empty JSON array []; (5) otherwise produce {K} QAs, cross-checking each interval against the audio. Merge recurrences across labels: the annotation may split the SAME underlying sound across multiple event classes (e.g. “Rub” and “Scrape”; “Speech” and “Male speech”). Gather intervals from EVERY label that describes the SAME acoustic phenomenon. Do NOT invent events: every query MUST correspond to at least one event class actually listed in the DETECTED EVENTS block. Do not add sounds absent from the annotation, even if audible. Avoid near-full-duration intervals: any single interval whose length exceeds 80% of {duration} is FORBIDDEN unless the source annotation itself lists an interval that long AND no shorter grouping is possible. Prefer tight, specific intervals aligned with actual bursts you hear. Requirements for each QA: • query: a short natural-language phrase (3–12 words) describing a specific perceivable sound the way a human listener would (“scratchy sanding sound on wood”, “a man speaking”, “sharp metallic clink”). Not a class name. • intervals: list of [start, end] pairs (seconds, 1-decimal) covering EVERY interval where that sound occurs, as you HEAR them. Timestamps within [0.0, {duration}]. No interval may exceed 80% of {duration}. • n_occurrences_hint: “multi” if intervals has 2+ entries, “single” if exactly 1. • verified: true — on every emitted QA, to confirm you listened and cross-checked. Verification, balance, abstain: seek to each proposed interval and confirm the sound is audible there; drop/trim accordingly. Aim for roughly half the queries with 2+ intervals and half with exactly 1. Return [] if the audio is silent/unintelligible while the annotation describes clear events, if the annotation describes events (e.g. English speech) not heard at all, or if annotation and audio disagree on more than roughly half of the described events.

Output format. Return ONLY a JSON array of {K} objects (or [] to abstain). No prose, no markdown fences. Example: [{"query": "a scratchy sanding sound on wood", "intervals": [[0.8, 4.7],[12.1, 18.3]], "n_occurrences_hint": "multi", "verified": true}, {"query": "faint background rustling", "intervals": [[0.0, 30.0]], "n_occurrences_hint": "single", "verified": true}] .

User-turn template (audio attached; appended below the instructions above): AUDIO CLIP: (attached --- please listen) duration_sec: {duration} DETECTED EVENTS (label

\rightarrow

list of intervals, from an automated model — verify against the audio): {events_block} — one line per label: - <label>: [<on>, <off>], [<on>, <off>], … Now produce the JSON array with exactly {K} queries, or [] if you must abstain.

Appendix

Table 20: Prompt used to generate fine-grained audio queries for temporal grounding using dense audio captions. We use

K=2

Task. You are annotating temporal audio grounding (TAG) data. Given the audio and a coarse, time-aligned narrative, produce {K} temporal audio grounding QAs that locate specific sounds in time, or abstain if the narrative clearly mismatches the audio.

Inputs. You are given: 1. the audio recording — listen carefully; 2. a coarse, time-aligned narrative (COARSE CAPTIONS) produced by an earlier model, which may be imperfect or hallucinated.

Guidelines. Procedure: (1) LISTEN carefully; (2) READ the coarse captions; (3) VERIFY whether they plausibly describe what you hear; (4) if clearly wrong or mismatched, ABSTAIN by returning []; (5) otherwise produce {K} QAs, cross-checking each interval against the audio. Query style — specific, descriptive (critical): every query must read like a descriptive acoustic annotation sentence, NOT a terse class-like tag. Include the sound’s texture, loudness, rhythm, and prominence, using what you HEAR. Good: “Gentle, calming music can be faintly heard in the background”; “A steady, coarse scraping of sandpaper on wood dominates the foreground”; “Sharp, sudden metallic clanks interrupt the ambience at irregular intervals”. Bad (do NOT produce): “a woman laughing”; “sanding sound”; “background talking”; “car engine idling”. Extract specific sounds and find every occurrence: captions summarize whole scenes, so a single caption may contain multiple distinct sounds, and the SAME sound may recur across captions worded differently. Decompose the captions into specific perceivable sounds; collect intervals from EVERY caption whose scene contains that sound. Do NOT invent events: every query MUST correspond to a sound explicitly mentioned or unambiguously implied in the COARSE CAPTIONS block. Do not add sounds absent from the captions, even if audible; the purpose is to ground and refine the captions, not extend them. Avoid near-full-duration intervals: any single interval whose length exceeds 80% of {duration} is FORBIDDEN unless the underlying caption span itself is that long AND no tighter interval fits. Requirements for each QA: • query: a descriptive sentence (8–20 words) about ONE specific sound with acoustic modifiers (soft/loud/faint, rhythmic/irregular/continuous, sharp/gentle/muffled, foreground/background). NOT a class name or terse tag. • intervals: list of [start, end] pairs (seconds, 1-decimal), TIGHT to the actual span you hear; no single interval may exceed 80% of {duration}; timestamps within [0.0, {duration}]. • n_occurrences_hint: “multi” (2+ intervals) or “single” (1). • verified: true on every emitted QA. Verification, balance, abstain: seek to each interval before writing; aim for roughly half multi and half single; return [] if the audio is silent while captions describe clear events, if captions describe events not heard at all, or if captions and audio disagree on more than roughly half the events.

Output format. Return ONLY a JSON array of {K} objects (or [] to abstain). No prose, no markdown fences. Example: [{"query": "A steady, coarse scraping of sandpaper on wood dominates the foreground", "intervals": [[0.0, 12.5],[18.0, 30.0]], "n_occurrences_hint": "multi", "verified": true}, {"query": "A brief, muffled cough is barely audible in the background", "intervals": [[14.2, 14.9]], "n_occurrences_hint": "single", "verified": true}] .

User-turn template (audio attached; appended below the instructions above): AUDIO CLIP: (attached --- please listen) duration_sec: {duration} COARSE CAPTIONS (from an automated model — verify against the audio): {events_block} — one line per caption: - [<on> - <off>] <caption text> Now produce the JSON array with exactly {K} queries, or [] if you must abstain.

Appendix

Table 21: Prompt used to generate coarse audio queries for temporal grounding using audio timeline events. We use

K=2

Explore similar work

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding
Apr 14, 2026Luoyi Sun, Xiao Zhou, Zeqian Li +3Temporal Audio Grounding Audio-Language Models
Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China
TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Aug 30, 2026Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3Audio Understanding Audio-Language Models
University of Maryland, College Park, USA · CNH Industrial India
LongAudioSpan: Spanning the Duration and Depth of Audio Comprehension
Aug 26, 2026Wen Huang, Yunfei Chu, Meng Gao +2Audio-Language Model Evaluation Audio Reasoning
Qwen Team, Alibaba Group · Tsinghua University · The Chinese University of Hong Kong

Apr 14, 2026J/K move · Enter open · S save

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

Luoyi Sun, Xiao Zhou, Zeqian Li, Ya Zhang+2

Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China

Temporal Audio Grounding Audio-Language Models Audio-Language Model Evaluation Hallucination Mitigation

Open paper AskPDF arXiv

Large Audio-Language Models (ALMs) have recently demonstrated remarkable capabilities in holistic audio understanding, yet they remain unreliable for temporal grounding, i.e., the task of pinpointing exactly when an event occurs within long-form audio. This limitation stems from two factors: training data dominated by clip-level supervision lacking precise timestamps, and benchmarks that fail to simulate real-world scenarios where short events are obscured by dense background sounds. In this paper, we introduce SpotSound, an audio language model designed for grounding audio events. SpotSound incorporates a novel training objective, specifically designed to suppress hallucinated timestamps for events absent from the input. Additionally, we present SpotSound-Bench, a challenging temporal grounding benchmark where target events occupy less than ~10% of each clip, creating a rigorous 'needle-in-a-haystack' evaluation. Experiments demonstrate that SpotSound achieves state-of-the-art results on temporal grounding benchmarks while maintaining robust performance across general downstream audio-language tasks. Code, models and benchmark are released on https://loiesun.github.io/spotsound/

Abstract

Explore similar work

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

LongAudioSpan: Spanning the Duration and Depth of Audio Comprehension