Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding

Abdelrahman Mohamed; Lars Kai Hansen; Zheng-Hua Tan

cs.SDSep 26, 2026

Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding

Authors: Abdelrahman Mohamed, Lars Kai Hansen, Zheng-Hua Tan

Organizations: Aalborg University Pioneer Centre for AI · Technical University of Denmark Pioneer Centre for AI

Abstract

Audio emotion recognition (AER) typically assigns a single label to an entire recording, leaving the temporal scope of that label ambiguous when multiple speakers and affective events are present. We address this limitation by reformulating AER as a Temporal Affective Grounding (TAG) task that associates emotions with temporally bounded speech spans and vocal tone descriptions. To support this formulation, we curate temporally annotated versions of existing emotion recognition datasets and construct recordings containing two to four affective speech spans, including overlapping speech. Training in this longer format with a standard language-modeling objective can degrade both emotion recognition and temporal grounding performance, while tone descriptions can provide shortcuts for emotion prediction. To address these challenges, we introduce Masked Temporal Affective Grounding (M-TAG), a supervised training objective that combines full-sequence language modeling with emotion and timestamp cross-entropy losses under attention masking. The masking varies the context visible to emotion-label tokens to reduce reliance on shortcuts and improve generalization, while the timestamp loss incorporates a distance-aware weight to penalize larger temporal errors. We evaluate EMO-TAG, a model fine-tuned using our dataset and objective, on emotion recognition and affective temporal-grounding against three AER and audio-language baselines: Flamingo-Next, Audio-Reasoner, and AffectGPT. Our results show that existing models achieve limited affective temporal-grounding despite competitive emotion recognition performance.

Figures & tables

Figure 1: Overview of our objective. The first forward path calculates the language modeling loss using the full sequence. The second forward pass calculate our auxiliary losses using varying context through attention masking. All logits are discarded in the second forward path except for the ones of timestamp tokens and the first token of the emotion label.

Figure 2: Average performance across context settings. Bars compare generated tone (before low-level features) and refined tone (after low-level features) with full-context, while dashed lines show the partial-context SFT settings. All runs share the same random seed and hyperparameters.

	IEMOCAP			MELD
Model	Original	Refined	Noisy	Original	Refined	Noisy
Audio-Reasoner	40.76 / 44.65	38.71 / 42.24	35.26 / 36.41	40.15 / 44.14	40.36 / 43.02	39.71 / 49.37
AffectGPT	44.36 / 44.36	47.10 / 47.10	40.98 / 40.98	37.33 / 37.33	40.44 / 40.44	35.82 / 35.82
FlamingoNext	37.58 / 49.22	36.49 / 43.55	39.37 / 47.45	44.52 / 53.45	48.91 / 54.91	33.82 / 45.21
Qwen2-Audio-Instruct	43.52 / 43.52	41.94 / 41.94	43.66 / 43.66	38.88 / 39.08	36.68 / 36.68	30.15 / 30.15
EMO-TAG (ours)	65.62 / 66.39 [-1pt] $\pm 1.53\,/\,\pm 1.52$	65.32 / 65.78 [-1pt] $\pm 0.99\,/\,\pm 1.06$	63.51 / 63.67 [-1pt] $\pm 1.70\,/\,\pm 1.56$	54.65 / 55.73 [-1pt] $\pm 0.37\,/\,\pm 0.33$	57.91 / 58.12 [-1pt] $\pm 0.98\,/\,\pm 0.91$	44.12 / 51.47 [-1pt] $\pm 2.16\,/\,\pm 1.08$

Table 1: Emotion recognition across original, refined, and noisy MELD and IEMOCAP evaluation sets. Scores are reported as Emotion Segment Accuracy / Emo-Hit Accuracy (%). Results for EMO-TAG are averaged over three runs, with sample standard deviations shown beneath the means.

	MME-Emotion				MELD Multi-Span		IEMOCAP Multi-Span
Model	Emo-Segment Accuracy (%)				Emo-Duration F1 (%)	Emo-Hit Acc. (%)	Emo-Duration F1 (%)	Emo-Hit Acc. (%)
Model	Noise	Wild	Lab	Avg.	Emo-Duration F1 (%)	Emo-Hit Acc. (%)	Emo-Duration F1 (%)	Emo-Hit Acc. (%)
Audio-Reasoner	47.60 ‡	28.80 ‡	32.20 ‡	36.20	31.43	46.03	28.55	34.68
AffectGPT	16.80 ‡	8.30 ‡	18.00 ‡	14.37	–	–	–	–
FlamingoNext	50.77	32.50	37.63	40.30	28.33	42.53	24.33	42.53
Qwen2-Audio-Instruct	38.80 ‡	22.30 ‡	20.40 ‡	27.17	20.00	15.00	22.00	12.30

Table 2: Comparison on the MELD and IEMOCAP multi-span evaluation sets and the MME-Emotion benchmarks. Avg. denotes the unweighted average across Noise, Wild, and Lab. Results marked with

\ddagger

are taken from Zhang et al. (2026) . EMO-TAG Results are averaged over three runs.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Partial Context SFT
	MELD	IEMOCAP	MELD Multi-Span				IEMOCAP Multi-Span
Setting	Emo-Hit Acc.	Emo-Hit Acc.	Emo-Duration F1	Emotion F1	Span F1 @ 0.25 s	mIoU	Emo-Duration F1	Emotion F1	Span F1 @ 0.25 s	mIoU
Emotion only	54.44	66.36	52.23	54.40	–	–	62.85	60.11	–	–
Speech + Emotion	52.98	66.00	49.09	49.50	–	–	63.25	60.20	–	–
Tone + Emotion	51.87	56.30	49.91	52.15	–	–	52.75	51.23	–	–
Refined Tone + Emotion	53.60	60.12	49.98	53.44	–	–	56.21	53.86	–	–

Appendix

Table 3: Effect of training context and loss components. Single-span results are Emo-Hit Accuracy on the raw test sets. Temporal metrics are Span F1 at a tolerance of 0.25 s and mean timestamp IoU (mIoU).

Figure 3: Effect of context masking probability across emotion recognition benchmarks. All runs use our full objective with encoder–decoder LoRA and a trainable projection layer;

p=0

disables context masking. Markers show the evaluated probabilities

\{0,0.2,0.5,0.7,1\}

. All scores are percentages.

	MELD	IEMOCAP	MME	Multi-span
Configuration	Hit	Hit	Hit	Dur. F1
LLM only LoRA	54.03	63.40	42.35	54.11
+ Audio-encoder LoRA	54.64	63.24	44.75	58.26
+ Projection layer (Emo-TAG)	55.86	65.84	47.40	61.13

Appendix

Table 4: Ablation of trainable components. Components are added cumulatively. All configurations use the full objective with probability

p=1

. MME, MELD, and IEMOCAP report average emotion hit accuracy; Multi-span reports average emotion-duration F1.

MME-Emotion
	SFT	Emo-TAG	Emo-TAG + DEFW + MAFW
Supervision			Emotion only	Full	Emotion only	Full
Training spans			(Multi-Span)	(Multi-Span)	Single	Single
Lab	42.05	47.69	44.06	44.47	45.27	42.05
Wild	31.83	33.72	35.33	37.00	35.44	36.44
Noise	53.20	60.80	61.20	59.40	61.60	56.60

Appendix

Table 5: Effect of training with additional data. Single-span scores are mean Emo-Hit Accuracy over raw, clean, and noisy test sets; multi-span scores are Emotion-Duration F1. Multi-Span means using multi-span data in addition to single-span data. Full means using the full training objective rather than only the emotion loss for the additional data.

	IEMOCAP		MELD		MME
Method	Single-span	Multi-span	Single-span	Multi-span	Average
SFT	60.18	57.76	51.91	50.78	42.36
SFT + GRPO	61.15	58.21	52.33	51.01	43.16
EMO-TAG	65.84	65.65	55.86	56.60	47.40

Appendix

Table 6: Comparison of SFT, GRPO, and EMO-TAG. Single-span scores are mean Emo-Hit Accuracy over the Original, Refined, and Noisy sets; multi-span scores are Emotion-Duration F1 on the Multiple sets. MME scores are mean Emo-Segment Accuracy over Noise, Wild, and Lab.

Spans	MELD	EmoV-DB	IEMOCAP	DFEW	MAFW
1	6,818	2,565	3,484	1,911	1,311
2	2,233 + 703 ∗	1,066 + 502 ∗	1,331 + 555 ∗	241	163
3	1,606 + 520 ∗	822 + 411 ∗	967 + 411 ∗	188	132
4	186 + 61 ∗	105 + 54 ∗	108 + 42 ∗	20	10
Total	10,843 + 1,284 ∗	4,558 + 967 ∗	5,890 + 1,008 ∗	2,360	1,616

Appendix

Table 7: Dataset composition and filtering statistics. Top: composition of our training and test datasets, where MELD represents a more natural setting and EmoV-DB and IEMOCAP represent controlled and acted settings. ∗ Additional samples with overlapping speech spans. Bottom: filtering statistics for the original datasets.

Figure 4: Overview of the data-processing pipeline, from speech and tone validation to the construction of cleaned single- and multi-span datasets.

Version	Complete target text
Sample	meld_train_dia613_utt8
Original	[speech] <speaker1> : "Yes!! Oh." from <\|0.00\|>s to <\|1.23\|>s , [tone] Starts with a high pitch, fast, loud exclamation of ’Yes!’, then transitions to a soft, slow, low-pitched ’Oh.’ with subdued energy., [emotion] surprise.
Refined	[speech] <speaker1> : "Yes!! Oh." from <\|0.00\|>s to <\|1.23\|>s , [tone] Deliberate, restrained delivery with high, broadly varying pitch, strong, sustained loudness, slow, continuous pacing, and unstable pitch control; pitch gradually falls while loudness remains largely steady, [emotion] surprise.
Sample	meld_train_dia1010_utt9
Original	[speech] <speaker1> : "No-o-o! No way!" from <\|0.00\|>s to <\|3.39\|>s , [tone] Deep, drawn-out ’No’ with strain and a sigh, then rapid, loud ’No way’ without pitch change, showing escalating frustration and disbelief., [emotion] disgust.
Refined	[speech] <speaker1> : "No-o-o! No way!" from <\|0.00\|>s to <\|3.39\|>s , [tone] Alert, startled delivery with high, narrowly varying pitch, soft, even loudness, slow pacing with frequent pauses, and breathy-leaning voicing; pitch gradually rises while loudness remains largely steady, [emotion] disgust.

Appendix

Table 8: Selected comparisons showing the complete target_text from the original and refined files.

Source	Emotion–span sequence
Reference	anger $[0.00,2.00]$ ; sadness $[2.75,4.25]$
Emo-TAG	anger $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.00}}]$ ; sadness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.75}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.25}}]$
FlamingoNext	frustration $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}0.38}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.79}}]$ ; sadness $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.13}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}}]$
Audio-Reasoner	anger $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.47}}]$ ; sadness $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.47}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.08}}]$

Appendix

Table 13

Span / emotion / time	Speech and tone
Reference annotation
Reference / 1 <speaker1> anger <\|0.00\|>s – <\|2.00\|>s	Speech: “People have got to finish their stories!” Tone:
Reference / 2 <speaker2> sadness <\|2.75\|>s – <\|4.25\|>s	Speech: “Okay, I’m sorry.” Tone:
Emo-TAG prediction
Emo-TAG / 1 <speaker1> anger <\|0.00\|>s – <\|2.00\|>s	Speech: “ People have got to finish their stories !” Tone: High pitch, fast pace, and loud delivery with clear articulation, conveying urgency and frustration through heightened intensity
Emo-TAG / 2 <speaker2> sadness <\|2.75\|>s – <\|4.25\|>s	Speech: “ Okay , I’m sorry .” Tone: Very quickly, with a noticeably higher-than-normal pitch and a sense of mild strain or urgency rather than relaxed normal talking. She doesn’t hesitate long between phrases, so the entire exchange feels rushed and somewhat pressured

Appendix

Table 14

Source	Emotion–span sequence
Reference	disgust $[0.00,1.25]$ ; neutral $[2.00,3.50]$
Emo-TAG	disgust $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}1.25}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}3.50}}]$
FlamingoNext	happiness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}1.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}}]$
Audio-Reasoner	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.08}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.08}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.08}}]$

Appendix

Table 15

Span / emotion / time	Speech and tone
Reference annotation
Reference / 1 <speaker1> disgust <\|0.00\|>s – <\|1.25\|>s	Speech: “I mean,… how needy is that?” Tone:
Reference / 2 <speaker2> neutral <\|2.00\|>s – <\|3.50\|>s	Speech: “Okay, well then bring her in.” Tone:
Emo-TAG prediction
Emo-TAG / 1 <speaker1> disgust <\|0.00\|>s – <\|1.25\|>s	Speech: “ The boy ? How needy is that ?” Tone: Noticeably higher-than-normal pitch, sounding startled and slightly incredulous instead of neutral or casual. Her speaking speed stays steady around mid-range, neither rushed nor drawn out, conveying mild surprise through inflection rather than dramatic signs
Emo-TAG / 2 <speaker2> neutral <\|2.00\|>s – <\|3.50\|>s	Speech: “ Okay , well then bring her with you .” Tone: Neutral, composed delivery with steady pace, even stress distribution, and no emotional inflection or vocal variation

Appendix

Table 16

Source	Emotion–span sequence
Reference	disgust $[0.00,3.25]$ ; neutral $[3.75,7.75]$ ; neutral $[8.25,10.50]$
Emo-TAG	disgust $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}3.25}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}3.75}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}7.75}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}8.25}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}10.50}}]$
FlamingoNext	disgust $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}0.50}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.40}}]$ ; disgust $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.60}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.80}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}12.00}}]$ ; fear $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}12.50}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}15.00}}]$
Audio-Reasoner	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.98}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.98}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.08}}]$ ; fear $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.08}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}10.56}}]$

Appendix

Table 17

Span / emotion / time	Speech and tone
Reference annotation
Reference / 1 <speaker1> disgust <\|0.00\|>s – <\|3.25\|>s	Speech: “No … the leather sticks to my ass.” Tone:
Reference / 2 <speaker2> neutral <\|3.75\|>s – <\|7.75\|>s	Speech: “All-all right, well make sure you tell him that Joey Tribbiani stopped by to drop off all of these clothes.” Tone:
Reference / 3 <speaker3> neutral <\|8.25\|>s – <\|10.50\|>s	Speech: “I just have to clean that up.” Tone:
Emo-TAG prediction
Emo-TAG / 1 <speaker1> disgust <\|0.00\|>s – <\|3.25\|>s	Speech: “ No ! The leather sticks to my ass !” Tone: Flat, resigned delivery with low pitch, moderate volume, slow pace, heavy drawl, and noticeable breathing, conveying irritation and weariness

Appendix

Table 18

Source	Emotion–span sequence
Reference	surprise $[0.00,1.75]$ ; sadness $[2.75,4.00]$
Emo-TAG	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}1.75}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.75}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.00}}]$
FlamingoNext	neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}0.24}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.56}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.78}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.94}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.54}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.34}}]$
Audio-Reasoner	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.98}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.06}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.08}}]$

Appendix

Table 19

Span / emotion / time	Speech and tone
Reference annotation
Reference / 1 <speaker1> surprise <\|0.00\|>s – <\|1.75\|>s	Speech: “Ooh, do I sense a little bit of resentment?” Tone:
Reference / 2 <speaker2> sadness <\|2.75\|>s – <\|4.00\|>s	Speech: “No. Sorry.” Tone:
Emo-TAG prediction
Emo-TAG / 1 <speaker1> neutral <\|0.00\|>s – <\|1.75\|>s	Speech: “ Do I sense a little bit of resentment ?” Tone: Calm, conversational delivery with subtle rising intonation and deliberate pace, suggesting mild curiosity and\u5C0F\u5FC3\u601D
Emo-TAG / 2 <speaker2> neutral <\|2.75\|>s – <\|4.00\|>s	Speech: “ No . Sorry .” Tone: calm, controlled, soft-to-moderate volume, steady cadence, no emotional inflection

Appendix

Table 20

Source	Emotion–span sequence
Reference	happiness $[0.00,4.25]$ ; surprise $[5.25,6.00]$ ; neutral $[6.75,7.00]$
Emo-TAG	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.25}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$
FlamingoNext	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}7.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$
Audio-Reasoner	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.47}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.47}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.08}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.08}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.78}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.80}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.08}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.08}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.78}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.80}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.08}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.08}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.78}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.80}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.08}}]$

Appendix

Table 21

Span / emotion / time	Speech and tone
Reference annotation
Reference / 1 <speaker1> happiness <\|0.00\|>s – <\|4.25\|>s	Speech: “Unless, we give her all gifts she can use after she’s done being pregnant.” Tone:
Reference / 2 <speaker2> surprise <\|5.25\|>s – <\|6.00\|>s	Speech: “Oh my god.” Tone:
Reference / 3 <speaker3> neutral <\|6.75\|>s – <\|7.00\|>s	Speech: “You think?” Tone:
Emo-TAG prediction
Emo-TAG / 1 <speaker1> neutral <\|0.00\|>s – <\|4.25\|>s	Speech: “ Unless we give her all gifts she can use after she’s done being pregnant .” Tone: Steady, even delivery with consistent pitch and moderate volume, showing no emotional inflection or variation in pace

Appendix

Table 22

Source	Emotion–span sequence
Reference	anger $[0.00,4.75]$ ; neutral $[5.50,10.25]$ ; neutral $[10.75,11.25]$
Emo-TAG	anger $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.75}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}5.50}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.00}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}10.75}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}11.25}}]$
FlamingoNext	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}19.00}}]$
Audio-Reasoner	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}10.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}10.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}14.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}14.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}18.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}18.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}20.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}20.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}21.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}21.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}22.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}22.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}23.00}}]$

Appendix

Table 23

Span / emotion / time	Speech and tone
Reference annotation
Reference / 1 <speaker1> anger <\|0.00\|>s – <\|4.75\|>s	Speech: “Oh yeah? If I can’t, I’ll knock five bucks off the price off the unit.” Tone:
Reference / 2 <speaker2> neutral <\|5.50\|>s – <\|10.25\|>s	Speech: “Ohh, well. Y’know we were always together, in fact the had a nickname for the three of us.” Tone:
Reference / 3 <speaker3> neutral <\|10.75\|>s – <\|11.25\|>s	Speech: “Diverse.” Tone:
Emo-TAG prediction
Emo-TAG / 1 <speaker1> anger <\|0.00\|>s – <\|4.75\|>s	Speech: “ Oh yeah ? If I can’t , I’ll knock five bucks off the price of the unit .” Tone: Starts with high pitch, fast-paced excitement, shifts to calm explanation, then ends with low-pitched, slow, grand statement

Appendix

Table 24

Source	Emotion–span sequence
Reference	anger $[0.00,4.25]$ ; neutral $[5.00,6.75]$
Emo-TAG	anger $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.25}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}5.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}6.75}}]$
FlamingoNext	anger $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}1.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}}]$ ; anger $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}11.00}}]$
Audio-Reasoner	anger $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.75}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.75}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}6.75}}]$

Appendix

Table 25

Span / emotion / time	Speech and tone
Reference
Reference / 1 <speaker1> anger <\|0.00\|>s – <\|4.25\|>s	Speech: “I have been a good son for too long. A good sucker. I’m through with it.” Tone:
Reference / 2 <speaker2> neutral <\|5.00\|>s – <\|6.75\|>s	Speech: “Um- How many people do you want to live with?” Tone:
Emo-TAG
Emo-TAG / 1 <speaker1> anger <\|0.00\|>s – <\|4.25\|>s	Speech: “ I have been a good son for too long . A good sucker . I’m through with it .” Tone: Low pitch, fast pace, and high-volume delivery with clear signs of anger, conveying defiant frustration and decisiveness
Emo-TAG / 2 <speaker2> neutral <\|5.00\|>s – <\|6.75\|>s	Speech: “ How many people do you want to live with ?” Tone: Calm but tensely, with slight hesitation on ’with,’ rising intonation, and restrained urgency suggesting mild impatience beneath neutral phrasing

Appendix

Table 26

Source	Emotion–span sequence
Reference	frustration $[0.00,1.75]$ ; sadness $[2.00,2.75]$
Emo-TAG	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}1.75}}]$ ; sadness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.75}}]$
FlamingoNext	sadness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}1.00}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}}]$
Audio-Reasoner	sadness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.00}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}}]$

Appendix

Table 27

Span / emotion / time	Speech and tone
Reference
Reference / 1 <speaker1> frustration <\|0.00\|>s – <\|1.75\|>s	Speech: “This is humiliating.” Tone:
Reference / 2 <speaker1> sadness <\|2.00\|>s – <\|2.75\|>s	Speech: “I didn’t say that.” Tone:
Emo-TAG
Emo-TAG / 1 <speaker1> frustration <\|0.00\|>s – <\|1.75\|>s	Speech: “ This is humiliating .” Tone: Voice remains high pitch, loud, and fast pace with tense urgency, reflecting clear frustration and irritation
Emo-TAG / 2 <speaker2> sadness <\|2.00\|>s – <\|2.75\|>s	Speech: “ I didn’t say that .” Tone: Low-pitched voice, speaks softly, moves slowly with hesitant rhythm, conveying subdued introspection and quiet sorrow through measured pace and restrained delivery

Appendix

Table 28

Source	Emotion–span sequence
Reference	frustration $[0.00,3.50]$ ; happiness $[4.25,7.50]$ ; sadness $[8.50,10.75]$
Emo-TAG	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}3.50}}]$ ; happiness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.25}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}7.50}}]$ ; sadness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}8.50}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}10.75}}]$
FlamingoNext	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}1.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.00}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$ ; disgust $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}}]$ ; sadness $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}10.00}}]$
Audio-Reasoner	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}}]$ ; surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.25}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}}]$

Appendix

Table 29

Span / emotion / time	Speech and tone
Reference
Reference / 1 <speaker1> frustration <\|0.00\|>s – <\|3.50\|>s	Speech: “I’m growing quite tired and bored of this conversation.” Tone:
Reference / 2 <speaker2> happiness <\|4.25\|>s – <\|7.50\|>s	Speech: “oh, God I kissed Annie. I kissed Annie. I can’t How long have I waited to do that?” Tone:
Reference / 3 <speaker2> sadness <\|8.50\|>s – <\|10.75\|>s	Speech: “And I guess that included you.” Tone:
Emo-TAG
Emo-TAG / 1 <speaker1> frustration <\|0.00\|>s – <\|3.50\|>s	Speech: “ I am growing quite tired and bored of this conversation .” Tone: Mildly frustrated and weary, with a steady, conversational pace and slight strain, indicating irritation rather than intense anger or sadness

Appendix

Table 30

Source	Emotion–span sequence
Reference	neutral $[0.00,2.25]$ ; sadness $[3.25,4.25]$
Emo-TAG	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}2.25}}]$ ; sadness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}3.25}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.25}}]$
FlamingoNext	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}}]$ ; sadness $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}}]$
Audio-Reasoner	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}6.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$

Appendix

Table 31

Span / emotion / time	Speech and tone
Reference
Reference / 1 <speaker1> neutral <\|0.00\|>s – <\|2.25\|>s	Speech: “Are you taking the late fee off of my account then?” Tone:
Reference / 2 <speaker2> sadness <\|3.25\|>s – <\|4.25\|>s	Speech: “He cried hard?” Tone:
Emo-TAG
Emo-TAG / 1 <speaker1> frustration <\|0.00\|>s – <\|2.25\|>s	Speech: “ Are you taking the late fee off of my account then ?” Tone: Slightly raised pitch, quickened tempo, and subtle tension suggest mild frustration, with controlled delivery maintaining polite composure
Emo-TAG / 2 <speaker2> sadness <\|3.25\|>s – <\|4.25\|>s	Speech: “ He cried hard .” Tone: Low pitch, soft-spoken, slow pace with steady volume, conveying quiet sorrow through subdued vocal delivery

Appendix

Table 32

Source	Emotion–span sequence
Reference	neutral $[0.00,1.25]$ ; frustration $[2.00,3.25]$ ; frustration $[3.75,5.75]$
Emo-TAG	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.25}}]$ ; frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}3.75}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}5.75}}]$
FlamingoNext	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}1.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}2.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}5.00}}]$ ; anger (untimed) ; neutral (untimed)
Audio-Reasoner	frustration $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}}]$ ; anger $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.50}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.50}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}}]$ ; unknown $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}4.25}}]$

Appendix

Table 33

Span / emotion / time	Speech and tone
Reference
Reference / 1 <speaker1> neutral <\|0.00\|>s – <\|1.25\|>s	Speech: “oh, She’s back at work. yeah.” Tone:
Reference / 2 <speaker1> frustration <\|2.00\|>s – <\|3.25\|>s	Speech: “And you couldn’t find it?” Tone:
Reference / 3 <speaker2> frustration <\|3.75\|>s – <\|5.75\|>s	Speech: “It’s not fair.” Tone:
Emo-TAG
Emo-TAG / 1 <speaker1> frustration <\|0.00\|>s – <\|3.25\|>s	Speech: “ Oh , she’s back at work . And you couldn’t find it .” Tone: A subtle frustration tinged with resignation, conveyed through low pitch, slow delivery and moderate volume

Appendix

Table 34

Source	Emotion–span sequence
Reference	neutral $[0.00,4.50]$ ; happiness $[5.00,6.50]$ ; neutral $[7.25,8.75]$
Emo-TAG	neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}4.50}}]$ ; happiness $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.25}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}6.50}}]$ ; neutral $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}7.25}},\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}8.75}}]$
FlamingoNext	surprise $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}0.50}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.75}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}3.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.25}}]$ ; happiness $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}8.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.50}}]$
Audio-Reasoner	happiness $[\hbox{\pagecolor{qualCorrectBackground}{\color[rgb]{0.0781,0.3555,0.1328}0.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}}]$ ; neutral $[\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}7.00}},\hbox{\pagecolor{qualIncorrectBackground}{\color[rgb]{0.5938,0.1055,0.1055}9.00}}]$

Appendix

Table 35

Span / emotion / time	Speech and tone
Reference
Reference / 1 <speaker1> neutral <\|0.00\|>s – <\|4.50\|>s	Speech: “Wow. Well, that’s really important, obviously. But, um –” Tone:
Reference / 2 <speaker1> happiness <\|5.00\|>s – <\|6.50\|>s	Speech: “Yeah. We’ll have so much fun.” Tone:
Reference / 3 <speaker2> neutral <\|7.25\|>s – <\|8.75\|>s	Speech: “Well We will see them this year.” Tone:
Emo-TAG
Emo-TAG / 1 <speaker1> neutral <\|0.00\|>s – <\|4.50\|>s	Speech: “ Wow . Well that’s really important obviously , but um –” Tone: Calm and composed with slight hesitation, moderate volume, steady pitch, and steady pace

Appendix

Table 36

Explore similar work

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Aug 30, 2026Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3Audio Understanding Audio-Language Models
University of Maryland, College Park, USA · CNH Industrial India
ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding
Oct 12, 2026Ashutosh Chaubey, Jinxi Guo, Minh Tran +2Temporal Audio Grounding Audio Understanding
Meta Superintelligence Labs · University of Southern California
SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding
Apr 14, 2026Luoyi Sun, Xiao Zhou, Zeqian Li +3Temporal Audio Grounding Audio-Language Models
Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China

Aug 30, 2026J/K move · Enter open · S save

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich+2

University of Maryland, College Park, USA · CNH Industrial India

Audio Understanding Audio-Language Models Temporal Audio Grounding Language Model Post-Training

Open paper AskPDF arXiv

Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.

Abstract

Explore similar work

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding