Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding

Haoyue Liu; Zhichao Wang; Ye Chen; Haonan Deng; Xiaoying Tang

cs.SDAug 21, 2026

Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding

Authors: Haoyue Liu, Zhichao Wang, Ye Chen, Haonan Deng, Xiaoying Tang

Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China · XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China · University of California, Berkeley · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)

Abstract

Streaming emotion understanding uses historical state while continuously interpreting current audio, often feeding the model's previous prediction back as context. We show that this history conditioning can distort current perception. On a balanced CREMA-D-Stream counterfactual diagnostic, changing only the injected previous emotion label while holding the audio fixed reduces current-audio accuracy from 72.50% to 30.42% and flips 65.69% of predictions. The effect is strongly label-asymmetric, with prior pull ranging from 4.76% to 98.20%, revealing a failure we call previous-belief contamination (PBC). To address PBC, we introduce EmoUpdate, a training-free framework that separates current-audio perception from historical state revision through three components: (1) a prior-blind acoustic firewall that prevents historical state from entering perception; (2) an evidence-shrunk causal belief filter that introduces history only after observation formation and retains label-asymmetric transition structure only when supported by observed evidence; and (3) a closed-form decontamination operator derived from the same counterfactual measurements for serving stacks where firewalling is unavailable. Across four SpeechLMs and two streaming emotion benchmarks, EmoUpdate achieves the best step accuracy and state-balanced accuracy in all eight model--benchmark settings, improving S-BAcc by up to 69.71 points and step accuracy by up to 38.41 points over the strongest controlled baselines.

Figures & tables

Figure 1: Previous-belief contamination and its repair. (a) A wrong injected prior collapses current-audio accuracy (CREMA-D-Stream, Qwen2-Audio). (b) EmoUpdate attains the best causal step accuracy in all eight combinations of models and benchmarks.

Figure 2: From entangled perception to causal belief revision. (1) Direct history conditioning writes the previous label into the perception prompt; a counterfactual intervention on that label quantifies previous-belief contamination. (2) EmoUpdate restores the observation with a prior-blind acoustic firewall, then revises it with an evidence-shrunk causal filter. (3) Trajectory quality improves; a prompt-free decontamination operator is the fallback where firewalling is unavailable.

Figure 3: Counterfactual previous-label intervention (CREMA-D-Stream, Qwen2-Audio). Eligible predictions captured by the wrong injected previous label (terracotta: injected label wins). A twentyfold spread shows that no uniform stickiness repairs the effect. Destinations: Figure 7 .

		Ambiguous chunks				Easy chunks
Data	Model	$\Delta$ correct	Pull	$\Delta$ EmoUpdate	$p$	$\Delta$ correct	$\Delta$ EmoUpdate
HumDial-En	Qwen2-Audio	+39.29	63.7	+24.07	$<$ 1e–4	+0.00	+0.00
	Phi-4MM	+27.27	30.3	+8.66	0.019	+5.88	+0.00
	MiniCPM-o 2.6	+17.24	26.4	+2.74	0.125	+0.00	+0.00
CREMA-D-Stream	Qwen2-Audio	+36.07	86.9	+3.07	0.040	+4.35	+0.00
	Phi-4MM	+25.86	49.0	+0.58	0.664	+42.86	+0.00

Table 1: History is worth most exactly where perception is weakest. A chunk is ambiguous when its prior-blind posterior margin falls below the development median and easy when margin and duration are both above it, so the two groups are not complementary; thresholds are never tuned on test.

\Delta

correct is the gain from supplying the correct previous label, pull the rate at which a wrong one is adopted, and

\Delta

EmoUpdate the gain over independent chunk inference on one shared posterior cache (exact paired McNemar

p

). All five difficulty groups are in Appendix F .

		HumDial-En				CREMA-D-Stream
Model	Method	S-BAcc $\uparrow$	Step-MaF1 $\uparrow$	Step $\uparrow$	Final $\uparrow$	S-BAcc $\uparrow$	Step-MaF1 $\uparrow$	Step $\uparrow$	Final $\uparrow$
Qwen2-Audio	DHC	1.43 $\pm 0.63$	20.68 $\pm 1.81$	20.00 $\pm 1.83$	4.76 $\pm 2.08$	18.28 $\pm 2.18$	10.97 $\pm 1.39$	20.70 $\pm 2.17$	18.85 $\pm 2.50$
	ICI	29.14 $\pm 1.94$	68.28 $\pm 2.00$	60.95 $\pm 2.15$	89.52 $\pm 3.00$	70.33 $\pm 2.61$	68.41 $\pm 2.01$	69.57 $\pm 2.20$	70.08 $\pm 2.95$
	CGR	21.71 $\pm 1.36$	55.73 $\pm 2.13$	53.97 $\pm 2.27$	72.38 $\pm 4.19$	70.00 $\pm 2.63$	68.25 $\pm 2.02$	69.57 $\pm 2.21$	70.08 $\pm 2.95$
	MGR	24.57 $\pm 1.25$	60.73 $\pm 2.07$	57.14 $\pm 2.17$	81.90 $\pm 3.67$	70.25 $\pm 2.63$	68.48 $\pm 2.01$	69.88 $\pm 2.21$	70.49 $\pm 2.98$
	C-HMM	29.43 $\pm 2.54$	67.26 $\pm 2.08$	60.00 $\pm 2.34$	85.71 $\pm 3.37$	70.33 $\pm 2.61$	68.66 $\pm 2.00$	69.98 $\pm 2.20$	70.90 $\pm 2.97$

Table 2: Controlled comparison across four frozen SpeechLMs (%, point estimates with bootstrap standard errors; shared posterior cache, one call per chunk). Primary metrics first: state-balanced accuracy and class-balanced step macro-F1, then step accuracy and conventional final-turn accuracy. Bold: per-column best; shaded row: EmoUpdate ;

\Delta

: EmoUpdate minus the per-metric best controlled baseline.

	HumDial-En						CREMA-D-Stream
Model	Neither	Policy	Ground	Both	$\Delta$ grd.	$\Delta$ flt.	Neither	Policy	Ground	Both	$\Delta$ grd.	$\Delta$ flt.
Qwen2-Audio	29.14	37.43	53.71	70.57	+24.57	+16.86	70.33	68.69	74.75	74.02	+4.43	$-$ 0.74
Qwen2.5-Omni	81.43	86.29	91.43	96.29	+10.00	+4.86	42.30	45.00	50.66	53.85	+8.36	+3.20
Phi-4MM	21.14	28.57	84.29	90.86	+63.14	+6.57	15.98	16.39	18.69	18.93	+2.70	+0.25
MiniCPM-o 2.6	79.71	80.57	90.29	91.43	+10.57	+1.14	20.08	21.31	24.02	24.18	+3.93	+0.16

Table 3: Component ablation on state-balanced accuracy (%, point estimates). All arms replay the same episodes. “Policy” applies the deployed evidence-shrunk asymmetric filter (Eq. 6 ) to the contaminated seed observation; “Ground” applies chunkwise updates to the prior-blind observation; and “Both” is the full firewalled inference path. The two stages of this path are complementary: grounding restores the current-audio observation, while causal filtering incorporates temporal state only after perception has been decoupled from historical belief.

Figure 4: Causal step accuracy over the streaming trajectory , all backbones and controlled baselines. Shading is EmoUpdate ’s margin over the best baseline (leads / trails). The CREMA-D-Stream row uses per-panel

y

-ranges; read the ticks first.

Figure 5: The locked stay-probability is not a delicate choice. Band: every

\rho

within 2 S-BAcc points of the panel best (dot). Dashed line: the

\rho

locked on development data, never read off this plot; inside the band in six of eight settings and within 2.9 points of the optimum in all eight.

\alpha

is the coefficient of Equation 6 .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Property	CREMA-D-Stream	HumDial-En
Episode source	Same-speaker clips	Original dialogue order
Temporal pattern	Stable or one shift	Naturally occurring shifts
Split key	Speaker identity	Released source group
Dev/eval episodes	120 / 244	45 / 105
Dev/eval chunks	480 / 976	135 / 315
Label ontology	6	7

Appendix

Table 4: Construction and leakage audit; each audio file appears in at most one trajectory.

Acronym	Name	Causal update rule
DHC	Direct history conditioning	Put $\hat{y}_{t-1}$ in the SpeechLM prompt
ICI	Independent chunk inference	$\hat{y}_{t}=\arg\max_{y}q_{t}(y)$
CGR	Confidence-gated revision	Revise if $\max_{y}q_{t}(y)\geq\tau$
MGR	Margin-gated revision	Revise if $q_{t}(y^{*})-q_{t}(\hat{y}_{t-1})\geq\tau$
C-HMM	Causal HMM	Symmetric Markov prediction followed by Bayes update
IJSR	Instantaneous JSD revision	Revise if $D_{\rm JS}(q_{t},q_{t-1})\geq\tau$

Appendix

Table 5: Controlled baseline rules (

\tau

: development-selected scalar).

Benchmark	Model	Selected candidate
CREMA-D-Stream	Qwen2-Audio	acoustic_first_no_copy
CREMA-D-Stream	Qwen2.5-Omni	acoustic_first_original
CREMA-D-Stream	Phi-4MM	current_audio_contract
CREMA-D-Stream	MiniCPM-o 2.6	prior_deleted_check_short
HumDial-En	Qwen2-Audio	seed_acoustic_short
HumDial-En	Qwen2.5-Omni	acoustic_over_semantic_short

Appendix

Table 6: Selected prior-blind prompt per setting.

Candidate family	Controlled design intent
seed_acoustic_short	Minimal current-audio instruction
acoustic_first_original	Enumerate prosodic evidence
prior_distractor	Mark prior labels as distractors
acoustic_first_no_copy	Explicitly forbid label copying
current_audio_contract	Restrict all evidence to current audio
acoustic_over_semantic	Prefer delivery under modality conflict

Appendix

Table 7: Ten prompt families in the fixed development pool.

Data	Model	Chunks	$N$	No prior	Correct	Pull	$N$	ICI	EmoUpdate	$\Delta$	$p$
			Prior intervention				Shared-cache policy
HumDial-En	Qwen2-Audio	all	70	81.43	98.57	36.19	315	70.16	82.54	+12.38	$<$ 1e–4
		short	32	71.88	100.00	41.15	124	68.55	85.48	+16.94	$<$ 1e–4
		ambiguous	28	57.14	96.43	63.69	162	47.53	71.60	+24.07	$<$ 1e–4
		hard	14	42.86	100.00	67.86	69	49.28	79.71	+30.43	$<$ 1e–4
		easy	24	100.00	100.00	15.97	98	94.90	94.90	+0.00	1.00

Appendix

Table 8: Complete difficulty-stratified analysis for the three backbones on which the balanced injection was run. Development-set medians define the groups: short is duration below the median, ambiguous is prior-blind posterior margin below the median, hard is both at once, and easy is neither. Left: prior intervention on the balanced sample. Right: all policies replayed on one shared posterior cache, with exact paired McNemar

p

against independent chunk inference.

Data	Model	Method	Stable $\uparrow$	Shift $\uparrow$	False sw. $\downarrow$	Delay $\downarrow$	TF1 $\uparrow$
HumDial-En	Qwen2-Audio	ICI	40.00	67.43	71.43	0.07	88.89
		C-HMM	40.00	66.86	68.57	0.07	87.57
		EmoUpdate	54.29	86.86	60.00	0.04	92.31
	Phi-4MM	ICI	88.57	80.00	17.14	0.01	95.68
		C-HMM	88.57	80.00	17.14	0.01	95.68
		EmoUpdate	97.14	84.57	8.57	0.01	96.21

Appendix

Table 9: Stable and shift steps improve together where the trajectory carries information. Decomposition of the shared-cache replay by whether the true state changed at that step (%). A policy that merely revises more often would buy shift accuracy at the cost of stable accuracy and a higher false-switch rate. On HumDial-En (human recordings with many true shifts), EmoUpdate improves shift accuracy on all three backbones while improving stable accuracy on two and preserving it on the third, with fewer spurious switches. On CREMA-D-Stream, whose transitions are uniform by construction and where the shrinkage estimator returns

\alpha=0

(Eq. 6 ), it is conservative rather than aggressive: stable accuracy rises in all three settings without a gain on shift steps.

Figure 6: Contamination, and its repair, on every backbone. Writing the previous label into the prompt (red) costs up to 49 points of causal step accuracy relative to prior-blind inference (grey) across all eight settings; EmoUpdate (green) is highest in all eight. Same quantities as the step-accuracy columns of Table 2 , drawn for comparison at a glance.

Data	Model	Test chunks	Calls/chunk	Test calls	Prompt cand.	Dev calls	Policy cand.
HumDial	Qwen2-Audio	315	1	315	10	1,350	10
	Qwen2.5-Omni	315	1	315	10	1,350	10
	Phi-4MM	315	1	315	10	1,350	10
	MiniCPM-o 2.6	315	1	315	10	1,350	10
CREMA-D-Stream	Qwen2-Audio	976	1	976	10	4,800	10
	Qwen2.5-Omni	976	1	976	10	4,800	10

Appendix

Table 10: Compute audit: zero extra model calls, zero trainable parameters.

Figure 7: Where the prediction lands under each wrong injected prior (CREMA-D-Stream, Qwen2-Audio). The leading terracotta segment is the share captured by the injected label (identical to Figure 3 ), and the slate segments give the destinations of the remainder. Eligibility is the audit rule used throughout: a condition counts only when the injected prior differs from the prior-blind control prediction. A wrong happy prior is adopted

4.8

% of the time and its mass lands mostly on neutral , whereas a wrong fearful prior is adopted

98.2

% of the time.

Figure 8: All ten grounding candidates under the same balanced injection. Dark bars state explicitly that the previous label must not be used;

\star

is the deployed candidate, which never receives a prior at test time. No instruction falls below 59.7% wrong-prior pull, the three most emphatic are the worst, and pull rises with prior-blind accuracy (

r=0.81

). The green rule marks what architectural removal achieves.

Panel of Figure 9 — Figure 9: HumDial-En previous-belief diagnostics under three prompt conditions. Phi-4MM’s accuracy rises under exposure while its flip rate holds at 43.67%, so accuracy change alone is not a contamination measure.

Data	Model	Condition	$N$	Ctrl.	Exposed	Drop	Flip	Wrong pull	Evidence wins	JSD
HumDial	Qwen2.5-Omni	Grounded	35	97.14	95.10	2.04	2.04	0.48	97.55	0.03
		Naive	35	97.14	92.65	4.49	4.49	1.90	95.10	0.03
		Sticky	35	97.14	93.88	3.27	3.27	0.95	97.06	0.02
HumDial	Phi-4MM	Grounded	35	45.71	84.49	$-38.78$	43.67	7.62	98.96	0.30
		Naive	35	45.71	74.69	$-28.98$	43.67	18.57	96.88	0.29
		Sticky	35	45.71	83.27	$-37.55$	43.67	11.43	100.00	0.29

Appendix

Table 11: Additional history-intervention summaries (%); negative drop means exposure helps.

Injected belief	Exposed accuracy $\uparrow$	Prior pull $\downarrow$	Posterior JSD $\downarrow$
neutral	49.17	39.13	19.78
happy	50.00	4.76	23.24
sad	23.33	90.32	49.41
angry	21.67	92.05	50.61
disgust	20.00	96.40	62.68
fearful	18.33	98.20	83.67

Appendix

Table 12: Per-prior breakdown, CREMA-D-Stream Qwen2-Audio (%; JSD on 0–100).

Figure 10: Measurement-based decontamination (CREMA-D-Stream, Qwen2-Audio; two-fold cross-validated). The operator recovers 56% of the accuracy damage and collapses wrong-prior pull from 71.0 to 12.2.

Quantity	Contaminated	Corrected	Prior-blind control
Exposed accuracy $\uparrow$	30.42	53.89	72.50
Prediction flip rate $\downarrow$	65.69	39.31	–
Wrong-prior pull $\downarrow$	71.00	12.17	–

Appendix

Table 13: Two-fold cross-validated decontamination, CREMA-D-Stream Qwen2-Audio (%).

Fitting conditions	Exposed acc. $\uparrow$	Range	Wrong-prior pull $\downarrow$
36	53.33	45.28–62.50	12.00
72	54.54	47.50–58.06	14.44
108	54.72	50.28–58.06	13.44
144	54.44	51.11–57.22	13.89
180	54.49	51.39–58.06	12.94
216	54.54	50.83–57.22	12.33

Appendix

Table 14: Decontamination accuracy versus the size of the fitting grid (CREMA-D-Stream, Qwen2-Audio; held-out fold, three seeds

\times

two folds). Uncorrected exposed accuracy is 28.61 and the prior-blind control is 72.50.

Data	Model	Transition prior	Final	Step	TF1	S-BAcc
HumDial-En	Qwen2-Audio ( $\alpha{=}0.87$ )	Symmetric	88.57	69.84	63.84	53.43
		Full asym.	84.76	80.95	78.99	72.57
		Evidence-shrunk	86.67	82.22	81.54	70.29
		+ endpoint calib.	87.62	82.54	81.87	70.57
	Qwen2.5-Omni ( $\alpha{=}0.87$ )	Symmetric	91.43	90.48	90.49	95.14
		Full asym.	90.48	91.43	92.49	96.00

Appendix

Table 15: Transition-prior variants (%, point estimates). All variants replay identical caches under the same ten-candidate budget.

\alpha

is the shrinkage coefficient estimated from development transitions (Equation 6 ): it is 0 on CREMA-D-Stream, where the evidence-shrunk prior therefore coincides with the symmetric one by construction.

Data	Model	Method	TF1 $\uparrow$	Inertia $\downarrow$	Vol. $\downarrow$	Delay $\downarrow$
HumDial-En	Qwen2-Audio	Best ctl.	56.49	17.71	0.00	0.000
		EmoUpdate	81.87	2.86	53.57	0.038
	Qwen2.5-Omni	Best ctl.	87.68	3.47	9.52	0.013
		EmoUpdate	92.80	2.50	0.00	0.012
	Phi-4MM	Best ctl.	41.54	10.61	0.00	0.000
		EmoUpdate	84.55	2.82	3.03	0.020

Appendix

Table 16: Mechanism metrics (%, point estimates; revision delay in chunks). All four are lower-better except TF1. Inertia is the fraction of true shifts the policy fails to follow; volatility is the fraction of stable steps it destroys; “Best ctl.” is the per-metric best controlled baseline.

	HumDial-En				CREMA-D-Stream
Method	Qwen2-A.	Qwen2.5	Phi-4MM	MiniCPM	Qwen2-A.	Qwen2.5	Phi-4MM	MiniCPM
DHC	–	9.52	0.00	9.09	0.00	65.38	35.06	0.83
ICI	100.00	26.67	100.00	23.33	15.42	30.08	0.00	22.63
CGR	–	26.67	–	23.33	14.52	30.08	0.00	22.63
MGR	–	26.67	100.00	23.33	13.82	30.08	0.00	22.46
C-HMM	50.00	30.00	100.00	24.14	13.58	30.89	0.00	23.19

Appendix

Table 17: Conditional volatility (%, lower is better; point estimates). Dashes mark an empty conditional denominator, not a missing run.

		TF1 $\uparrow$			S-BAcc $\uparrow$
Data	Model	DVL	EmoUpdate	$\Delta$	DVL	EmoUpdate	$\Delta$
CREMA-D-Stream	Qwen2-Audio	19.83	51.55	+31.72	48.25	74.02	+25.77
	Qwen2.5-Omni	10.14	22.89	+12.75	32.05	53.85	+21.80
	Phi-4MM	2.53	4.43	+1.90	17.10	18.93	+1.83
	MiniCPM-o 2.6	3.99	4.68	+0.69	19.33	24.18	+4.85
HumDial-En	Qwen2-Audio	13.23	81.87	+68.64	24.86	70.57	+45.71

Appendix

Table 18: Published DVL-CER adaptation versus EmoUpdate (%). Positive

\Delta

favors EmoUpdate ; uncertainty estimates are reported in Table 19 .

Data	Model	Method	Step	TF1	S-BAcc
CREMA-D-Stream	Qwen2-Audio	DVL-CER adapt.	$51.57\pm 6.33$	$19.83\pm 5.51$	$48.25\pm 5.29$
		EmoUpdate	$\mathbf{76.23\pm 2.06}$	$\mathbf{51.55\pm 3.71}$	$\mathbf{74.02\pm 2.58}$
	Qwen2.5-Omni	DVL-CER adapt.	$35.38\pm 8.70$	$10.14\pm 4.40$	$32.05\pm 8.58$
		EmoUpdate	$\mathbf{54.00\pm 2.08}$	$\mathbf{22.89\pm 2.44}$	$\mathbf{53.85\pm 2.51}$
	Phi-4MM	DVL-CER adapt.	$17.49\pm 0.90$	$2.53\pm 0.00$	$17.10\pm 0.67$
		EmoUpdate	$\mathbf{19.88\pm 1.99}$	$\mathbf{4.43\pm 1.74}$	$\mathbf{18.93\pm 2.18}$

Appendix

Table 19: Paper-derived DVL-CER adaptation versus EmoUpdate (%). DVL-CER reports mean

\pm

standard deviation over three randomized adaptation runs; EmoUpdate reports the full-set point estimate

\pm

episode-bootstrap standard error (

B=1000

Data	Model	Components	Final	Step	Step-MaF1	TF1	S-BAcc	Vol.
HumDial-En	Qwen2-Audio	Neither	89.52	60.95	68.28	50.75	29.14	100.00
		Policy only	89.52	66.35	72.97	58.55	37.43	83.33
		Ground only	87.62	70.16	71.40	63.33	53.71	61.54
		Both ( EmoUpdate )	87.62	82.54	82.31	81.87	70.57	53.57
	Qwen2.5-Omni	Neither	89.52	86.03	86.14	86.61	81.43	26.67
		Policy only	89.52	86.35	86.26	87.54	86.29	16.13

Appendix

Table 20: Complete component analysis across four backbones and both benchmarks (%, point estimates). “Neither”: seed prompt with chunkwise updates; “Policy only”: seed prompt with the causal filter; “Ground only”: prior-blind prompt with chunkwise updates; “Both”: the deployed system. Volatility is lower-better; a dash marks an empty conditional denominator.

Data	Model	$k{=}1$	$k{=}3$	$k{=}5$	$k{=}10$	Stable
HumDial-En	Qwen2-Audio	0.1/def	0.1/def	0.1/def	0.1/def	✓
	Qwen2.5-Omni	0.1/def	0.1/def	0.4/def	0.4/def
	Phi-4MM	0.1/def	0.2/def	0.4/def	0.5/LS
	MiniCPM-o 2.6	0.1/def	0.1/LS	0.1/LS	0.1/LS	✓
CREMA-D-Stream	Qwen2-Audio	0.1/def	0.2/def	0.4/def	0.4/def
	Qwen2.5-Omni	0.1/def	0.1/LS	0.1/LS	0.1/LS	✓

Appendix

Table 21: Selection-budget sensitivity. Each cell is the candidate (

\rho

/endpoint mode) chosen when only the first

k

of the ten pre-specified joint candidates are available; “def” is the acoustic endpoint and “LS” the label-shift endpoint. A check mark means budgets 3, 5, and 10 agree.

Setting	Metric or diagnostic	Observation	Implication
HumDial, Qwen2-Audio	Final accuracy	$-1.90$ vs. ICI	Final-only ranking hides trajectory gains
CREMA-D-Stream, Phi-4MM	TF1	$-0.02$ vs. DHC	Low-information acoustic posterior
HumDial, Phi-4MM	Prior exposure	Accuracy $+38.78$ , flip $43.67$	Text may regularize yet still alter perception
HumDial stable steps	Volatility	Some denominators empty	Report “–” rather than impute a score

Appendix

Table 22: Pre-specified diagnostic cases across model–benchmark settings (gaps in points).

Artifact	Contents
Episode manifests	Ordered chunks, labels, split keys, source-audio hashes
Perception caches	One dense label posterior per frozen-model chunk call
Selection locks	Candidate budget, development ranking, chosen prompt/policy
Policy predictions	Per-step prior, action, posterior, and predicted state
Bootstrap summaries	Point estimates with $B{=}1000$ bootstrap standard errors
Intervention records	Control and every injected-prior posterior

Appendix

Table 23: Reproducibility artifact inventory.

Emotion Recognition in Conversations

Emotion Recognition in Conversations studies how dialogue context, speaker dynamics, and language, voice, and visual cues reveal changing emotional states.

See all 61 papers

Sep 17, 2026

From Generative to Discriminative: Lightweight Adaptation of SpeechLLMs for Multimodal Emotion Recognition

SpeechLLMs can recognise emotion by generating a label token, but this approach relies on the LLM's existing output representations. We compare this generative approach with a discriminative alternative that uses a linear classification head on top of the LLM's hidden states. Because most multimodal emotion recognition studies evaluate with ground-truth transcripts, whereas deployed systems receive ASR output, we run every comparison with both, using a compact speechLLM (frozen Whisper-medium encoder, 3B LLM) on IEMOCAP and MELD. The classification head improves Macro F1 in 10 of 14 configurations. On IEMOCAP, it improves every configuration using ASR transcripts, reaching 76.97% Macro F1, with a significant gain when using speech alone. On MELD, where ASR transcripts have a 32.38% word error rate, it helps with speech alone but not with ASR transcripts. The benefit thus depends on both the dataset and the transcript source: in 4 of the 6 configurations that use text, the head's effect changes sign between ground-truth and ASR transcripts. Conclusions drawn with ground-truth transcripts therefore need not hold with ASR transcripts. Finally, comparing the learned class vectors with the LLM's vocabulary shows that, on IEMOCAP and MELD, their nearest tokens are mostly emotion-related words rather than explicit emotion labels.

Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello +4

Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, USA & India +1

Speech Emotion Recognition Speech Language Models Emotion Representation in Language Models

Ask PDF

Mar 9, 2026

Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction

Speech emotion recognition plays an important role in various applications. However, most existing approaches predict a single emotion label, oversimplifying the inherently ambiguous nature of human emotional expression. Recent large audio-language models show promise in generating richer outputs, but their reasoning ability for ambiguous emotional understanding remains limited. In this work, we reformulate ambiguous emotion recognition as a distributional reasoning problem and present the first systematic study of ambiguity-aware reasoning in LALMs. Our framework comprises two complementary components: an ambiguity-aware objective that aligns predictions with human perceptual distributions, and a structured ambiguity-aware chain-of-thought supervision that guides reasoning over emotional cues. Experiments on IEMOCAP and CREMA-D demonstrate consistent improvements across SFT, DPO, and GRPO training strategies.

Xiaofeng Yu, Jiaheng Dong, Jean Honorio +3

University of Auckland, New Zealand · The University of Melbourne, Australia · University of Birmingham, United Kingdom

Speech Emotion Recognition Emotion Recognition Audio-Language Models CoT Reasoning Label Distribution Learning

Ask PDF

Sep 30, 2026

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models

Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.

Hezhao Zhang, Thomas Hain

School of Computer Science, University of Sheffield Sheffield, United Kingdom

Concept Bottleneck Models Emotion Recognition in Conversations Speech Emotion Recognition Speech Language Models

Ask PDF

Abstract

Emotion Recognition in Conversations

Explore similar work

From Generative to Discriminative: Lightweight Adaptation of SpeechLLMs for Multimodal Emotion Recognition

Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models