Selective, Risk-Controlled Emotion Forecasting in Conversation: From a Persistence Barrier to Departure Detection with Conformal Guarantees
Authors: Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan
Organizations: School of Computer Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia
Forecasting emotion in conversation is increasingly framed as predicting the next emotional state's value. We show this targets a hard quantity in a state-observed testbed and propose a learnable, deployable reformulation instead. On valence-arousal-dominance trajectories from a dyadic corpus, repeating a speaker's own previous state sets a persistence baseline that no frozen-encoder correction meaningfully improves upon and a state-space filter exceeds; the floor requires the historical state to be observed, since an estimated anchor collapses persistence below a direct forecaster. What survives is the timing of a departure from persistence, learnable on three of four corpora and near chance on the highest-base-rate corpus, a negative result we report. We recast the task as selective departure forecasting: a lead-time-weighted metric with a leakage protocol, and commitment policies whose best choice switches with the departure rate. A dialogue-level conformal risk control bounds the expected loss of committed predictions, meeting the target in all twelve configurations tested, though the guarantee is bought through abstention, with coverage and alarm recall falling sharply on the hardest corpus. One tested language model's confidence fails as a ranking signal where another succeeds, a model-dependent failure not yet attributable to family versus scale. Code and result logs are available from the corresponding author upon request.
Figures & tables
Figure 1: The Persistence Barrier (IEMOCAP, 5-fold session-held-out, CCC averaged over valence/arousal/dominance). Solid bars: zero/few-parameter baselines; hatched bars: trained value forecasters. Both classical same-speaker baselines dominate every trained model at every horizon; free-form latent dynamics sits at essentially zero CCC.
TABLE I: Same-speaker persistence, oracle vs. estimated anchor (IEMOCAP, 5-fold, mean CCC over V/A/D). “Estimated” substitutes a same-turn frozen-encoder probe’s own output for the gold anchor value.
Predictor
t+1
t+2
t+3
Same-speaker persistence (0 par.)
0.625
0.609
0.552
Same-speaker Kalman filter (1 hyperpar.)
0.662
0.647
0.603
Same-speaker AR(1) (2 par.)
0.560
0.539
0.471
Turn-level persistence (0 par.)
0.446
0.549
0.426
Linear extrapolation (0 par.)
0.186
0.201
0.051
Unconstrained direct value forecasters:
TABLE II: The Persistence Barrier (IEMOCAP, 5-fold session-held-out, CCC over valence/arousal/dominance).
Figure 3: Three audits. (a) Freezing the correction head changes CCC by at most 0.02 on all folds. (b) A 60× capacity sweep moves CCC in the third decimal (3 seeds). (c) Across four feature configurations, the head’s end-to-end CCC increment is negative with a bootstrap CI excluding zero in 23 of 24 cells at t+2/t+3 .
Channel (gap =1 )
CCC
n pairs
Ratio to ceiling
Mean-label persistence
0.627
9737
–
Within-annotator
0.620
20786
–
Cross-annotator
0.281
24916
68.4%
Agreement ceiling
0.411
12456
100%
TABLE III: Annotator-artifact control (IEMOCAP, same-speaker gap 1, mean over V/A/D unless noted). Cross-annotator retains 68.4% of the ceiling on average, concealing a per-dimension spread of 39.9 – 85.1% at gap 1.
Figure 4: Base rate is the switch, but not the whole story. Left: a rate-thresholded adaptive policy wins at low departure rates ( +5.1 points over the fixed threshold on DailyDialog) and saturates at high rates, where optimal stopping takes over. Right: within DailyDialog, optimal stopping loses most heavily in the highest-shift stratum ( −13.7 points, CI excluding zero).
Figure 5: Opposite earliness-accuracy profiles across architectures (3 seeds, mean line with seed-range shading, DailyDialog, single seed set).
Dataset
Model
α
Mean risk
95% CI
Cond. risk
Frac. ≤α
Coverage
DailyDialog
GRU-MH
0.1
0.086
[0.061,0.113]
0.154
0.85
0.169
0.2
0.166
[0.127,0.209]
0.217
0.95
0.308
0.3
0.247
[0.212,0.287]
0.279
1.00
0.471
DailyDialog
PsyDyn
0.1
0.074
[0.050,0.097]
0.140
1.00
0.160
0.2
0.156
[0.133,0.185]
0.179
1.00
0.388
0.3
0.203
[0.182,0.225]
0.206
1.00
0.773
TABLE V: Dialogue-level CRC, test-only calibration/evaluation splits (20 repeats per cell, per-dialogue-envelope calibrator). “Frac. ≤α ” is the fraction of repeats individually at or below target; mean risk is at or below α in all 12 cells. “Cond. risk” restricts to dialogues with at least one committed prediction (not what the Proposition bounds).
Figure 6: Risk-coverage. Left: discrete tasks admit a tunable trade-off. Right: in the continuous setting the trade-off is nearly binary – quantile-recalibrated α jumps coverage from under 5% directly to 99.5% .
Dataset
Dialogues (tr/val/te)
Turns
Shift rate
DailyDialog
13118 (10494/1311/1313)
102979
0.173
MELD (mp)
1432 (1145/143/144)
13708
0.554
Hi-EF
53 (42/5/6)
4779
0.640
EmoryNLP
897 (717/89/91)
12606
0.646
IEMOCAP (5-fold CV)
151
10039
N/A (VAD)
TABLE VI: Dataset statistics. All splits are dialogue-level, verified by a leakage test suite.
Model
DailyDialog
MELD (dyadic)
MELD (multi-party) †
EmoryNLP (3-seed)
Random@prior
0.179
0.581
0.581 †
0.609±0.025
GRU direct
0.393
0.685
– †
0.737±0.031
GRU multi-horizon
0.387
0.697
– †
0.673±0.088
Free-form dynamics
0.389
0.710
– †
0.673±0.150
OU-coupled (PsyDyn)
0.371
0.688
0.665 †
0.665±0.117
TABLE VII: Departure forecasting across corpora (shift-F1). MELD is split into its dyadic subset (all four families) and the full multi-party corpus (PsyDyn only, marked † ). EmoryNLP is a 3-seed mean ± std; bold marks the best departure forecaster where its shift-F1 exceeds Random@prior on the same pool.
Dataset
Model
AUROC [range]
MCC
DailyDialog
GRU-direct
0.720 [0.717,0.726]
0.247
DailyDialog
GRU-MH
0.716 [0.713,0.721]
0.239
DailyDialog
Free-form
0.720 [0.719,0.722]
0.242
DailyDialog
PsyDyn
0.687 [0.683,0.689]
0.220
MELD
GRU-direct
0.529 [0.497,0.556]
0.031
MELD
GRU-MH
0.516 [0.492,0.545]
0.057
TABLE VIII: Threshold-free occurrence learnability (3 seeds each). AUROC/MCC are 0.5/0 for both baselines by construction; EmoryNLP is indistinguishable from chance across the full model family.
Family
Dataset
Shift-F1
ECE raw
ECE cal.
Δ 50%
Δ 70%
Δ 90%
14B
DailyDialog
0.101
0.084
0.029
−8.84
−9.22
−5.48
14B
MELD
0.334
0.368
0.261
−16.24
−5.22
−3.72
7B
DailyDialog
0.274
0.704
0.549
+3.26
+3.26
+0.85
7B
MELD
0.719
0.330
0.221
+1.62
+1.49
+0.67
TABLE IX: Both LLMs tested: full metrics. The 14B model’s ranking deltas are negative in 6 of 6 cells; the 7B model’s are positive in 6 of 6 cells.
Figure 7: Calibrated confidence does not imply reliable ranking, for at least one of the two LLMs tested. Top: temperature scaling improves ECE for both. Bottom: abstention by the 14B model’s confidence hurts in 6/6 cells; by the 7B model’s, it helps in 6/6.
Emotion Recognition in Conversations studies how dialogue context, speaker dynamics, and language, voice, and visual cues reveal changing emotional states.
School of Computer Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia