Information-Matched Auditing of Conversational Emotion-Shift Forecasting: A Leakage-Safe Evaluation Protocol
Authors: Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan
Organizations: School of Computer Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia
Comparisons in conversational emotion-shift forecasting are easily distorted when a learned model receives rich features but its transition baseline is given a noisy or unavailable current-emotion label, when hyperparameter budgets differ, or when target-turn information enters the representation. We present an information-matched, leakage-safe auditing protocol that closes these gaps with an explicit commitment point, executable causality tests, equal four-configuration validation searches, and seed-dialogue-aware inference. Applying it to six causal model families on four text corpora and two feature-source sensitivity configurations shows that a deployable train-only current-emotion classifier feeding the transition baseline yields a different verdict than an oracle diagnostic where both sides receive gold current emotion. Under the deployable condition, learned models improve over the predicted-label baseline on all four primary corpora; DailyDialog gains are large and survive a 72-comparison Holm correction (best ROC-AUC gain = +0.167, adjusted p=.0144). Under matched gold information, the baseline remains competitive on IEMOCAP and stronger on MELD, while every model retains a large DailyDialog advantage. We additionally report precision-recall AUC, calibration, a strictly utterance-local feature control, same-person self-shift forecasting, and a future-access dose-response test. An eight-item leakage checklist accompanies the protocol. Code, raw seed predictions, and reporting artifacts are available from the corresponding author upon request.
Figures & tables
Threat
Failure mechanism
Detection
Required control
Target-turn exposure
xn+1 enters a window, pooled context, or auxiliary input
nominally per-turn xn was extracted with dialogue-level future context
trace the feature extractor, not only the classifier
utterance-local extraction or causal re-extraction
Split contamination
duplicate dialogue text or the same global speaker occurs across folds
hash dialogue text; audit global identities
group-aware split and duplicate removal
Selection leakage
test fold determines threshold, epoch, or model family
inspect tuning log
validation-only selection with fixed test evaluation
Metric degeneracy
high base rate makes always-shift F1 look competitive
report prediction rate and balanced metrics
AUC, balanced accuracy, and constant baselines
TABLE I: Threat model for conversational emotion forecasting. The audit treats target-derived features and target-derived model states as zero-tolerance failures; the other threats require separate statistical or reporting controls.
Dataset
#Dial
#Utt
Shift r
Always-shift F1
Transition BA
IEMOCAP
151
7,433
0.429
0.600
0.611
MELD
1,432
13,708
0.581
0.735
0.500*
EmoryNLP
827
9,489
0.722
0.839
0.500*
DailyDialog
13,118
102,979
0.198
0.331
0.681
IEMOCAP-mm
151
7,433
0.429
0.600
0.500*
MELD-mm
1,432
13,708
0.581
0.735
0.500*
TABLE II: Dataset statistics and F1 degeneracy. Always-shift F1 =2r/(1+r) for shift rate r (analytic). Transition-BA ≈ 0.50 on MELD/EmoryNLP/IEMOCAP-mm shows standard F1 sits at the degenerate ceiling on exactly the corpora where shift is common; only IEMOCAP and DailyDialog clear it.
Corpus
Immediate N
Same-next
Shift ∣ same
Shift ∣ switch
Self N
Self shift
Mean gap
IEMOCAP
1,592
0.278
0.222
0.509
1,561
0.263
1.98
MELD
2,330
0.242
0.423
0.631
1,864
0.538
2.21
EmoryNLP
905
0.091
0.744
0.719
739
0.673
2.51
DailyDialog
6,740
0.000
–
0.198
5,740
0.180
2.00
TABLE III: Speaker relation and target composition on the four unique test corpora. Same-next is P(sn+1=sn) . Self-shift uses the next-own-utterance target in Eq. ( 3 ); gap is the mean number of turns from n to g(n) .
Figure 1: Speaker composition and emotion-shift rate at adjacent test decision points. Filled bars denote same-speaker transitions; hatched bars denote speaker switches. DailyDialog has no adjacent same-speaker decisions. Exact denominators are given in Table III .
Immediate
Self-shift
Corpus
Global
Speaker
Global
Speaker
IEMOCAP
0.642
0.642
0.586
0.586
MELD
0.676
0.676
0.663
0.663
EmoryNLP
0.608
0.608
0.592
0.592
DailyDialog
0.691
0.691
0.657
0.657
TABLE IV: Train-fold transition baselines under the two targets (test AUC). These are label-only diagnostics, not learned-model results.
Corpus
Gap 1
Gap 2
Gap ≥3
IEMOCAP
442 / .222
864 / .234
255 / .431
MELD
565 / .423
961 / .580
338 / .612
EmoryNLP
82 / .744
464 / .619
193 / .772
DailyDialog
–
5,740 / .180
–
TABLE V: Self-shift composition by forecast gap. Cells show test N / shift rate; – indicates the released test split contains no decision at that gap.
Corpus
Trans-Pred
GRU
DRNN
DGCN
DAG-ERC
PEC
PseudoFut
IEMOCAP
.541
.609
.602
.597
.598
.593
.605
MELD
.584
.593
.600
.608
.610
.593
.593
EmoryNLP
.518
.509
.503
.514
.515
.501
.523
DailyDialog
.577
.699
.706
.698
.691
.695
.693
TABLE VI: Complete next-own-utterance self-shift ROC–AUC (three-seed mean). Trans-Pred is information-matched. DailyDialog gains all survive the separate 24-comparison Holm correction ( padj=.0048 ).
Dataset
Trans-Pred
GRU
DRNN
DGCN
DAG-ERC
PEC
PseudoFut
IEMOCAP
.576
.647
.627
.609
.607
.632
.631
MELD
.588
.621
.629
.623
.608
.620
.600
EmoryNLP
.501
.521
.533
.536
.554
.539
.542
DailyDialog
.580
.746
.744
.748
.744
.743
.741
IEMOCAP-mm
.566
.560
.598
.593
.559
.581
.578
MELD-mm
.584
.723
.716
.724
.720
.720
.723
TABLE VII: Deployable shift ROC–AUC (three-seed mean). Trans-Pred uses current-emotion predictions from a train-only linear classifier; learned models receive no gold emotion. Bold values exceed the matched baseline. DailyDialog is duplicate-free; MM rows are feature-source sensitivity analyses.
Dataset
Trans-Gold
GRU
DRNN
DGCN
DAG-ERC
PEC
PseudoFut
IEMOCAP
.642
.649
.615
.617
.616
.631
.610
MELD
.676
.630
.629
.628
.644
.632
.612
EmoryNLP
.608
.553
.561
.561
.563
.591
.525
DailyDialog
.694
.808
.820
.826
.822
.804
.800
IEMOCAP-mm
.642
.566
.590
.598
.577
.594
.569
MELD-mm
.676
.802
.791
.805
.807
.798
.798
TABLE VIII: Oracle information-matched shift ROC–AUC (three-seed mean). Both the transition baseline and every learned model receive gold current emotion. Bold values exceed Trans-Gold.
Corpus
GRU
Trans-Pred
Δ [95% CI]
IEMOCAP
.642
.552
+.089 [.010,.154]
MELD
.539
.538
+.001 [-.027,.028]
EmoryNLP
.555
.494
+.061 [.014,.112]
DailyDialog
.673
.509
+.164 [.141,.185]
TABLE IX: Strict utterance-local feature control (three-seed ROC–AUC). Both the GRU features and the baseline’s current-emotion classifier are refit from the same train-only TF–IDF/SVD rows.
Figure 2: Future-access dose response with one fixed Transformer architecture. Error bars show three-seed standard deviation. MELD and DailyDialog peak when the target utterance alone becomes visible; none of the six configurations supports a general monotonic-more-future claim.
Test
Validation
Corpus
N
Dup.
N
Dup.
IEMOCAP
31
0
12
0
MELD
280
1
114
0
EmoryNLP
79
0
89
0
DailyDialog
1,000
134
1,000
101
TABLE X: Exact train-fold dialogue duplicates. A duplicate has an identical full ordered text sequence, not merely an overlapping utterance.
Emotion Recognition in Conversations studies how dialogue context, speaker dynamics, and language, voice, and visual cues reveal changing emotional states.
Forecasting emotion in conversation is increasingly framed as predicting the next emotional state's value. We show this targets a hard quantity in a state-observed testbed and propose a learnable, deployable reformulation instead. On valence-arousal-dominance trajectories from a dyadic corpus, repeating a speaker's own previous state sets a persistence baseline that no frozen-encoder correction meaningfully improves upon and a state-space filter exceeds; the floor requires the historical state to be observed, since an estimated anchor collapses persistence below a direct forecaster. What survives is the timing of a departure from persistence, learnable on three of four corpora and near chance on the highest-base-rate corpus, a negative result we report. We recast the task as selective departure forecasting: a lead-time-weighted metric with a leakage protocol, and commitment policies whose best choice switches with the departure rate. A dialogue-level conformal risk control bounds the expected loss of committed predictions, meeting the target in all twelve configurations tested, though the guarantee is bought through abstention, with coverage and alarm recall falling sharply on the hardest corpus. One tested language model's confidence fails as a ranking signal where another succeeds, a model-dependent failure not yet attributable to family versus scale. Code and result logs are available from the corresponding author upon request.
Bin Wen, Dai-Qiao Zhang, Tien-Ping Tan
School of Computer Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia
Conformal prediction (CP) for language models is typically evaluated under a matched protocol, in which calibration and deployment use the same post-training inference configuration, such as quantization mode and inference temperature. In deployment, these choices often differ from those used at calibration to meet memory or throughput constraints or serving policies. We argue that this exposes a limitation in current CP evaluation protocols for language models. We formalize the underlying phenomenon as inference-time model shift: a change in the conformal score distribution induced by inference configuration rather than by the input--label distribution, and theoretically characterize how it breaks split-conformal exchangeability even when the task distribution and trained model are unchanged. We then evaluate its consequences across 8 language models, 4 finite-label benchmarks, and 12 inference configurations. We find that matched evaluation can substantially overstate deployment-time coverage, and that the effect is heterogeneous across models and datasets. We argue that inference configuration should be reported as a first-class experimental axis in CP evaluations for language-model systems.
Achref Doula, Max Mühlhäuser
Tongji University, Shanghai, China · Technical University of Darmstadt, Darmstadt, Germany
Streaming emotion understanding uses historical state while continuously interpreting current audio, often feeding the model's previous prediction back as context. We show that this history conditioning can distort current perception. On a balanced CREMA-D-Stream counterfactual diagnostic, changing only the injected previous emotion label while holding the audio fixed reduces current-audio accuracy from 72.50% to 30.42% and flips 65.69% of predictions. The effect is strongly label-asymmetric, with prior pull ranging from 4.76% to 98.20%, revealing a failure we call previous-belief contamination (PBC). To address PBC, we introduce EmoUpdate, a training-free framework that separates current-audio perception from historical state revision through three components: (1) a prior-blind acoustic firewall that prevents historical state from entering perception; (2) an evidence-shrunk causal belief filter that introduces history only after observation formation and retains label-asymmetric transition structure only when supported by observed evidence; and (3) a closed-form decontamination operator derived from the same counterfactual measurements for serving stacks where firewalling is unavailable. Across four SpeechLMs and two streaming emotion benchmarks, EmoUpdate achieves the best step accuracy and state-balanced accuracy in all eight model--benchmark settings, improving S-BAcc by up to 69.71 points and step accuracy by up to 38.41 points over the strongest controlled baselines.
Haoyue Liu, Zhichao Wang, Ye Chen +2
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China · XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China · University of California, Berkeley +1