Information-Matched Auditing of Conversational Emotion-Shift Forecasting: A Leakage-Safe Evaluation Protocol
Organizations: School of Computer Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia · School of Mathematical Sciences, Universiti Sains Malaysia, 11800 Gelugor, Penang, Malaysia
Abstract
Comparisons in conversational emotion-shift forecasting are easily distorted when a learned model receives rich features but its transition baseline is given a noisy or unavailable current-emotion label, when hyperparameter budgets differ, or when target-turn information enters the representation. We present an information-matched, leakage-safe auditing protocol that closes these gaps with an explicit commitment point, executable causality tests, equal four-configuration validation searches, and seed-dialogue-aware inference. Applying it to six causal model families on four text corpora and two feature-source sensitivity configurations shows that a deployable train-only current-emotion classifier feeding the transition baseline yields a different verdict than an oracle diagnostic where both sides receive gold current emotion. Under the deployable condition, learned models improve over the predicted-label baseline on all four primary corpora; DailyDialog gains are large and survive a 72-comparison Holm correction (best ROC-AUC gain = +0.167, adjusted p=.0144). Under matched gold information, the baseline remains competitive on IEMOCAP and stronger on MELD, while every model retains a large DailyDialog advantage. We additionally report precision-recall AUC, calibration, a strictly utterance-local feature control, same-person self-shift forecasting, and a future-access dose-response test. An eight-item leakage checklist accompanies the protocol. Code, raw seed predictions, and reporting artifacts are available from the corresponding author upon request.
Figures & tables
| Threat | Failure mechanism | Detection | Required control |
| Target-turn exposure | enters a window, pooled context, or auxiliary input | inspect window boundaries; sweep visible future turns | truncate at before any aggregation |
| Bidirectional state | output at depends on representations from | perturb future inputs and compare prefix logits | unidirectional recurrence or explicit causal mask |
| Feature provenance | nominally per-turn was extracted with dialogue-level future context | trace the feature extractor, not only the classifier | utterance-local extraction or causal re-extraction |
| Split contamination | duplicate dialogue text or the same global speaker occurs across folds | hash dialogue text; audit global identities | group-aware split and duplicate removal |
| Selection leakage | test fold determines threshold, epoch, or model family | inspect tuning log | validation-only selection with fixed test evaluation |
| Metric degeneracy | high base rate makes always-shift F1 look competitive | report prediction rate and balanced metrics | AUC, balanced accuracy, and constant baselines |
| Dataset | #Dial | #Utt | Shift | Always-shift F1 | Transition BA |
| IEMOCAP | 151 | 7,433 | 0.429 | 0.600 | 0.611 |
| MELD | 1,432 | 13,708 | 0.581 | 0.735 | 0.500* |
| EmoryNLP | 827 | 9,489 | 0.722 | 0.839 | 0.500* |
| DailyDialog | 13,118 | 102,979 | 0.198 | 0.331 | 0.681 |
| IEMOCAP-mm | 151 | 7,433 | 0.429 | 0.600 | 0.500* |
| MELD-mm | 1,432 | 13,708 | 0.581 | 0.735 | 0.500* |
| Corpus | Immediate | Same-next | Shift same | Shift switch | Self | Self shift | Mean gap |
| IEMOCAP | 1,592 | 0.278 | 0.222 | 0.509 | 1,561 | 0.263 | 1.98 |
| MELD | 2,330 | 0.242 | 0.423 | 0.631 | 1,864 | 0.538 | 2.21 |
| EmoryNLP | 905 | 0.091 | 0.744 | 0.719 | 739 | 0.673 | 2.51 |
| DailyDialog | 6,740 | 0.000 | – | 0.198 | 5,740 | 0.180 | 2.00 |
| Immediate | Self-shift | |||
| Corpus | Global | Speaker | Global | Speaker |
| IEMOCAP | 0.642 | 0.642 | 0.586 | 0.586 |
| MELD | 0.676 | 0.676 | 0.663 | 0.663 |
| EmoryNLP | 0.608 | 0.608 | 0.592 | 0.592 |
| DailyDialog | 0.691 | 0.691 | 0.657 | 0.657 |
| Corpus | Gap 1 | Gap 2 | Gap |
| IEMOCAP | 442 / .222 | 864 / .234 | 255 / .431 |
| MELD | 565 / .423 | 961 / .580 | 338 / .612 |
| EmoryNLP | 82 / .744 | 464 / .619 | 193 / .772 |
| DailyDialog | – | 5,740 / .180 | – |
| Corpus | Trans-Pred | GRU | DRNN | DGCN | DAG-ERC | PEC | PseudoFut |
| IEMOCAP | .541 | .609 | .602 | .597 | .598 | .593 | .605 |
| MELD | .584 | .593 | .600 | .608 | .610 | .593 | .593 |
| EmoryNLP | .518 | .509 | .503 | .514 | .515 | .501 | .523 |
| DailyDialog | .577 | .699 | .706 | .698 | .691 | .695 | .693 |
| Dataset | Trans-Pred | GRU | DRNN | DGCN | DAG-ERC | PEC | PseudoFut |
| IEMOCAP | .576 | .647 | .627 | .609 | .607 | .632 | .631 |
| MELD | .588 | .621 | .629 | .623 | .608 | .620 | .600 |
| EmoryNLP | .501 | .521 | .533 | .536 | .554 | .539 | .542 |
| DailyDialog | .580 | .746 | .744 | .748 | .744 | .743 | .741 |
| IEMOCAP-mm | .566 | .560 | .598 | .593 | .559 | .581 | .578 |
| MELD-mm | .584 | .723 | .716 | .724 | .720 | .720 | .723 |
| Dataset | Trans-Gold | GRU | DRNN | DGCN | DAG-ERC | PEC | PseudoFut |
| IEMOCAP | .642 | .649 | .615 | .617 | .616 | .631 | .610 |
| MELD | .676 | .630 | .629 | .628 | .644 | .632 | .612 |
| EmoryNLP | .608 | .553 | .561 | .561 | .563 | .591 | .525 |
| DailyDialog | .694 | .808 | .820 | .826 | .822 | .804 | .800 |
| IEMOCAP-mm | .642 | .566 | .590 | .598 | .577 | .594 | .569 |
| MELD-mm | .676 | .802 | .791 | .805 | .807 | .798 | .798 |
| Corpus | GRU | Trans-Pred | [95% CI] |
| IEMOCAP | .642 | .552 | +.089 [.010,.154] |
| MELD | .539 | .538 | +.001 [-.027,.028] |
| EmoryNLP | .555 | .494 | +.061 [.014,.112] |
| DailyDialog | .673 | .509 | +.164 [.141,.185] |
| Test | Validation | |||
| Corpus | Dup. | Dup. | ||
| IEMOCAP | 31 | 0 | 12 | 0 |
| MELD | 280 | 1 | 114 | 0 |
| EmoryNLP | 79 | 0 | 89 | 0 |
| DailyDialog | 1,000 | 134 | 1,000 | 101 |
Emotion Recognition in Conversations
Emotion Recognition in Conversations studies how dialogue context, speaker dynamics, and language, voice, and visual cues reveal changing emotional states.
See all 65 papers