Estimating Emotional Mimicry Intensity (EMI) in naturalistic environments is a critical yet challenging task in affective computing. The primary difficulty lies in effectively modeling the complex, nonlinear temporal dynamics across highly heterogeneous modalities, especially when physical signals are corrupted or missing. To tackle this, we propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition. Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm. Instead, we leverage textual transcript--which inherently encode a stable, time-independent semantic prior--as central anchors. Specifically, we introduce a Text-Anchored Dual Cross-Attention mechanism that utilizes these robust textual queries to actively filter out frame-level redundancies and align the noisy physical streams. Furthermore, to prevent catastrophic performance degradation caused by inevitably missing data in unconstrained real-world scenarios, we integrate Learnable Missing-Modality Tokens and a Modality Dropout strategy during training. Extensive experiments on the Hume-Vidmimic2 dataset demonstrate that TAEMI effectively captures fine-grained emotional variations and maintains robust predictive resilience under imperfect conditions. Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.
Figures & tables
Figure 1: The overall architecture of TAEMI. The framework processes temporally aligned audio, visual, and textual modalities. Following modality-specific feature extraction and linear alignment, the temporal representations are fed into a Text-Anchored Dual Cross-Attention module. Here, the textual features serve as queries to aggregate pertinent audio and visual contexts. The resulting fused multimodal representation is then processed by a Multi-Layer Perceptron (MLP) to predict the continuous emotion intensity vector. During training, learnable missing-modality tokens and modality dropout are strategically incorporated to enhance the model’s robustness.
Partition
Duration
Samples
Train
15:07:03
8072
Validation
9:12:02
4588
Test
9:04:05
4586
∑
33:23:10
17246
Table 1: Statistical overview of the Hume-Vidmimic2 dataset partitions. The dataset is strictly divided to ensure no data leakage between training, validation, and testing phases.
Category
Model
Modality
Mean ρval
Baseline
ViT (baseline) [ 14 ]
V
0.09
Wav2Vec2 (baseline) [ 14 ]
A
0.24
ViT + Wav2Vec2 (baseline) [ 14 ]
V+A
0.25
Previous Solutions
Yu et al. [ 34 ]
V+A
0.32
Hallmen et al. [ 3 ]
A
0.38
Savchenko et al. [ 28 ]
V+A+T
0.44
Table 2: Performance comparison on the validation set of the EMI Estimation Challenge. We report the mean Pearson correlation coefficient ( ρval ) across six emotional targets. V , A , and T denote visual, acoustic, and textual modalities, respectively. The best results are highlighted in bold .
TA-CA
LMT
MD
Mean ρval ( ↑ )
Δ
Simple Feature Concat
0.45
-0.10
×
✓
✓
0.48
-0.07
✓
×
✓
0.52
-0.03
✓
✓
×
0.51
-0.04
✓
✓
✓
0.55
-
Table 3: Ablation study on key components of TAEMI. We perform a leave-one-out evaluation to isolate the individual contributions of Text-Anchored Cross-Attention (TA-CA), Learnable Missing Tokens (LMT), and Modality Dropout (MD). The Δ column explicitly highlights the performance drop when a specific module is removed.
Anchor Modality (Query)
Mean ρval ( ↑ )
Vision-Anchored
0.39
Audio-Anchored
0.51
Text-Anchored
0.55
Table 4: Impact of different anchor modalities in the Dual Cross-Attention mechanism. We compare utilizing Vision, Audio, or Text as the central Query to attend to the other two modalities. Text serves as the most effective query anchor due to its high-level semantic stability.
We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy, from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text--audio--vision--motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction.
We participated in the 10th ABAW Challenge, focusing on the Emotional Mimicry Intensity (EMI) Estimation track on the Hume-Vidmimic2 dataset. This task aims to predict six continuous emotion dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy. Through systematic multimodal exploration of pretrained high-level features, we found that, under our pretrained feature setting, direct feature concatenation outperformed the more complex fusion strategies we tested. This empirical finding motivated us to design a systematic approach built upon three core principles: (i) preserving modality-specific attributes through feature-level concatenation; (ii) improving training stability and metric alignment via multi-objective optimization; and (iii) enriching acoustic representations with a VAD-inspired latent prior. Our final framework integrates concatenation-based multimodal fusion, a shared six-dimensional regression head, multi-objective optimization with MSE, Pearson-correlation, and auxiliary branch supervision, EMA for parameter stabilization, and a VAD-inspired latent prior for the acoustic branch. On the official validation set, the proposed scheme achieved our best mean Pearson Correlation Coefficient of 0.478567.
Jiawen Huang, Chenxi Huang, Zhuofan Wen +7
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tianjin Normal University
Affective judgment in real interaction is rarely a purely local prediction problem. Emotional meaning often depends on prior trajectory, accumulated context, and multimodal evidence that may be weak, noisy, or incomplete at the current moment. Although multimodal emotion recognition (MER) has improved the integration of text, speech, and visual signals, many existing systems remain optimized for short-range inference and provide limited support for persistent affective memory, long-horizon dependency modeling, and robust interpretation under imperfect input. This technical report presents the Memory Bear AI Memory Science Engine, a memory-centered framework for multimodal affective intelligence. Instead of treating emotion as a transient output label, the framework models affective information as a structured and evolving variable within a memory system. It organizes processing through structured memory formation, working-memory aggregation, long-term consolidation, memory-driven retrieval, dynamic fusion calibration, and continuous memory updating. At its core, multimodal signals are transformed into structured Emotion Memory Units (EMUs), enabling affective information to be preserved, reactivated, and revised across interaction horizons. Experimental results show consistent gains over comparison systems across benchmark and business-grounded settings, with stronger accuracy and robustness, especially under noisy or missing-modality conditions. The framework offers a practical step from local emotion recognition toward more continuous, robust, and deployment-relevant affective intelligence.