cs.MMMar 16, 2026

Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation

Authors: Lingsi Zhu, Yuefeng Zou, Yunxiang Zhang, Naixiang Zheng, Guoyuan Wang, Jun Yu, Jiaen Liang, Wei Huang, +2 more

Organizations: University of Science and Technology of China · Unisound AI Technology Co., Ltd. · Pingan Technology Co., Ltd.

Abstract

Estimating Emotional Mimicry Intensity (EMI) in naturalistic environments is a critical yet challenging task in affective computing. The primary difficulty lies in effectively modeling the complex, nonlinear temporal dynamics across highly heterogeneous modalities, especially when physical signals are corrupted or missing. To tackle this, we propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition. Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm. Instead, we leverage textual transcript--which inherently encode a stable, time-independent semantic prior--as central anchors. Specifically, we introduce a Text-Anchored Dual Cross-Attention mechanism that utilizes these robust textual queries to actively filter out frame-level redundancies and align the noisy physical streams. Furthermore, to prevent catastrophic performance degradation caused by inevitably missing data in unconstrained real-world scenarios, we integrate Learnable Missing-Modality Tokens and a Modality Dropout strategy during training. Extensive experiments on the Hume-Vidmimic2 dataset demonstrate that TAEMI effectively captures fine-grained emotional variations and maintains robust predictive resilience under imperfect conditions. Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.

Figures & tables

Explore similar work

CardsList
  1. Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

    May 21, 2026Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala +2Multimodal Emotion RecognitionMultimodal Fusion

  2. Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track

    Mar 14, 2026Jiawen Huang, Chenxi Huang, Zhuofan Wen +7Emotion RecognitionMultimodal Emotion Recognition