Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Trine University, Angola, IN, USA · Sun Yat-sen University, Shenzhen Campus, Shenzhen, China · School of Computation, Information and Technology, Technical University of Munich, Germany · School of Electronical Engineering and Computer Science, Queen Mary University of London, England · Alibaba Group, China · Graduate School of Informatics, Nagoya University, Nagoya, Japan · University of Virginia, USA · German Research Center for Artificial Intelligence (DFKI), Lübeck, Germany · CHI – Chair of Health Informatics, Technical University of Munich, Germany · GLAM – Group on Language, Audio & Music, Imperial College London, U.K.
Multimodal multi-label emotion recognition is important for affect-aware systems, but models trained with complete text, visual, and acoustic streams often degrade sharply when some streams are unavailable at deployment. We address this train-test gap by learning a compact representation that preserves emotion-specific evidence from any available subset of modalities. We propose the Multi-modal Emotion Recognition Perceiver (MERP), which maps the joint sequence into a fixed-size latent bank, thereby keeping repeated reasoning independent of raw input length; it extracts label-specific evidence before and after latent reasoning and aligns the two levels through reconstruction. MERP further uses positive and negative class prototypes, a noise-gated feature enhancer with frequency-shaped perturbations during training, and confidence-aware class-wise fusion to strengthen separability, reduce sensitivity to missing inputs, and combine complementary predictors. Under 20 percent test-time random modality drop, MERP achieves micro-F1 scores of 0.563 on CMU-MOSEI and 0.780 on M3ED, exceeding the strongest evaluated baselines by 0.004 and 0.005, respectively; in the severe CMU-MOSEI audio-only setting, it retains 0.473 micro-F1 versus 0.022 for LDDU. These results show that fixed latent reasoning provides a practical, testable route to robust multimodal emotion recognition without requiring missing-modality examples during training. Code is available at https://github.com/Creative-zcx/MERP.
Figures & tables
Fig. 1: Fixed latent reasoning preserves class-specific evidence under compound modality loss. With text and visual streams unavailable, the schematic input-bound path loses the Surprise label after late fusion (a). MERP maps the same zeroed input into a fixed latent workspace and uses class-specific readouts to retain both labels (b).
Fig. 2: Depth-wise missing-modality diagnosis on the CMU-MOSEI validation split. Each colored strip reports linear CKA across all six fusion layers for one corruption pattern, with white-edged dots marking the measured layer outputs. Color is interpolated only along depth within each categorical row. The aligned marginal plot reports the mean and range of class-wise flip rates. The horizontal divider separates single-modality drops from single-retained inputs.
Fig. 3: Overall computation pipeline of MERP. (1) Joint multimodal tokens U are compressed by learnable latent queries Z(0) into Zlow and refined to Zhigh . Class-specific readouts yield Flow and Fhigh , while a class-query decoder reconstructs Frec and enforces cross-level consistency. (2) Normalised high-level features are matched to positive and negative class prototypes to form Fproto ; the prototype, low-level, and high-level features are concatenated and enhanced by SNGA to obtain Fen . (3) High-level, enhanced, and aggregate probabilities are fused class-wise using learned gate preferences calibrated by r=∣p−0.5∣ , then thresholded for multi-label prediction.
Fig. 4: Architecture of the spectral noise-gated enhancement (SNGA) block. The concatenated class features Fcat are reduced to R . The response branch filters white noise ξ in the Fourier domain to obtain a training-only 1/f perturbation ϵ , whereas the gating branch computes γ and G=R⊙γ . The modulated response is regularised by dropout, added residually to R , and projected to Fen ; at inference, ϵ=0 .
Dataset
Method
Acc
P
R
F0.5
F2
micro-F1
Params
GPU Mem.
HHMPN [ 4 ]
0.380
0.554
0.422
0.521
0.443
0.479
32.5 M ∗
> 15 360 MiB
TAILOR [ 5 ]
0.333
0.553
0.374
0.505
0.400
0.446
14.6 M ∗
6 518 MiB
CARAT [ 6 ]
0.390
0.520
0.500
0.516
0.504
0.510
6.8 M ∗
11 236 MiB
MISA [ 29 ]
0.445
0.588
0.475
0.561
0.494
0.525
1.6 M ∗
376 MiB
MulT [ 11 ]
0.470
0.610
0.511
0.587
0.528
0.556
4.0 M ∗
1 336 MiB
AMP [ 20 ]
0.475
0.607
0.509
0.584
0.526
0.554
11.0 M ∗
11 722 MiB
TABLE I: Performance under 20% test-time random modality drop ( pmiss=0.2 , seed=1) on CMU-MOSEI and M 3 ED. Acc is example-based Jaccard accuracy; P and R are micro-averaged over all sample–label decisions, and Fβ=(1+β2)PR/(β2P+R) (micro-F1 uses β=1 ). Within each dataset, the best result in each column is in bold and the second-best is underlined ; higher is better except for Params and GPU Mem. MERP rows are highlighted in very light purple.
Method
CMU-MOSEI
M 3 ED
Params
Latency
Params
Latency
LDDU [ 7 ]
5.97 M
7.26 ms
6.41 M
7.29 ms
MERP (ours)
5.28 M
5.28 ms
10.92 M
5.21 ms
TABLE II: Inference cost on a single A100 GPU. Latency is averaged over 200 single-sample forward passes after a 20-step warm-up. The best result in each column is in bold ; lower is better, and the MERP row is highlighted in very light purple.
Fig. 5: MERP remains robust across complementary missing-modality evaluations. (a) Absolute micro-F1 decrease from complete input to 20% random modality drop on CMU-MOSEI and M 3 ED; lower is better. (b) Micro-F1 across increasing random-drop rates, with endpoint labels giving the absolute change from pmiss=0 to 0.5. (c) Exact micro-F1 under fixed missing-modality combinations applied to every CMU-MOSEI test sample. The shaded regions highlight MERP in (a) and LDDU’s text–visual collapse in (c).
Fig. 6: MERP more consistently preserves multi-label predictions across diverse modality-loss patterns. Six CMU-MOSEI examples cover complete input, visual or audio removal, and visual-only, audio-only, or text-only views. We compare ground truth (GT) with LDDU, HHMPN, and MERP; gray hatched boxes denote missed labels, while red-hatched labels denote spurious predictions. MERP most closely matches the ground-truth label sets across the six cases, including the severe audio-only condition.
Fig. 7: MERP is stable across structural settings, while reconstruction and enhancement objectives have the largest ablation effects. (a)–(c) Structural ablations with two-seed micro-F1 means and sample-standard-deviation whiskers; purple denotes the default. (d) Loss ablations, with micro-F1 recomputed from micro-P/R. Bar labels are scores and the right column gives changes from the dashed full-model reference; plum marks the largest drop. All ablations use CMU-MOSEI with pmiss=0 .
Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference. Our code and project website are available at https://www.jiaqizhang-sengoku.com/Code/PriMD and https://www.jiaqizhang-sengoku.com/PriMD/ , respectively.
Jiaqi Zhang, Zheng Pang, Mengting Li +9
Jiangsu University · Griffith University · National University of Singapore +8
Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance. However, existing fusion approaches in MERC often ignore modality-specific uncertainty across utterances caused by conflicting cues, varying noise, and missing modality-specific signals. We propose EmoEUS, an explicit uncertainty supervision framework for MERC. EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates. We also introduce an explicitly supervised loss that aligns each utterance's predicted variance with the distance between the utterance's distributional representation and its emotion- and modality-specific cluster center. Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.
Zilong Huang, Kong Aik Lee, Junjie Li +2
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong
In multimodal emotion recognition (MER), human affective states are inferred by integrating complementary cues from multiple modalities. In audio-text MER, affective cues are often entangled with speaker style and lexical content, while cross-modal disagreement further complicates how the evidence should be integrated. Under conventional discriminative fusion, multimodal evidence is compressed into a terminal prediction, with modality-specific cues and conflict information insufficiently preserved. In large generative affective models, by contrast, affective reasoning is typically embedded in language decoding, leaving emotion evidence implicit and difficult to verify in a structured space. To address these limitations, BiCFlow-MER (Bidirectional Conditional Flow for Multimodal Emotion Recognition) is proposed as a conditional-flow framework in which audio-text MER is formulated as generative evidence transport within a structured emotion space. Within BiCFlow-MER, emotion-oriented evidence is disentangled from speaker-style and lexical-content factors to construct a conflict-aware affective condition. Guided by this condition, each utterance is transported to an explicit emotion-space endpoint through a bidirectional rectified flow. Candidate emotions are jointly verified through adaptive prototype-cloud scoring of the transported endpoint and backward class-to-condition consistency with the original multimodal condition, enabling conflict-aware recognition. BiCFlow-MER is shown to outperform all compared methods across IEMOCAP, MELD, and the zero-shot CASE benchmark. By orchestrating discriminative recognition and generative evidence modeling through conditional transport, BiCFlow-MER defines a new MER paradigm.