MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition

Chuangxin Zhao; Meimei Zhang; Canran Xiao; Jiachen Luo; Jiajun He; Kaijie Chen; Huy Phan; Tomoki Toda

cs.AIOct 21, 2026

MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition

Authors: Chuangxin Zhao, Meimei Zhang, Canran Xiao, Jiachen Luo, Jiajun He, Kaijie Chen, Huy Phan, Tomoki Toda,

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Trine University, Angola, IN, USA · Sun Yat-sen University, Shenzhen Campus, Shenzhen, China · School of Computation, Information and Technology, Technical University of Munich, Germany · School of Electronical Engineering and Computer Science, Queen Mary University of London, England · Alibaba Group, China · Graduate School of Informatics, Nagoya University, Nagoya, Japan · University of Virginia, USA · German Research Center for Artificial Intelligence (DFKI), Lübeck, Germany · CHI – Chair of Health Informatics, Technical University of Munich, Germany · GLAM – Group on Language, Audio & Music, Imperial College London, U.K.

Abstract

Multimodal multi-label emotion recognition is important for affect-aware systems, but models trained with complete text, visual, and acoustic streams often degrade sharply when some streams are unavailable at deployment. We address this train-test gap by learning a compact representation that preserves emotion-specific evidence from any available subset of modalities. We propose the Multi-modal Emotion Recognition Perceiver (MERP), which maps the joint sequence into a fixed-size latent bank, thereby keeping repeated reasoning independent of raw input length; it extracts label-specific evidence before and after latent reasoning and aligns the two levels through reconstruction. MERP further uses positive and negative class prototypes, a noise-gated feature enhancer with frequency-shaped perturbations during training, and confidence-aware class-wise fusion to strengthen separability, reduce sensitivity to missing inputs, and combine complementary predictors. Under 20 percent test-time random modality drop, MERP achieves micro-F1 scores of 0.563 on CMU-MOSEI and 0.780 on M3ED, exceeding the strongest evaluated baselines by 0.004 and 0.005, respectively; in the severe CMU-MOSEI audio-only setting, it retains 0.473 micro-F1 versus 0.022 for LDDU. These results show that fixed latent reasoning provides a practical, testable route to robust multimodal emotion recognition without requiring missing-modality examples during training. Code is available at https://github.com/Creative-zcx/MERP.

Figures & tables

Fig. 1: Fixed latent reasoning preserves class-specific evidence under compound modality loss. With text and visual streams unavailable, the schematic input-bound path loses the Surprise label after late fusion (a). MERP maps the same zeroed input into a fixed latent workspace and uses class-specific readouts to retain both labels (b).

Fig. 2: Depth-wise missing-modality diagnosis on the CMU-MOSEI validation split. Each colored strip reports linear CKA across all six fusion layers for one corruption pattern, with white-edged dots marking the measured layer outputs. Color is interpolated only along depth within each categorical row. The aligned marginal plot reports the mean and range of class-wise flip rates. The horizontal divider separates single-modality drops from single-retained inputs.

Fig. 3: Overall computation pipeline of MERP. (1) Joint multimodal tokens $\mathbf{U}$ are compressed by learnable latent queries $\mathbf{Z}^{(0)}$ into $\mathbf{Z}_{\mathrm{low}}$ and refined to $\mathbf{Z}_{\mathrm{high}}$ . Class-specific readouts yield $\mathbf{F}_{\mathrm{low}}$ and $\mathbf{F}_{\mathrm{high}}$ , while a class-query decoder reconstructs $\mathbf{F}_{\mathrm{rec}}$ and enforces cross-level consistency. (2) Normalised high-level features are matched to positive and negative class prototypes to form $\mathbf{F}_{\mathrm{proto}}$ ; the prototype, low-level, and high-level features are concatenated and enhanced by SNGA to obtain $\mathbf{F}_{\mathrm{en}}$ . (3) High-level, enhanced, and aggregate probabilities are fused class-wise using learned gate preferences calibrated by $r=|p-0.5|$ , then thresholded for multi-label prediction.

Fig. 4: Architecture of the spectral noise-gated enhancement (SNGA) block. The concatenated class features $\mathbf{F}_{\mathrm{cat}}$ are reduced to $\mathbf{R}$ . The response branch filters white noise $\boldsymbol{\xi}$ in the Fourier domain to obtain a training-only $1/f$ perturbation $\boldsymbol{\epsilon}$ , whereas the gating branch computes $\boldsymbol{\gamma}$ and $\mathbf{G}=\mathbf{R}\odot\boldsymbol{\gamma}$ . The modulated response is regularised by dropout, added residually to $\mathbf{R}$ , and projected to $\mathbf{F}_{\mathrm{en}}$ ; at inference, $\boldsymbol{\epsilon}=\mathbf{0}$ .

Method	Acc	P	R	$\mathbf{F_{0.5}}$	$\mathbf{F_{2}}$	micro-F1	Params	GPU Mem.
HHMPN [ 4 ]	0.380	0.554	0.422	0.521	0.443	0.479	32.5 M ∗	$>$ 15 360 MiB
TAILOR [ 5 ]	0.333	0.553	0.374	0.505	0.400	0.446	14.6 M ∗	6 518 MiB
CARAT [ 6 ]	0.390	0.520	0.500	0.516	0.504	0.510	6.8 M ∗	11 236 MiB
MISA [ 29 ]	0.445	0.588	0.475	0.561	0.494	0.525	1.6 M ∗	376 MiB
MulT [ 11 ]	0.470	0.610	0.511	0.587	0.528	0.556	4.0 M ∗	1 336 MiB
AMP [ 20 ]	0.475	0.607	0.509	0.584	0.526	0.554	11.0 M ∗	11 722 MiB

TABLE I: Performance under 20% test-time random modality drop (

p_{\text{miss}}{=}0.2

, seed=1) on CMU-MOSEI and M 3 ED. Acc is example-based Jaccard accuracy; P and R are micro-averaged over all sample–label decisions, and

F_{\beta}=(1+\beta^{2})PR/(\beta^{2}P+R)

(micro-F1 uses

\beta{=}1

). Within each dataset, the best result in each column is in bold and the second-best is underlined ; higher is better except for Params and GPU Mem. MERP rows are highlighted in very light purple.

Method	CMU-MOSEI		M 3 ED
	Params	Latency	Params	Latency
LDDU [ 7 ]	5.97 M	7.26 ms	6.41 M	7.29 ms
MERP (ours)	5.28 M	5.28 ms	10.92 M	5.21 ms

TABLE II: Inference cost on a single A100 GPU. Latency is averaged over 200 single-sample forward passes after a 20-step warm-up. The best result in each column is in bold ; lower is better, and the MERP row is highlighted in very light purple.

Fig. 5: MERP remains robust across complementary missing-modality evaluations. (a) Absolute micro-F1 decrease from complete input to 20% random modality drop on CMU-MOSEI and M 3 ED; lower is better. (b) Micro-F1 across increasing random-drop rates, with endpoint labels giving the absolute change from

p_{\text{miss}}{=}0

to 0.5. (c) Exact micro-F1 under fixed missing-modality combinations applied to every CMU-MOSEI test sample. The shaded regions highlight MERP in (a) and LDDU’s text–visual collapse in (c).

Fig. 6: MERP more consistently preserves multi-label predictions across diverse modality-loss patterns. Six CMU-MOSEI examples cover complete input, visual or audio removal, and visual-only, audio-only, or text-only views. We compare ground truth (GT) with LDDU, HHMPN, and MERP; gray hatched boxes denote missed labels, while red-hatched labels denote spurious predictions. MERP most closely matches the ground-truth label sets across the six cases, including the severe audio-only condition.

Fig. 7: MERP is stable across structural settings, while reconstruction and enhancement objectives have the largest ablation effects. (a)–(c) Structural ablations with two-seed micro-F1 means and sample-standard-deviation whiskers; purple denotes the default. (d) Loss ablations, with micro-F1 recomputed from micro-P/R. Bar labels are scores and the right column gives changes from the dashed full-model reference; plum marks the largest drop. All ablations use CMU-MOSEI with

p_{\text{miss}}{=}0

Multimodal Emotion Recognition

Multimodal emotion recognition combines signals such as speech, text, facial expressions, video, and physiology to infer affect.

See all 150 papers

Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Aug 31, 2026Jiaqi Zhang, Zheng Pang, Mengting Li +9Missing-Modality Learning Memory-Augmented Neural Networks
Jiangsu University · Griffith University · National University of Singapore +8
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Jul 19, 2026Zilong Huang, Kong Aik Lee, Junjie Li +2Emotion Recognition in Conversations Multimodal Emotion Recognition
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong
BiCFlow-MER: Orchestrating Discriminative and Generative Multimodal Emotion Recognition via Conditional Transport
Sep 23, 2026Yanbing Wang, Shenyue Wang, Chunyang YuEmotion Recognition in Conversations Multimodal Emotion Recognition
OPPO Research Institute

Aug 31, 2026J/K move · Enter open · S save

Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective

Jiaqi Zhang, Zheng Pang, Mengting Li, Yiqi Wang+8

Jiangsu University · Griffith University · National University of Singapore+8

Missing-Modality Learning Memory-Augmented Neural Networks Cross-Modal Knowledge Distillation Multimodal Disentangled Representation Learning Multimodal Emotion Recognition

Open paper AskPDF arXiv

Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference. Our code and project website are available at https://www.jiaqizhang-sengoku.com/Code/PriMD and https://www.jiaqizhang-sengoku.com/PriMD/ , respectively.

Abstract

Multimodal Emotion Recognition

Explore similar work

Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective

EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

BiCFlow-MER: Orchestrating Discriminative and Generative Multimodal Emotion Recognition via Conditional Transport