eess.ASSep 24, 2026

Oracle Complementarity Is Not Realizable Complementarity in Frozen-Encoder Audio-Visual Emotion Recognition

Authors: Benjamin Hurt

Organizations: University of California, Santa Barbara

Abstract

Complementarity analyses of multimodal systems commonly report an oracle ceiling (the fraction of examples on which at least one unimodal branch is correct) and interpret the gap between it and realized fusion accuracy as recoverable headroom. We show that this ceiling is not a fusion target. Using frozen self-supervised audio and visual encoders with a trained head, we measure the ceiling and realized gain on CREMA-D under two audio encoders (one whose fine-tuning lineage includes CREMA-D, one clean) and on EAV under the contaminated encoder. Replacing the fine-tuned encoder with a self-supervised one triples the headroom, from +0.055to +0.186. A second clean encoder and a controlled degradation of the contaminated one fall on the same curve, so provenance moves the headroom by changing audio-branch accuracy. Concatenation realizes at most about half of the headroom (converted fraction 0.55), and on EAV the converted fraction is indistinguishable from zero despite larger headroom. A learned router is beaten by plain concatenation everywhere. Oracle headroom measures branch disagreement, not a fusion budget.

Figures & tables

Explore similar work

CardsList
  1. Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

    Sep 25, 2026Philip H. Lee, Shreeram Suresh Chandra, John H. L. HansenMultimodal Emotion RecognitionEEG Emotion Recognition

  2. Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning

    May 3, 2026Xinmeng Xu, Haoran Xie, S. Joe Qin +3Audio-Visual UnderstandingAudio-Visual Representation Learning

  3. Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition

    Mar 11, 2026Inyong Koo, yeeun Seong, Minseok Son +2Multimodal Emotion RecognitionAudio-Visual Alignment