Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition
Authors: Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid
Organizations: University of the Basque Country UPV/EHU, San Sebastian, Spain · Laboratory of LESIA, University of Biskra, Algeria · Lab. de Génie Electrique (LAGE), University Kasdi Merbah Ouargla, Ouargla, Algeria · Institute of Electronics, Microelectronics and Nanotechnology (IEMN), Polytechnic University of Hauts-de-France, University of Lille, Valenciennes, France · Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, UAE
Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels -- saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the disagreements between what is said, how it sounds, and what the face shows. We present ConflictAwareAH, a multimodal framework built for this problem. Three pre-trained encoders extract video, audio, and text representations. Pairwise conflict features -- element-wise absolute differences between modality embeddings -- serve as bidirectional cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary text-guided late fusion strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches 0.694 Macro F1 on the labelled test split and 0.715 on the private leaderboard, outperforming published multimodal baselines by over 10 points -- all on a single GPU in under 25 minutes of training.
Figures & tables
Modality
Backbone
Params
Dim
Video
VideoMAE-Base [ 26 ]
86M
768
Audio
HuBERT-Base-LS960 [ 10 ]
94M
768
Text
RoBERTa-GoEmotions [ 23 , 5 ]
125M
768
Table 1: Encoder backbones. All three output 768-dimensional embeddings and were selected for strong pre-trained representations while keeping the model footprint moderate.
Figure 1: ConflictAwareAH architecture. Video frames, audio waveform, and transcript pass through pre-trained encoders and attention pooling to produce embeddings v , a , t . Three pairwise conflict vectors are concatenated with the modality embeddings and fed to a two-layer FFN, producing the full-fusion logit ℓfull . A parallel text-only head produces ℓtext . At inference the two are blended via Eq. 9 .
Figure 2: Train loss and validation Macro F1 over epochs (Run A).
Split
Videos
A/H (+)
No A/H ( − )
Train
778
385 (49%)
393 (51%)
Val
124
75 (60%)
49 (40%)
Test
525
318 (61%)
207 (39%)
Test (unlabeled)
161
—
—
Table 2: BAH dataset split statistics for the ABAW10 A/H challenge.
Method
Macro F1
BAH: Zero-shot M-LLM (vision only) [ 7 ]
0.283
BAH: Video-FocalNet base [ 7 ]
0.566
BAH: LFAN (V+A+T, co-attention) [ 7 ]
0.593
BAH: Zero-shot M-LLM + transcript [ 7 ]
0.634
Ours — single model (1 or 5 windows)
0.690–0.692
Ours — ensemble (2 ckpts × 5 win.)
0.694
Table 3: Comparison on the ABAW10 A/H test split (525 videos). BAH baselines are from González et al. [ 7 ] .
We present a multimodal framework for Ambivalence/Hesitancy (A/H) recognition in video, developed for the ABAW11 challenge at ECCV 2026. The proposed approach fuses textual, acoustic, and visual modalities extracted from the BAH dataset using three pretrained encoders: F2LLM-v2-0.6B for transcripts (1024-d), WavLM-Large for audio (1024-d), and VideoMAE V2 for facial video (768-d). We first establish comprehensive unimodal baselines using classical classifiers (MLP, Random Forest, GBDT), each optimized via Optuna, and obtain a best unimodal Macro F1 of 0.6659 on the test set using text features alone -- substantially outperforming the zero-shot Video-LLaVA baseline (Macro F1: 0.2827). Building on these baselines, we propose a multimodal fusion architecture that combines bidirectional cross-attention across all three modalities with a Gated Multimodal Unit (GMU), with both architectural and optimization hyperparameters selected through a 50-trial Optuna search. This model achieves a Macro F1 of 0.7394 on the validation set, a relative improvement of 11.0% over the best unimodal baseline, confirming that explicit cross-modal interaction captures complementary cues that no single modality provides in isolation. Final predictions on the official, unlabeled private test set are generated using this model and submitted according to the challenge protocol. Code is publicly available at https://github.com/yassineouzar/IUSD_AH/
We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026). We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels. Visual features are encoded as Action Units (AUs) extracted via Py-Feat, audio via Wav2Vec 2.0, and text via BERT. Each modality is processed by a BiLSTM with attention pooling and projected into a shared embedding space. The fusion module computes pairwise absolute differences between modality embeddings, directly capturing the incongruence that characterizes A/H. On the BAH dataset, our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827. Statistical analysis across 1{,}132 videos confirms that temporal variability of AUs is the dominant visual discriminator of A/H.
Aislan Gabriel O. Souza, Agostinho Freire, Leandro Honorato Silva +6
Universidade de Pernambuco (UPE), Escola Polit´ecnica de Pernambuco (POLI), Recife, Brazil
Automatic recognition of ambivalence and hesitancy is challenging because these states may be expressed through inconsistent linguistic, acoustic, facial, and contextual patterns, while top-performing systems often rely on computationally expensive ensembles. We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition for the 11th Affective & Behavior Analysis in-the-Wild (ABAW) Challenge. The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model. Text Residual Fusion treats text as the anchor modality and applies gated residual adjustments based on the other modalities. Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality. The Text Residual Fusion model achieves an average Macro F1-score (MF1) of 75.14% across the Development and Public Test subsets. On the Private Test subset, it reaches an MF1 of 78.24%, outperforming the text model by 4.03%. These results demonstrate that complementary multimodal information can improve recognition performance without requiring a large model ensemble.
Elena Ryumina, Maxim Markitantov, Alexandr Axyonov +4
St. Petersburg Federal Research Center of the Russian Academy of Sciences · St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg, Russia · HSE University, St. Petersburg, Russia +1