Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition

Salah Eddine Bekhouche; Hichem Telli; Azeddine Benlamoudi; Salah Eddine Herrouz; Abdelmalik Taleb-Ahmed; Abdenour Hadid

cs.CVMar 16, 2026

Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition

Authors: Salah Eddine Bekhouche, Hichem Telli, Azeddine Benlamoudi, Salah Eddine Herrouz, Abdelmalik Taleb-Ahmed, Abdenour Hadid

Organizations: University of the Basque Country UPV/EHU, San Sebastian, Spain · Laboratory of LESIA, University of Biskra, Algeria · Lab. de Génie Electrique (LAGE), University Kasdi Merbah Ouargla, Ouargla, Algeria · Institute of Electronics, Microelectronics and Nanotechnology (IEMN), Polytechnic University of Hauts-de-France, University of Lille, Valenciennes, France · Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, UAE

Abstract

Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels -- saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the disagreements between what is said, how it sounds, and what the face shows. We present ConflictAwareAH, a multimodal framework built for this problem. Three pre-trained encoders extract video, audio, and text representations. Pairwise conflict features -- element-wise absolute differences between modality embeddings -- serve as bidirectional cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary text-guided late fusion strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches 0.694 Macro F1 on the labelled test split and 0.715 on the private leaderboard, outperforming published multimodal baselines by over 10 points -- all on a single GPU in under 25 minutes of training.

Figures & tables

Modality	Backbone	Params	Dim
Video	VideoMAE-Base [ 26 ]	86M	768
Audio	HuBERT-Base-LS960 [ 10 ]	94M	768
Text	RoBERTa-GoEmotions [ 23 , 5 ]	125M	768

Table 1: Encoder backbones. All three output 768-dimensional embeddings and were selected for strong pre-trained representations while keeping the model footprint moderate.

Figure 1: ConflictAwareAH architecture. Video frames, audio waveform, and transcript pass through pre-trained encoders and attention pooling to produce embeddings $\mathbf{v}$ , $\mathbf{a}$ , $\mathbf{t}$ . Three pairwise conflict vectors are concatenated with the modality embeddings and fed to a two-layer FFN, producing the full-fusion logit $\ell_{\mathrm{full}}$ . A parallel text-only head produces $\ell_{\mathrm{text}}$ . At inference the two are blended via Eq. 9 .

Figure 2: Train loss and validation Macro F1 over epochs (Run A).

Split	Videos	A/H (+)	No A/H ( $-$ )
Train	778	385 (49%)	393 (51%)
Val	124	75 (60%)	49 (40%)
Test	525	318 (61%)	207 (39%)
Test (unlabeled)	161	—	—

Table 2: BAH dataset split statistics for the ABAW10 A/H challenge.

Method	Macro F1
BAH: Zero-shot M-LLM (vision only) [ 7 ]	0.283
BAH: Video-FocalNet base [ 7 ]	0.566
BAH: LFAN (V+A+T, co-attention) [ 7 ]	0.593
BAH: Zero-shot M-LLM + transcript [ 7 ]	0.634
Ours — single model (1 or 5 windows)	0.690–0.692
Ours — ensemble (2 ckpts $\times$ 5 win.)	0.694

Table 3: Comparison on the ABAW10 A/H test split (525 videos). BAH baselines are from González et al. [ 7 ] .

Configuration	Macro F1
No conflict features (v+a+t only)	0.681
Window-aligned transcript only	0.698
Text blend $\alpha=0.0$ (full-fusion only)	0.649
Text blend $\alpha=0.5$	0.689
Text blend $\alpha=0.6$ (ours)	0.690
Text blend $\alpha=1.0$ (text-only at inference)	0.700

Table 4: Component ablation on the test split.

Modality	Macro F1	F1-AH	F1-NoAH
Video only	0.534	0.560	0.508
Audio only	0.523	0.645	0.400
Text only	0.737	0.804	0.670
Video + Audio	0.527	0.573	0.481
Video + Text	0.734	0.783	0.685
Audio + Text	0.736	0.789	0.683

Table 5: Modality ablation (validation split, 124 videos).

Setting	Macro F1
Inference: 1 window	0.692
Inference: 3 windows	0.690
Inference: 5 windows (ours)	0.690
Single checkpoint (5 windows)	0.690
Ensemble of 2 checkpoints (5 win.)	0.694

Table 6: Inference settings on the test split.

Explore similar work

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion
Jul 17, 2026Oussama Berhili, Yassine Ouzar, Larbi BoubchirMultimodal Fusion Ambivalence and Hesitancy Recognition
University of Paris 8, Saint-Denis, France
Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion
Mar 15, 2026Aislan Gabriel O. Souza, Agostinho Freire, Leandro Honorato Silva +6Ambivalence and Hesitancy Recognition Multimodal Fusion
Universidade de Pernambuco (UPE), Escola Polit´ecnica de Pernambuco (POLI), Recife, Brazil
Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach
Jul 16, 2026Elena Ryumina, Maxim Markitantov, Alexandr Axyonov +4Multimodal Fusion Ambivalence and Hesitancy Recognition
St. Petersburg Federal Research Center of the Russian Academy of Sciences · St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), St. Petersburg, Russia · HSE University, St. Petersburg, Russia +1

Jul 17, 2026J/K move · Enter open · S save

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

Oussama Berhili, Yassine Ouzar, Larbi Boubchir

University of Paris 8, Saint-Denis, France

Multimodal Fusion Ambivalence and Hesitancy Recognition

Open paper AskPDF arXiv

We present a multimodal framework for Ambivalence/Hesitancy (A/H) recognition in video, developed for the ABAW11 challenge at ECCV 2026. The proposed approach fuses textual, acoustic, and visual modalities extracted from the BAH dataset using three pretrained encoders: F2LLM-v2-0.6B for transcripts (1024-d), WavLM-Large for audio (1024-d), and VideoMAE V2 for facial video (768-d). We first establish comprehensive unimodal baselines using classical classifiers (MLP, Random Forest, GBDT), each optimized via Optuna, and obtain a best unimodal Macro F1 of 0.6659 on the test set using text features alone -- substantially outperforming the zero-shot Video-LLaVA baseline (Macro F1: 0.2827). Building on these baselines, we propose a multimodal fusion architecture that combines bidirectional cross-attention across all three modalities with a Gated Multimodal Unit (GMU), with both architectural and optimization hyperparameters selected through a 50-trial Optuna search. This model achieves a Macro F1 of 0.7394 on the validation set, a relative improvement of 11.0% over the best unimodal baseline, confirming that explicit cross-modal interaction captures complementary cues that no single modality provides in isolation. Final predictions on the official, unlabeled private test set are generated using this model and submitted according to the challenge protocol. Code is publicly available at https://github.com/yassineouzar/IUSD_AH/

Abstract

Explore similar work

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion

Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach