Hierarchical Multimodal Interaction Reasoning for Dyadic Relationship Recognition

Wang Tang; Fethiye Irmak Dogan; Linbo Qing; Ce Zhu; Hatice Gunes

cs.CVOct 26, 2026

Hierarchical Multimodal Interaction Reasoning for Dyadic Relationship Recognition

Authors: Wang Tang, Fethiye Irmak Dogan, Linbo Qing, Ce Zhu, Hatice Gunes

Organizations: School of Software Engineering, Chengdu University of Information Technology, Chengdu 610103, China · AFAR Lab, Department of Computer Science and Technology, University of Cambridge, CB3 0FT Cambridge, U.K. · College of Electronics and Information Engineering, Sichuan University, Chengdu 610065, China · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 610051, China

Abstract

Dyadic relationship recognition aims to identify the latent social relationship between two individuals from their multimodal interactions. Existing methods typically project heterogeneous cues into a shared representation and optimise a flat classification objective, overlooking that relationships may rely on different interaction patterns. For instance, distant relationships can often be inferred from direct behavioural correspondence, whereas close relationships are better reflected by coordinated affective, behavioural, and multimodal responses. To address this limitation, we propose HiRe-DRR, a Hierarchical Multimodal Interaction Reasoning framework that organises relationship evidence from three complementary perspectives: feature level, interaction type, and relationship hierarchy. HiRe-DRR decomposes facial, bodily, acoustic, and textual cues into low- and high-level representations and constructs homogeneous and heterogeneous cross-participant interactions. We further introduce the Mixture-of-Interaction Transformer (MoI-Former), which treats diverse interactions as specialised experts and employs relation-aware sparse routing to identify relationship-relevant evidence while preserving contextual interaction information. The proposed Hierarchical Conditional Loss (HC-Loss) exploits the hierarchical nature of social relationships through coarse-to-fine supervision, enabling structured relationship reasoning. Extensive experiments on three dyadic relationship benchmarks demonstrate that HiRe-DRR outperforms recent representative image-, video-, and multimodal methods. Our routing analysis further reveals that distant relationships tend to rely on low-level homogeneous interactions, whereas closer relationships focus on high-level heterogeneous interactions. Our code is publicly available at: https://github.com/tw-repository/HiRe-DRR.

Figures & tables

Fig. 1: Existing SRR paradigms and three challenges of DRR: (a) representative image- and video-based settings; (b) multi-level multimodal cues; (c) homogeneous and heterogeneous interactions; and (d) hierarchical relationship structure. The examples in (a) are from PISC [ 8 ] and MovieGraphs [ 27 ] , while those in (c) are from NoXi [ 28 ] . Acq. , Fri. , and Vgf. denote Acquaintance, Friend, and Very Good Friend, respectively.

Fig. 2: Overview of the proposed HiRe-DRR framework. (a) Input video partition. (b) Low- and high-level feature extraction. (c) Homogeneous and heterogeneous interaction features construction. (d) Relationship reasoning with MoI-Former. (e) Hierarchical Supervision with HC-Loss. The screenshots are taken from the Seamless Interaction dataset [ 47 ] . Note : Acq. , Fri. , Fam. , and Cou. denote Acquaintance, Friend, Family, and Couple, respectively.

Fig. 3: Interaction Feature Construction. At each feature level, the face, body, audio, and text representations of participants $A$ and $B$ are paired to form a $4\times 4$ interaction matrix.

Fig. 4: Overview of the proposed MoI-Former. Low- and high-level homogeneous and heterogeneous interactions are organised into a unified interaction token bank. Relation-aware sparse routing selects the Top- $K$ informative tokens for contextual refinement, while the complete interaction bank remains available as keys and values. Updated tokens are written back to their original positions, and routing is progressively refined across layers. The decoder aggregates the selected interaction tokens into the final relationship representation.

Modality	Low-level Features		High-level Features
Modality	Features	Extractor	Features	Extractor
Face	Facial landmarks, Head pose coordinates	OpenFace 2.0 [ 50 ]	Facial action units, Eye gaze directions, Valence and Arousal, Visual appearance	OpenFace 2.0 [ 50 ] , EmoNet [ 51 ] , TimeSformer [ 52 ]
Body	Body pose coordinates	Keypoint R-CNN [ 53 ]	Body openness and orientation, Visual appearance	Pose-based geometric modelling, TimeSformer [ 52 ]
Audio	20 MFCC, F0, RMS	Librosa [ 54 ]	Acoustic emotion, Spectral flux Speech/Silence duration, Pause duration/frequency, Alpha ratio, Hammarberg index	HuBERT-Large [ 55 ] , Librosa [ 54 ] , openSMILE [ 56 ]
Text	Semantic embedding	BERT-base [ 57 ]	Sentiment of the speech	DeBERTa-base [ 58 ]

TABLE I: Pre-trained models used for feature extraction.

Methods	NoXi-I					NoXi-J					mUAR	mAP
Methods	Str.	Acq.	Fri.	Vgf.	UAR	Str.	Acq.	Fri.	Vgf.	UAR	mUAR	mAP
GR2N [ 14 ]	65.3	49.0	17.7	40.7	43.2 $\pm$ 8.6	57.8	59.9	9.1	28.1	39.7 $\pm$ 10.5	40.9 $\pm$ 8.9	48.3 $\pm$ 9.9
HGM [ 11 ]	65.1	38.9	34.8	48.3	46.7 $\pm$ 4.5	66.2	49.9	15.2	22.6	38.5 $\pm$ 8.6	42.6 $\pm$ 7.6	49.5 $\pm$ 9.6
PGCN [ 12 ]	21.4	19.1	0.0	64.1	26.2 $\pm$ 4.0	19.4	21.8	50.7	17.9	27.5 $\pm$ 7.6	26.8 $\pm$ 6.6	48.9 $\pm$ 6.9
LIReC [ 13 ]	51.6	45.4	19.3	33.5	37.0 $\pm$ 3.1	70.6	45.4	19.0	10.4	36.4 $\pm$ 13.9	36.9 $\pm$ 9.9	49.6 $\pm$ 8.6
CT [ 19 ]	61.6	57.2	33.2	8.6	40.2 $\pm$ 16.3	47.7	57.7	30.0	25.1	40.1 $\pm$ 11.1	40.1 $\pm$ 13.9	51.7 $\pm$ 7.8

TABLE II: Comparison with representative baselines on three dyadic relationship benchmarks.

Configuration	All modalities
Configuration	NoXi	UDIVA	SeamInt
(a) L	43.6 $\pm$ 2.5	51.6 $\pm$ 2.0	39.9 $\pm$ 1.9
(b) H	49.0 $\pm$ 5.6	55.9 $\pm$ 2.0	55.3 $\pm$ 4.2 *
(c) LH+MoI	55.1 $\pm$ 5.0	65.9 $\pm$ 4.5 *	62.4 $\pm$ 2.4
(d) LH+MoI+HC	63.1 $\pm$ 5.6	66.8 $\pm$ 6.4	68.5 $\pm$ 2.6

TABLE III: Ablation results using all modalities.

Configuration	NoXi				UDIVA				SeamInt
Configuration	Face	Body	Audio	Text	Face	Body	Audio	Text	Face	Body	Audio	Text
(a) L	39.7 $\pm$ 8.9	39.9 $\pm$ 8.8	42.2 $\pm$ 6.0	41.4 $\pm$ 1.4	44.4 $\pm$ 4.0	45.4 $\pm$ 4.7	47.4 $\pm$ 5.1	49.1 $\pm$ 3.1	32.8 $\pm$ 0.3	43.2 $\pm$ 2.7	37.5 $\pm$ 1.9	39.3 $\pm$ 9.5
(b) H	45.8 $\pm$ 5.8	47.7 $\pm$ 4.7	47.8 $\pm$ 2.6	47.3 $\pm$ 5.3	52.2 $\pm$ 1.7 *	50.8 $\pm$ 1.0	53.4 $\pm$ 3.2 *	52.8 $\pm$ 1.9	49.8 $\pm$ 3.9 *	50.1 $\pm$ 3.3 *	49.1 $\pm$ 9.4	47.6 $\pm$ 5.2
(c) LH+MoI	51.3 $\pm$ 4.7 *	53.5 $\pm$ 2.1	52.7 $\pm$ 1.4	51.8 $\pm$ 5.9 *	64.3 $\pm$ 8.4	61.6 $\pm$ 4.8	64.7 $\pm$ 5.9 *	64.5 $\pm$ 7.2 *	54.5 $\pm$ 7.5	53.7 $\pm$ 3.3	56.5 $\pm$ 4.3	55.9 $\pm$ 8.5
(d) LH+MoI+HC	61.5 $\pm$ 2.6 *	60.7 $\pm$ 3.5 *	61.3 $\pm$ 5.6	61.0 $\pm$ 5.2	64.6 $\pm$ 3.5 *	62.3 $\pm$ 0.8	65.6 $\pm$ 5.3	65.1 $\pm$ 0.4	63.8 $\pm$ 1.7	65.7 $\pm$ 4.5	64.5 $\pm$ 5.7	65.9 $\pm$ 6.7

TABLE IV: Ablation results by modality.

Fig. 5: Class-wise routing preferences over low- and high-level interactions on NoXi and SeamInt datasets.

Fig. 6: Class-wise routing preferences over homogeneous and heterogeneous interactions on NoXi and SeamInt datasets.

Fig. 7: Class-wise routing preferences of MoI-Former over low- and high-level homogeneous and heterogeneous interactions. F, B, A, and T denote face, body, audio, and text, respectively; darker cells indicate larger mean routing scores.

Configuration	NoXi	UDIVA	SeamInt
Full model	63.1 $\pm$ 5.6	66.8 $\pm$ 6.4	68.5 $\pm$ 2.6
Mask high-preference tokens	61.7 $\pm$ 4.6 (-1.4)	65.0 $\pm$ 6.0 (-1.8)	67.0 $\pm$ 5.4 (-1.5)
Mask low-preference tokens	63.8 $\pm$ 5.0 (+0.7)	67.0 $\pm$ 3.3 (+0.2)	68.8 $\pm$ 3.3 (+0.3)

TABLE V: Token masking based on routing-preference.

Fig. 8: t-SNE projections of relationship representations learned with and without HC-Loss on NoXi and SeamInt datasets.

Fig. 9: Qualitative analysis of MoI-Former routing on NoXi and Seamless Interaction. Cases (a)-(b) are from NoXi, and Cases (c)-(f) are from Seamless Interaction. The routed Top- $5$ interactions reveal relationship-dependent evidence selection, where blue and orange indicate low- and high-level interactions and colour intensity reflects routing preference. Correct and failure cases illustrate the effectiveness and limitations of sparse interaction reasoning under ambiguous relationship patterns. Note: GT: Ground Truth, Pre: Prediction, Conf: Confidence .

Relational Reasoning

Relational reasoning studies how AI systems represent and infer relationships among objects, entities, events, and facts.

See all 35 papers

Inter-Stance: A Dyadic Multimodal Corpus for Conversational Stance Analysis
Apr 24, 2026Xiang Zhang, Xiaotian Li, Taoyue Wang +9Multimodal Learning
State University of New York at Binghamton · Choate Rosemary Hall · Rensselaer Polytechnic Institute +2
PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models
Jun 22, 2026Shuxiang Zhang, Yiting Yin, Wenxuan Song +2Multimodal Large Language Models Multimodal Reasoning
Sun Yat-sen University · University of Michigan · Tsinghua University
Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference
Sep 9, 2026Yaohan Guan, Yen-Ju Lu, Yuzhe Wang +5Spoken Language Understanding Multi-Agent Debate
Center for Language and Speech Processing Department of Electrical and Computer Engineering Johns Hopkins University Baltimore, MD, USA

Apr 24, 2026J/K move · Enter open · S save

Inter-Stance: A Dyadic Multimodal Corpus for Conversational Stance Analysis

Xiang Zhang, Xiaotian Li, Taoyue Wang, Nan Bi+8

State University of New York at Binghamton · Choate Rosemary Hall · Rensselaer Polytechnic Institute+2

Multimodal Learning

Open paper AskPDF arXiv

Social interactions dominate our perceptions of the world and shape our daily behavior by attaching social meaning to acts as simple and spontaneous as gestures, facial expressions, voice, and speech. People mimic and otherwise respond to each other's postures, facial expressions, mannerisms, and other verbal and nonverbal behavior, and form appraisals or evaluations in the process. Yet, no publicly-available dataset includes multimodal recordings and self-report measures of multiple persons in social interaction. Dyadic recordings and annotation are lacking. We present a new data corpus of multimodal dyadic interaction (45 dyads, 90 persons) that includes synchronized multi-modality behavior (2D face video, 3D face geometry, thermal spectrum dynamics, voice and speech behavior, physiology (PPG, EDA, heart-rate, blood pressure, and respiration), and self-reported affect of all participants in a communicative interaction scenario. Two types of dyads are included: persons with shared past history and strangers. Annotations include social signals, agreement, disagreement, and neutral stance. With a potent emotion induction, these multimodal data will enable novel modeling of multimodal interpersonal behavior. We present extensive experiments to evaluate multimodal dyadic communication of dyads with and without interpersonal history, and their affect. This new database will make multimodal modeling of social interaction never possible before. The dataset includes 20TB of multimodal data to share with the research community.

Abstract

Relational Reasoning

Explore similar work

Inter-Stance: A Dyadic Multimodal Corpus for Conversational Stance Analysis

PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship Reasoning in Multimodal Large Language Models

Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference