Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
Authors: Xinpeng Li, Bolin Lai, Hardy Chen, Shijian Deng, Cihang Xie, Yuyin Zhou, James Matthew Rehg, Yapeng Tian
Organizations: University of Texas at Dallas · Georgia Institute of Technology · University of California, Santa Cruz · University of Illinois Urbana-Champaign
We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attributed social cues (e.g., who is speaking what) and reasoning about the social interaction (e.g., whom the speaker refers to). This task is essential for developing AI assistants that can perceive and respond to human interactions. Unlike prior studies that operate on oracle-preprocessed social cues, Omni-MMSI reflects realistic scenarios where AI assistants must perceive and reason from raw data. However, existing pipelines and multi-modal LLMs perform poorly on Omni-MMSI because they lack reliable identity attribution capabilities, which leads to inaccurate social interaction understanding. To address this challenge, we propose Omni-MMSI-R, a reference-guided pipeline that produces identity-attributed social cues with tools and conducts chain-of-thought social reasoning. To facilitate this pipeline, we construct participant-level reference pairs and curate reasoning annotations on top of the existing datasets. Experiments demonstrate that Omni-MMSI-R outperforms advanced LLMs and counterparts on Omni-MMSI. Project page: https://sampson-lee.github.io/omni-mmsi-project-page.
Figures & tables
Figure 1: Overview of the Omni-MMSI task and Omni-MMSI-R pipeline. The Omni-MMSI explores social interaction understanding in a multi-party social scene only using raw audio and video, unlike prior studies that assume identity-attributed social cues are perfectly provided. To address the challenge of attribution, our Omni-MMSI-R is explicitly guided by individual references to generate identity-attributed multi-modal cues and performs CoT reasoning for accurate social interaction understanding.
Figure 2: Illustration of the challenge in Omni-MMSI. The quantitative results (left) show prior pipelines, humans, and advanced Omni-LLMs show substantial accuracy drops when transitioning from oracle cues to raw audio-video input. Typical attribution failures (right), where speech and bounding boxes are mismatched to identities, reveal the weak multi-modal identity attribution of advanced Omni-LLMs.
Figure 3: Overview of the Omni-MMSI-R pipeline. Given a query audio-video segment with multiple participants, the system first retrieves reference audio-vision pairs that represent each individual. Task-specific tools, for transcription, diarization, detection and ReID, generate identity-attributed verbal and non-verbal social cues, specifying who speaks what and where they are. These cues, together with the references and the raw audio-video stream, form the reference-guided input. The Omni-LLM (Qwen2.5 Omni 7B fine-tuned with LoRA) then performs chain-of-thought reasoning over this input to produce an accurate response for social interaction understanding.
Figure 4: Illustration of preparation of reference audio-vision pairs for each participant, which serve as anchors for identity attribution.
Figure 5: Illustration of the construction of CoT datasets.
Figure 6: Qualitative comparison between Gemini2.5 Pro and our proposed Omni-MMSI-R on multi-party situations. Gemini2.5 Pro often misattributes utterances to incorrect visual identities, leading to wrong referent predictions, while Omni-MMSI-R accurately aligns verbal and non-verbal cues with individual references, yielding reliable identity-attributed social cues for social interaction reasoning.
Pipeline (Ego4D)
STI
PCR
Avg. Acc.
Qwen2.5 Omni 7B [ 90 ]
26.29
28.57
27.43
Phi-4-Multimodal [ 1 ]
14.86
8.93
11.90
HumanOmni [ 104 ]
21.71
14.29
18.00
R1-Omni [ 103 ]
0.57
0.00
0.29
OmniVinci [ 92 ]
24.57
37.50
31.04
Qwen3 Omni 30B [ 91 ]
30.86
28.57
29.72
Table 1: Performance comparison of different pipelines on STI and PCR tasks. The upper block reports results (%) on Ego4D, and the lower on YouTube. The results highlight the effectiveness of our Omni-MMSI-R pipeline design for Omni-MMSI.
Pipeline (Ego4D)
Verbal
Non-Verbal
Avg.
OmniVinci [ 92 ]
54.04
27.42
40.73
Qwen3 Omni 30B [ 91 ]
52.61
57.61
55.11
Gemini 2.5 Pro [ 18 ]
44.75
26.52
35.64
Omni-MMSI-R (ours)
71.09
86.48
78.79
Pipeline (YouTube)
Verbal
Non-Verbal
Avg.
OmniVinci [ 92 ]
58.10
32.35
45.23
Table 2: Performance comparison of different pipelines on social cues attribution, including Verbal, Non-Verbal, and Average Attribution accuracy (%). The upper block reports results on Ego4D, and the lower block reports results on YouTube. The results show our Omni-MMSI-R pipeline can achieve better identity attribution.
Pipeline (Ego4D)
STI
PCR
Avg. Acc.
Qwen2.5 Omni 7B [ 90 ]
21.23
10.71
15.97
OmniVinci [ 92 ]
24.00
36.61
30.31
Qwen3 Omni 30B [ 91 ]
28.57
39.28
33.94
Gemini 2.5 Pro [ 18 ]
40.57
44.64
42.61
Omni-MMSI-R (ours)
40.57
45.54
43.06
Pipeline (YouTube)
STI
PCR
Avg. Acc.
Table 3: Performance comparison of different referential pipelines on STI and PCR tasks. The upper block reports results (%) on Ego4D, and the lower on YouTube. The results highlight the effectiveness of our Omni-MMSI-R pipeline design for Omni-MMSI.
Pipeline (Ego4D)
Verbal
N-Verbal
Avg.
Qwen3 Omni 30B [ 91 ]
21.92
71.26
46.59
Gemini 2.5 Pro [ 18 ]
59.09
35.24
47.17
Omni-MMSI-R (ours)
71.09
86.48
78.79
Pipeline (YouTube)
Verbal
N-Verbal
Avg.
Qwen3 Omni 30B [ 91 ]
23.37
50.39
36.88
Gemini 2.5 Pro [ 18 ]
60.61
65.45
63.03
Table 4: Comparison of referential pipelines on social cues attribution, including Verbal, Non-Verbal, and Average Attribution accuracy (%). The upper block reports results on Ego4D, and the lower block on YouTube. The results show Omni-MMSI-R, using tools, provides the strongest attribution performance.
RA
RV
VC
NC
STI
PCR
Avg. Acc.
✗
✗
✗
✗
29.19
38.68
33.97
✓
✓
✗
✗
32.78
39.18
35.98
✗
✗
✓
✓
37.14
41.75
39.44
✓
✗
✓
✗
37.89
41.78
39.84
✗
✓
✗
✓
34.78
42.33
38.56
✓
✓
✓
✓
40.57
45.54
43.06
Table 5: Effect of different reference-guided input configurations on social interaction understanding (%). RA : Reference Audio, RV : Reference Visual Image, VC : Verbal Cues, NC : Non-Verbal Cues. The results show leveraging audio-visual reference and tool-extracted cue together brings the highest performance.
Reference
Reasoning
STI
PCR
Avg. Acc.
✗
✗
29.19
38.75
33.97
✗
✓
30.71
40.18
35.45
✓
None
36.57
42.25
39.41
✓
1-step
36.65
42.75
39.70
✓
2-step
40.57
45.54
43.06
✓
3-step
29.71
39.14
34.43
Table 6: Effect of CoT reasoning and different granularity on Ego4D. Reference indicates whether reference pairs and attributed social cues are provided. Reasoning controls the level of CoT supervision: None denotes no reasoning; CoT denotes generic reasoning without structured decomposition; 1-step , 2-step , and 3-step represent increasingly fine-grained reasoning strategies. The results show that CoT improves performance even without reference, while combining reference with structured reasoning yields the best results, with 2-step achieving the optimal balance.
Figure 7: Illustration of human study.
Figure 8: Illustration of human filtering.
YOLO + OSNet
Whisper + SpeechBrain
Qwen2.5 Omni (Answer)
Qwen2.5 Omni (CoT)
Latency
0.16s
0.21 real-time factor
1.05s
12.69s
Parameters
43.69M + 2.17M
1541.57M + 22.15M
8.93B
8.93B
Table 7: Latency and parameter size of the components in the Omni-MMSI-R pipeline, measured on an NVIDIA RTX A6000 GPU.
Figure 9: Robustness of the reference-based pipeline under audio and visual degradation. (a) Audio noise degradation evaluates the impact of Gaussian noise on verbal attribution and MMSI tasks. (b) Visual occlusion degradation tests the effect of partial masking on non-verbal attribution and MMSI tasks. The results indicate that our pipeline is highly resilient under audio-visual degradation.
Setting (Ego4D)
STI
PCR
Avg. Acc.
ZS (w/o ref)
21.14
13.61
17.38
SFT (w/o ref)
26.29
32.14
29.22
ZS (w/ ref)
21.23
10.71
15.97
SFT (w/ ref)
28.57
32.85
30.71
SFT (+tool+CoT)
33.14
36.56
34.85
Setting (YouTube)
STI
PCR
Avg. Acc.
Table 8: Comparison of different settings on Ego4D and YouTube (%). The Omni-LLM backbone is Qwen2.5 Omni 3B. ZS , SFT , ref , tool and CoT denote zero-shot inference, supervised fine-tuning, the use of raw reference pairs, tools for extracting identity-attributed social cues and chain-of-thought reasoning supervision.
Model
STI
PCR
Avg. Acc.
OmniVinci
31.43
12.50
21.96
Qwen3 Omni 30B
30.28
21.43
25.86
Gemini 2.5 Pro
36.00
33.92
34.96
Omni-MMSI-R
40.57
45.54
43.06
Table 9: Comparison on Ego4D using tool-extracted social cues. Our model significantly outperforms zero-shot Omni-LLMs, indicating that the gains are not solely from access to extracted cues but also from effective task formulation and CoT supervision.
Figure 10: Example of CoT. The CoT performs two key steps-last speaker confirmation and referent inference-to reach the final decision based on reference-based input, leading to reliable prediction.
Figure 11: System prompt for CoT generation.
Figure 12: System prompt for model training.
Figure 13: System prompt for Omni-LLMs evaluation without reference.
Figure 14: System prompt for Omni-LLMs evaluation with reference
Evaluating omni-modal large language models (OLMs) in multi-party dialogue requires more than answer correctness on pre-segmented inputs. We introduce SocialOmni, an offline diagnostic benchmark that separates three turn-level decisions: identifying who is speaking, deciding when a designated participant should enter at an annotated query time, and determining how that participant should continue the dialogue. SocialOmni contains 2,000 perception items and a quality-controlled core split of 200 interaction-generation items, including naturally occurring speaker-visibility mismatches. Each item is independently checked by three human annotators. Evaluated systems receive only query-time-bounded multimodal evidence, while manually verified reference continuations are reserved for response judging. A complete three-judge ensemble scores every eligible response, with leave-one-judge-out and family-sensitivity audits. Across 11 OLMs, rankings vary substantially by axis, and coverage-adjusted scores reveal when high conditional response quality depends on selective turn entry. The protocol does not measure persistent streaming state or wall-clock latency.
Tianyu Xie, Jinfa Huang, Yuexiao Ma +11
Xiamen University · University of Rochester · Nanyang Technological University +1
Training Multimodal Large Language Models for audio-visual social understanding is a crucial step toward embodied social intelligence. Chain-of-thought (CoT) reasoning has become the dominant approach, with HumanOmniV2 and its IntentBench benchmark as a prominent reference point. In this context, we report three findings. First, IntentBench is highly noisy: ∼7% of questions are broken and ∼23% are trivially answerable without the video input. We remove the affected questions and release Intentbench-Prime. Second, current reasoning approaches are expensive and surprisingly ineffective. A simple Vanilla SFT baseline matches or outperforms existing reasoning methods across three benchmarks at a fraction of the cost, establishing it as an essential baseline for evaluating novel fine-tuning techniques. Third, our analysis reveals that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT. These surprising findings reveal the limitations of current MLLMs when it comes to social understanding. IntentBench-Prime, Vanilla SFT model, and code are publicly available.
Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera +1
Inria at Univ. Grenoble Alpes, CNRS, LJK, France · University of Twente, The Netherlands
Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL CAPTION, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to generate relevant information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges demonstrate a path towards scaling automated evaluation of multimodal social understanding.
Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe +1
School of Computer Science, Carnegie Mellon University