cs.CVMar 31, 2026
SaveOmni-MMSI: Toward Identity-attributed Social Interaction Understanding
Organizations: University of Texas at Dallas · Georgia Institute of Technology · University of California, Santa Cruz · University of Illinois Urbana-Champaign
Abstract
We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attributed social cues (e.g., who is speaking what) and reasoning about the social interaction (e.g., whom the speaker refers to). This task is essential for developing AI assistants that can perceive and respond to human interactions. Unlike prior studies that operate on oracle-preprocessed social cues, Omni-MMSI reflects realistic scenarios where AI assistants must perceive and reason from raw data. However, existing pipelines and multi-modal LLMs perform poorly on Omni-MMSI because they lack reliable identity attribution capabilities, which leads to inaccurate social interaction understanding. To address this challenge, we propose Omni-MMSI-R, a reference-guided pipeline that produces identity-attributed social cues with tools and conducts chain-of-thought social reasoning. To facilitate this pipeline, we construct participant-level reference pairs and curate reasoning annotations on top of the existing datasets. Experiments demonstrate that Omni-MMSI-R outperforms advanced LLMs and counterparts on Omni-MMSI. Project page: https://sampson-lee.github.io/omni-mmsi-project-page.
Figures & tables
Figure 1: Overview of the Omni-MMSI task and Omni-MMSI-R pipeline. The Omni-MMSI explores social interaction understanding in a multi-party social scene only using raw audio and video, unlike prior studies that assume identity-attributed social cues are perfectly provided. To address the challenge of attribution, our Omni-MMSI-R is explicitly guided by individual references to generate identity-attributed multi-modal cues and performs CoT reasoning for accurate social interaction understanding.
Figure 2: Illustration of the challenge in Omni-MMSI. The quantitative results (left) show prior pipelines, humans, and advanced Omni-LLMs show substantial accuracy drops when transitioning from oracle cues to raw audio-video input. Typical attribution failures (right), where speech and bounding boxes are mismatched to identities, reveal the weak multi-modal identity attribution of advanced Omni-LLMs.
Figure 3: Overview of the Omni-MMSI-R pipeline. Given a query audio-video segment with multiple participants, the system first retrieves reference audio-vision pairs that represent each individual. Task-specific tools, for transcription, diarization, detection and ReID, generate identity-attributed verbal and non-verbal social cues, specifying who speaks what and where they are. These cues, together with the references and the raw audio-video stream, form the reference-guided input. The Omni-LLM (Qwen2.5 Omni 7B fine-tuned with LoRA) then performs chain-of-thought reasoning over this input to produce an accurate response for social interaction understanding.
Figure 4: Illustration of preparation of reference audio-vision pairs for each participant, which serve as anchors for identity attribution.
Figure 5: Illustration of the construction of CoT datasets.
Figure 6: Qualitative comparison between Gemini2.5 Pro and our proposed Omni-MMSI-R on multi-party situations. Gemini2.5 Pro often misattributes utterances to incorrect visual identities, leading to wrong referent predictions, while Omni-MMSI-R accurately aligns verbal and non-verbal cues with individual references, yielding reliable identity-attributed social cues for social interaction reasoning.
| Pipeline (Ego4D) | STI | PCR | Avg. Acc. |
|---|---|---|---|
| Qwen2.5 Omni 7B [ 90 ] | 26.29 | 28.57 | 27.43 |
| Phi-4-Multimodal [ 1 ] | 14.86 | 8.93 | 11.90 |
| HumanOmni [ 104 ] | 21.71 | 14.29 | 18.00 |
| R1-Omni [ 103 ] | 0.57 | 0.00 | 0.29 |
| OmniVinci [ 92 ] | 24.57 | 37.50 | 31.04 |
| Qwen3 Omni 30B [ 91 ] | 30.86 | 28.57 | 29.72 |
Table 1: Performance comparison of different pipelines on STI and PCR tasks. The upper block reports results (%) on Ego4D, and the lower on YouTube. The results highlight the effectiveness of our Omni-MMSI-R pipeline design for Omni-MMSI.
| Pipeline (Ego4D) | Verbal | Non-Verbal | Avg. |
|---|---|---|---|
| OmniVinci [ 92 ] | 54.04 | 27.42 | 40.73 |
| Qwen3 Omni 30B [ 91 ] | 52.61 | 57.61 | 55.11 |
| Gemini 2.5 Pro [ 18 ] | 44.75 | 26.52 | 35.64 |
| Omni-MMSI-R (ours) | 71.09 | 86.48 | 78.79 |
| Pipeline (YouTube) | Verbal | Non-Verbal | Avg. |
| OmniVinci [ 92 ] | 58.10 | 32.35 | 45.23 |
Table 2: Performance comparison of different pipelines on social cues attribution, including Verbal, Non-Verbal, and Average Attribution accuracy (%). The upper block reports results on Ego4D, and the lower block reports results on YouTube. The results show our Omni-MMSI-R pipeline can achieve better identity attribution.
| Pipeline (Ego4D) | STI | PCR | Avg. Acc. |
|---|---|---|---|
| Qwen2.5 Omni 7B [ 90 ] | 21.23 | 10.71 | 15.97 |
| OmniVinci [ 92 ] | 24.00 | 36.61 | 30.31 |
| Qwen3 Omni 30B [ 91 ] | 28.57 | 39.28 | 33.94 |
| Gemini 2.5 Pro [ 18 ] | 40.57 | 44.64 | 42.61 |
| Omni-MMSI-R (ours) | 40.57 | 45.54 | 43.06 |
| Pipeline (YouTube) | STI | PCR | Avg. Acc. |
Table 3: Performance comparison of different referential pipelines on STI and PCR tasks. The upper block reports results (%) on Ego4D, and the lower on YouTube. The results highlight the effectiveness of our Omni-MMSI-R pipeline design for Omni-MMSI.
| Pipeline (Ego4D) | Verbal | N-Verbal | Avg. |
|---|---|---|---|
| Qwen3 Omni 30B [ 91 ] | 21.92 | 71.26 | 46.59 |
| Gemini 2.5 Pro [ 18 ] | 59.09 | 35.24 | 47.17 |
| Omni-MMSI-R (ours) | 71.09 | 86.48 | 78.79 |
| Pipeline (YouTube) | Verbal | N-Verbal | Avg. |
| Qwen3 Omni 30B [ 91 ] | 23.37 | 50.39 | 36.88 |
| Gemini 2.5 Pro [ 18 ] | 60.61 | 65.45 | 63.03 |
Table 4: Comparison of referential pipelines on social cues attribution, including Verbal, Non-Verbal, and Average Attribution accuracy (%). The upper block reports results on Ego4D, and the lower block on YouTube. The results show Omni-MMSI-R, using tools, provides the strongest attribution performance.
| RA | RV | VC | NC | STI | PCR | Avg. Acc. |
|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | ✗ | 29.19 | 38.68 | 33.97 |
| ✓ | ✓ | ✗ | ✗ | 32.78 | 39.18 | 35.98 |
| ✗ | ✗ | ✓ | ✓ | 37.14 | 41.75 | 39.44 |
| ✓ | ✗ | ✓ | ✗ | 37.89 | 41.78 | 39.84 |
| ✗ | ✓ | ✗ | ✓ | 34.78 | 42.33 | 38.56 |
| ✓ | ✓ | ✓ | ✓ | 40.57 | 45.54 | 43.06 |
Table 5: Effect of different reference-guided input configurations on social interaction understanding (%). RA : Reference Audio, RV : Reference Visual Image, VC : Verbal Cues, NC : Non-Verbal Cues. The results show leveraging audio-visual reference and tool-extracted cue together brings the highest performance.
| Reference | Reasoning | STI | PCR | Avg. Acc. |
| ✗ | ✗ | 29.19 | 38.75 | 33.97 |
| ✗ | ✓ | 30.71 | 40.18 | 35.45 |
| ✓ | None | 36.57 | 42.25 | 39.41 |
| ✓ | 1-step | 36.65 | 42.75 | 39.70 |
| ✓ | 2-step | 40.57 | 45.54 | 43.06 |
| ✓ | 3-step | 29.71 | 39.14 | 34.43 |
Table 6: Effect of CoT reasoning and different granularity on Ego4D. Reference indicates whether reference pairs and attributed social cues are provided. Reasoning controls the level of CoT supervision: None denotes no reasoning; CoT denotes generic reasoning without structured decomposition; 1-step , 2-step , and 3-step represent increasingly fine-grained reasoning strategies. The results show that CoT improves performance even without reference, while combining reference with structured reasoning yields the best results, with 2-step achieving the optimal balance.
Figure 7: Illustration of human study.
Figure 8: Illustration of human filtering.
| YOLO + OSNet | Whisper + SpeechBrain | Qwen2.5 Omni (Answer) | Qwen2.5 Omni (CoT) | |
|---|---|---|---|---|
| Latency | 0.16s | 0.21 real-time factor | 1.05s | 12.69s |
| Parameters | 43.69M + 2.17M | 1541.57M + 22.15M | 8.93B | 8.93B |
Table 7: Latency and parameter size of the components in the Omni-MMSI-R pipeline, measured on an NVIDIA RTX A6000 GPU.
Figure 9: Robustness of the reference-based pipeline under audio and visual degradation. (a) Audio noise degradation evaluates the impact of Gaussian noise on verbal attribution and MMSI tasks. (b) Visual occlusion degradation tests the effect of partial masking on non-verbal attribution and MMSI tasks. The results indicate that our pipeline is highly resilient under audio-visual degradation.
| Setting (Ego4D) | STI | PCR | Avg. Acc. |
|---|---|---|---|
| ZS (w/o ref) | 21.14 | 13.61 | 17.38 |
| SFT (w/o ref) | 26.29 | 32.14 | 29.22 |
| ZS (w/ ref) | 21.23 | 10.71 | 15.97 |
| SFT (w/ ref) | 28.57 | 32.85 | 30.71 |
| SFT (+tool+CoT) | 33.14 | 36.56 | 34.85 |
| Setting (YouTube) | STI | PCR | Avg. Acc. |
Table 8: Comparison of different settings on Ego4D and YouTube (%). The Omni-LLM backbone is Qwen2.5 Omni 3B. ZS , SFT , ref , tool and CoT denote zero-shot inference, supervised fine-tuning, the use of raw reference pairs, tools for extracting identity-attributed social cues and chain-of-thought reasoning supervision.
| Model | STI | PCR | Avg. Acc. |
|---|---|---|---|
| OmniVinci | 31.43 | 12.50 | 21.96 |
| Qwen3 Omni 30B | 30.28 | 21.43 | 25.86 |
| Gemini 2.5 Pro | 36.00 | 33.92 | 34.96 |
| Omni-MMSI-R | 40.57 | 45.54 | 43.06 |
Table 9: Comparison on Ego4D using tool-extracted social cues. Our model significantly outperforms zero-shot Omni-LLMs, indicating that the gains are not solely from access to extracted cues but also from effective task formulation and CoT supervision.
Figure 10: Example of CoT. The CoT performs two key steps-last speaker confirmation and referent inference-to reach the final decision based on reference-based input, leading to reliable prediction.
Figure 11: System prompt for CoT generation.
Figure 12: System prompt for model training.
Figure 13: System prompt for Omni-LLMs evaluation without reference.
Figure 14: System prompt for Omni-LLMs evaluation with reference