Omni Large Language Models (Omni-LLMs) have demonstrated impressive capabilities in holistic multi-modal perception, yet they consistently falter in complex scenarios requiring synergistic omni-modal reasoning. Beyond understanding global multimodal context, effective reasoning also hinges on fine-grained cross-modal alignment, especially identifying shared referents across modalities, yet this aspect has been largely overlooked. To bridge this gap, we formalize the challenge as a cross-modal coreference problem, where a model must localize a referent in a source modality and re-identify it in a target modality. Building on this paradigm, we introduce CrossOmni, a dataset comprising nine tasks equipped with human-designed reasoning rationales to evaluate and enhance this capability. Experiments on 13 Omni-LLMs reveal systematic weaknesses in cross-modal coreference, which we attribute to the absence of coreference-aware thinking patterns. To address this, we enhance cross-modal alignment via two strategies: a training-free In-Context Learning method and a training-based SFT+GRPO framework designed to induce such thinking patterns. Both approaches yield substantial performance gains and generalize effectively to collaborative reasoning tasks. Overall, our findings highlight cross-modal coreference as a crucial missing piece for advancing robust omni-modal reasoning.
Figures & tables
Figure 1: The comparison between collaborative reasoning and cross-modal coreference alignment. The former focuses on coarse aggregation, whereas the latter targets fine-grained alignment of corresponding referents.
Figure 2: Overview of the CrossOmni dataset. The top section illustrates the annotation pipeline, including pre-process, data annotation, modality alignment, and QA generation. The bottom section presents example instances and the distribution of the test set.
Type
# Train
# Test
# Total
Videos
2,725
1,422
4,147
QA pairs
37,266
2,460
39,726
Table 2: Statistics of the CrossOmni dataset.
Single-modality Coreference Tasks
Audio-only (Au)
What is the emotion in the sentence “why would she do that”? A. calm B. excited C. angry D. sad
Vision-only (Vi)
Where does the third woman enter from? A. front side B. back side C. left side D. right side
Text-only (Te)
What was beckett’s first screenplay? A. the weight of silence B. silent echoes C. the weight of words D. echoes of the past
Cross-modality Coreference Tasks
Audio → Vision (Au → Vi)
What facial expression does the person who spoke with aversion emotion exhibit? A. disoriented look B. calm expression C. serious or concerned expression D. attentive look
Audio → Text (Au → Te)
What is the occupation of the person who said “when did he get out of prison”? A. nurse B. officer C. sergeant D. detective
Table 3: Examples of single-modality and cross-modality coreference tasks. In all cases, the model receives an audio-visual video and a textual biography as input .
Figure 3: Average performance on fixed-source and fixed-target cross-modal questions.
Figure 4: CoT case study on text → audio task. The green denotes important information, the blue denotes the right choice, and the red denotes the error rationales. The model fails to establish the correct bridge between the textual information (the person who experienced the car accident is a man) and the visual information (the man is sitting on the chair), and therefore cannot answer the question correctly. The full case is provide in Figure 6 in Appendix C .
Figure 5: Accuracy comparison of 13 models under default and Chain-of-Thought prompting.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Case study on text → audio task via Qwen3-Omni-30B-A3B-Thinking via CoT. The green denotes important information, the blue denotes the right choice, and the red denotes the error rationales. The model fails to establish the correct bridge between the textual information (the person who experienced the car accident is a man) and the visual information (the man is sitting on the chair), and therefore cannot answer the question correctly. The correct process should first identify Chase in the text and extract that this person is a man, then locate his position in the video, and finally link his speech in the audio.
Figure 7: Case study on visual → audio task via Qwen2.5-Omni-7b after training. The blue option denotes the right choice.
Figure 8: Case study on audio → text task via Qwen2.5-Omni-3b after training. The blue option denotes the right choice.
Prompt
Audio Annotation
Is the voice smooth?
Is the voice rough?
Is the voice nasal?
Is the pitch high-pitched?
Is the pitch low-pitched?
Appendix
Table 13: Prompt for modality annotation.
Prompt
Audio Double Check
Analyze the descriptions and identify any audio that stands out due to unique features such as:
1.Gender: Unusually male or female voice.
2.Emotion: Distinct emotional tone (e.g., anger, sadness).
3.Audio Quality: Notable differences in speed, pitch, or volume.
4.Other Unusual Traits: Background noise, interruptions, etc.
Appendix
Table 14: Prompt for modality annotation double check.
Prompt
QA Generation
Design a question where the model infers factual description from visual actions, using the following information:
- Overall Descriptions
- person biography
- object biography
Consider these aspects:
Appendix
Table 15: Prompt for QA generation. We use the visual → text questions in visual actions as a case study.
Prompt
QA Generation
Task:
You will produce detailed step-by-step reasoning (“Reasoning Details”) that explains and leads to the correct Answer.
Very important constraints:
- You must pretend that ALL useful information comes ONLY from:
(a) what is seen in the video (visual content), and
Appendix
Table 16: Prompt for CoT generation. We use the visual → text questions as a case study.
Prompt
In-Context Learning
Answer the question based on the video and the text information in options A/B/C/D.
You are REQUIRED to first think step by step and write out your reasoning, and THEN give the final answer.
You MUST strictly follow this output format:
<think>
Your detailed step-by-step reasoning in natural language. Explain how you identify the person or object from the question and video, describe its visual appearance, find the Additional Information using that description, and then use the Additional Information to choose the correct option.
Appendix
Table 17: Prompt for in-context learning. We use the visual → text questions as a case study.
Prompt
Contextual Consistency
You are assessing how well the ’hypothesis’ text covers the key information from the ’reference’ text. Differences in wording or extra details in the ’hypothesis’ are fine if the ’reference’s’ main points are included.:
Score based on this coverage:
5 points : Hypothesis clearly and accurately reflects significant core themes or key aspects of the reference. It demonstrates a good understanding of a substantial part of the reference material.
4 points : Hypothesis reflects some important themes or aspects of the reference. The connection is evident, though perhaps not as comprehensive or central as a 5.
2 points : Hypothesis shows a recognizable connection to themes or aspects of the reference, but it might be more superficial, focus on less central points, or only partially grasp a key aspect.
Appendix
Table 18: Prompt for GRPO. We use the visual → text questions as a case study.