cs.CLNov 17, 2026

Understanding Cross-Modal Inconsistency through Representation Subspaces in Omni Models

Authors: Seonghyeon Cho, Chanjun Park

Organizations: Soongsil University

Abstract

Omni Models process audio, vision, and text within a unified language backbone, yet often produce different answers for semantically equivalent inputs across modalities. Existing representation-level analyses do not distinguish between information that predicts whether disagreement will occur and the representation whose perturbation affects the model's answer. In this paper, we estimate a low-dimensional Answer Subspace from answer-position representations and use it to separate answer-related components from the remaining representation. Across three Omni Models and two multimodal reasoning tasks, information predictive of cross-modal inconsistency lies mainly outside the Answer Subspace, concentrating in a reproducible two-dimensional representation that is not explained by a fixed modality offset. Matched interventions reveal distinct behavioral roles: perturbing the Answer Subspace substantially changes the model's predictions, whereas perturbing the disagreement representation, like a random subspace of the same dimensionality, leaves them almost unchanged. Our key finding is that cross-modal disagreement can be induced by misaligning modality-specific answer representations: perturbing the Answer Subspace for only one modality induces disagreement, whereas applying the same perturbation to both largely preserves agreement. Thus, the representation that predicts disagreement is distinct from the answer-level representation through which disagreement can be induced, showing that linear decodability does not imply behavioral efficacy.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

    Mar 28, 2026Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan ParvezModality Reliance in Multimodal ModelsMultimodal Model Evaluation

  2. Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

    Sep 29, 2026Yueran Ma, Ronghao LinAudio QAMultimodal Large Language Models

  3. Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs

    May 13, 2026Trung Nguyen Quang, Yiming Gao, Fanyi Pu +3Multimodal GroundingMultimodal Large Language Models