Understanding Cross-Modal Inconsistency through Representation Subspaces in Omni Models
Organizations: Soongsil University
Abstract
Omni Models process audio, vision, and text within a unified language backbone, yet often produce different answers for semantically equivalent inputs across modalities. Existing representation-level analyses do not distinguish between information that predicts whether disagreement will occur and the representation whose perturbation affects the model's answer. In this paper, we estimate a low-dimensional Answer Subspace from answer-position representations and use it to separate answer-related components from the remaining representation. Across three Omni Models and two multimodal reasoning tasks, information predictive of cross-modal inconsistency lies mainly outside the Answer Subspace, concentrating in a reproducible two-dimensional representation that is not explained by a fixed modality offset. Matched interventions reveal distinct behavioral roles: perturbing the Answer Subspace substantially changes the model's predictions, whereas perturbing the disagreement representation, like a random subspace of the same dimensionality, leaves them almost unchanged. Our key finding is that cross-modal disagreement can be induced by misaligning modality-specific answer representations: perturbing the Answer Subspace for only one modality induces disagreement, whereas applying the same perturbation to both largely preserves agreement. Thus, the representation that predicts disagreement is distinct from the answer-level representation through which disagreement can be induced, showing that linear decodability does not imply behavioral efficacy.
Figures & tables
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Family | 3 blocks | 2 blocks | Overlap |
|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | 0.679 | 0.674 | 0.972 |
| Qwen2.5-Omni | Spa. | 0.704 | 0.687 | 0.985 |
| Gemma-4-E4B | Ext. | 0.718 | 0.712 | 0.988 |
| Gemma-4-E4B | Spa. | 0.681 | 0.662 | 0.989 |
| MiniCPM-o 4.5 | Ext. | 0.723 | 0.723 | 0.978 |
| MiniCPM-o 4.5 | Spa. | 0.639 | 0.638 | 0.991 |
| Model | Family | Layers | Items | Used | Agree | Disagree | Half-dec. layer | Half-dec. depth | Interv. depth |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | 29 | 1350 | 1350 | 1038 | 312 | 15 | 53.600 | 75.000 |
| Qwen2.5-Omni | Spa. | 29 | 465 | 465 | 165 | 300 | 19 | 67.900 | 75.000 |
| Gemma-4-E4B | Ext. | 43 | 1350 | 1350 | 528 | 822 | 18 | 42.900 | 76.200 |
| Gemma-4-E4B | Spa. | 43 | 465 | 465 | 213 | 252 | 24 | 57.100 | 76.200 |
| MiniCPM-o 4.5 | Ext. | 37 | 1350 | 1350 | 864 | 486 | 17 | 47.200 | 75.000 |
| MiniCPM-o 4.5 | Spa. | 37 | 465 | 465 | 141 | 324 | 20 | 55.600 | 75.000 |
| Model | Family | Inside | Compl. | Random | PCA | Full |
|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | 0.510 | 0.704 | 0.511 | 0.518 | 0.705 |
| Qwen2.5-Omni | Spa. | 0.516 | 0.692 | 0.510 | 0.517 | 0.706 |
| Gemma-4-E4B | Ext. | 0.586 | 0.692 | 0.534 | 0.623 | 0.699 |
| Gemma-4-E4B | Spa. | 0.645 | 0.687 | 0.572 | 0.631 | 0.692 |
| MiniCPM-o 4.5 | Ext. | 0.500 | 0.725 | 0.679 | 0.685 | 0.718 |
| MiniCPM-o 4.5 | Spa. | 0.495 | 0.659 | 0.516 | 0.544 | 0.651 |
| Model | Family | Random | Bootstrap align. | Min | Null | |||
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | 0.593 | 0.692 | 0.694 | 0.515 | 0.733 | 0.683 | 0.033 |
| Qwen2.5-Omni | Spa. | 0.666 | 0.694 | 0.691 | 0.512 | 0.725 | 0.632 | 0.031 |
| Gemma-4-E4B | Ext. | 0.649 | 0.670 | 0.664 | 0.535 | 0.713 | 0.674 | 0.037 |
| Gemma-4-E4B | Spa. | 0.629 | 0.660 | 0.674 | 0.566 | 0.719 | 0.670 | 0.039 |
| MiniCPM-o 4.5 | Ext. | 0.625 | 0.721 | 0.718 | 0.568 | 0.724 | 0.693 | 0.029 |
| MiniCPM-o 4.5 | Spa. | 0.611 | 0.639 | 0.641 | 0.507 | 0.747 | 0.700 | 0.028 |
| Agree | Disagree | ||||
|---|---|---|---|---|---|
| Model | Family | Aud. | Img. | Aud. | Img. |
| Qwen2.5-Omni | Ext. | 0.992 | 0.990 | 0.978 | 0.955 |
| Qwen2.5-Omni | Spa. | 0.939 | 0.982 | 0.970 | 0.997 |
| Gemma-4-E4B | Ext. | 0.987 | 0.991 | 0.987 | 0.996 |
| Gemma-4-E4B | Spa. | 0.967 | 0.981 | 0.937 | 0.952 |
| MiniCPM-o 4.5 | Ext. | 0.983 | 0.987 | 0.949 | 0.969 |
| Model behavior | Decoded answer agrees with | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Family | Content | Letter | Other | Model | Model content | Model letter | |
| Qwen2.5-Omni | Ext. | 2700 | 0.919 | 0.028 | 0.053 | 0.985 | 0.991 | 0.895 |
| Qwen2.5-Omni | Spa. | 1320 | 0.618 | 0.175 | 0.207 | 0.948 | 0.971 | 0.935 |
| Gemma-4-E4B | Ext. | 2700 | 0.632 | 0.184 | 0.184 | 0.984 | 0.989 | 0.978 |
| Gemma-4-E4B | Spa. | 1320 | 0.366 | 0.305 | 0.329 | 0.967 | 0.969 | 0.970 |
| MiniCPM-o 4.5 | Ext. | 2700 | 0.880 | 0.043 | 0.076 | 0.985 | 0.990 | 0.966 |
| Pair distance | Modality | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Family | Agree | Disagree | Ratio | Excess/dim | Separability | answer | disagree |
| Qwen2.5-Omni | Ext. | 0.170 | 0.225 | 1.324 | 7.310 | 1.000 | 0.054 | 0.023 |
| Qwen2.5-Omni | Spa. | 0.173 | 0.218 | 1.260 | 3.170 | 1.000 | 0.034 | 0.023 |
| Gemma-4-E4B | Ext. | 0.681 | 0.931 | 1.367 | 8.620 | 1.000 | 0.065 | 0.036 |
| Gemma-4-E4B | Spa. | 0.841 | 0.899 | 1.069 | 4.750 | 1.000 | 0.040 | 0.060 |
| MiniCPM-o 4.5 | Ext. | 0.892 | 1.060 | 1.188 | 2.920 | 1.000 | 0.046 | 0.017 |
| Model | Family | Strata | Unrestr. | Matched | Null | |
|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | 4 | 675 | 0.635 | 0.670 | 0.509 |
| Qwen2.5-Omni | Spa. | 4 | 233 | 0.558 | 0.726 | 0.587 |
| Gemma-4-E4B | Ext. | 4 | 675 | 0.642 | 0.712 | 0.528 |
| Gemma-4-E4B | Spa. | 2 | 157 | 0.648 | 0.703 | 0.586 |
| MiniCPM-o 4.5 | Ext. | 4 | 675 | 0.683 | 0.697 | 0.534 |
| MiniCPM-o 4.5 | Spa. | 2 | 171 | 0.605 | 0.711 | 0.524 |
| Model | Family | Disagree | Dist. disagree | Dist. agree | Ratio | |
|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | 1350 | 312 | 2.199 | 0.418 | 5.263 |
| Qwen2.5-Omni | Spa. | 465 | 300 | 2.694 | 1.117 | 2.413 |
| Gemma-4-E4B | Ext. | 1350 | 822 | 2.877 | 0.573 | 5.021 |
| Gemma-4-E4B | Spa. | 465 | 252 | 2.247 | 0.913 | 2.460 |
| MiniCPM-o 4.5 | Ext. | 1350 | 486 | 2.113 | 0.896 | 2.358 |
| MiniCPM-o 4.5 | Spa. | 465 | 324 | 2.615 | 1.362 | 1.921 |
| Subspace | Qwen Ext. | Qwen Spa. | Gemma Ext. | Gemma Spa. | MiniCPM Ext. | MiniCPM Spa. | |
|---|---|---|---|---|---|---|---|
| Answer | 2 | 0.040 | 0.065 | 0.020 | 0.110 | 0.105 | 0.085 |
| 5 | 0.090 | 0.135 | 0.075 | 0.275 | 0.180 | 0.195 | |
| 10 | 0.170 | 0.250 | 0.295 | 0.500 | 0.740 | 0.810 | |
| 20 | 0.640 | 0.485 | 0.860 | 0.760 | 0.900 | 0.995 | |
| 40 | 0.980 | 0.895 | 0.980 | 0.825 | 1.000 | 0.995 | |
| Disagreement | 2 | 0.000 | 0.005 | 0.000 | 0.010 | 0.005 | 0.005 |
| At intervention depth | Before answer forms | |||||||
| Answer | Disagree | Random | Answer | Disagree | ||||
| Model | Family | |||||||
| Qwen2.5-Omni | Ext. | 28 | 98 | 1 | 0 | 0 | 0 | |
| Qwen2.5-Omni | Spa. | 45 | 96 | 6 | 2 | 1 | 2 | |
| Gemma-4-E4B | Ext. | 36 | 100 | 0 | 1 | 1 | 0 | |
| Gemma-4-E4B | Spa. | 56 | 89 | 6 | 18 | 2 | 9 | |
| Changed at | Settling depth (%) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Family | Agree | Disagree | Agree | Disagree | Agree | Disagree | |||
| Qwen2.5-Omni | Ext. | 69 | 31 | 0.159 | 0.548 | 0.389 | 49.900 | 57.300 | 7.400 | |
| Qwen2.5-Omni | Spa. | 34 | 66 | 0.471 | 0.439 | -0.031 | 59.600 | 59.600 | 0.000 | |
| Gemma-4-E4B | Ext. | 59 | 41 | 0.390 | 0.317 | -0.073 | 50.800 | 49.900 | -0.900 | |
| Gemma-4-E4B | Spa. | 47 | 53 | 0.426 | 0.679 | 0.254 | 58.200 | 62.900 | 4.700 | |
| MiniCPM-o 4.5 | Ext. | 70 | 30 | 0.714 | 0.833 | 0.119 | 48.000 | 52.900 | 4.900 | |
| Model | Family | Disagree ( ) | Random ( ) | Random ( , b) | Answer ( ) | Random ( ) |
|---|---|---|---|---|---|---|
| Qwen2.5-Omni | Ext. | -0.002 | +0.002 | +0.008 | -0.103 | -0.033 |
| Qwen2.5-Omni | Spa. | +0.002 | -0.000 | -0.000 | -0.041 | -0.009 |
| Gemma-4-E4B | Ext. | +0.008 | +0.043 | -0.027 | -0.054 | +0.006 |
| Gemma-4-E4B | Spa. | +0.003 | -0.004 | -0.007 | +0.039 | -0.002 |
| MiniCPM-o 4.5 | Ext. | -0.015 | +0.006 | -0.013 | -0.416 | -0.019 |
| MiniCPM-o 4.5 | Spa. | -0.005 | -0.001 | -0.011 | -0.027 | -0.018 |
| Asymmetric | Symmetric control | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Family | |||||||||||
| Qwen2.5-Omni | Ext. | 60 | 0.000 | 0.000 | 0.017 | 0.500 | 0.983 | 0.033 | 0.083 | 0.183 | 0.300 | 0.050 |
| Qwen2.5-Omni | Spa. | 60 | 0.033 | 0.033 | 0.100 | 0.367 | 0.833 | 0.200 | 0.300 | 0.550 | 0.600 | 0.233 |
| Gemma-4-E4B | Ext. | 60 | 0.000 | 0.000 | 0.200 | 0.817 | 0.950 | 0.033 | 0.133 | 0.383 | 0.183 | 0.083 |
| Gemma-4-E4B | Spa. | 60 | 0.000 | 0.217 | 0.400 | 0.700 | 0.767 | 0.067 | 0.283 | 0.433 | 0.550 | 0.250 |
| MiniCPM-o 4.5 | Ext. | 60 | 0.017 | 0.033 | 0.717 | 0.950 | 1.000 | 0.083 | 0.217 | 0.233 | 0.200 | 0.100 |
Multimodal Model Interpretability
Multimodal model interpretability studies how AI systems combine and use images, text, audio, video, and other signals, and how to explain their decisions.
See all 141 papers