Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts

Yen-Ting Piao; Shu-Yun Chen; Chin-Hui Chu; Chun-Wei Chen; Shih-Yun Shan Kuan; Hung-yi Lee; Yun-Nung Chen

cs.CLSep 5, 2026

Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts

Authors: Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen

Abstract

Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e.g., a photograph or recording of a dog) and propositional signals (e.g., the declarative claim "this is a dog"), such that any measured modality bias is inherently confounded with evidence-form bias, precluding clean attribution to either source. To address this, we introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Evaluating five OLLMs, we find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio. Further analyses via layer-wise linear probing and contrastive decoding reveal that modality bias is already linearly decodable from early representation layers and can only be partially mitigated, calling for mitigation strategies beyond surface-level interventions.

Figures & tables

Figure 1: Tri-modal conflict with perceptual (direct sensory) or propositional (declarative) evidence in vision and audio, and propositional evidence in text; cross-modal disagreement may induce modality bias.

Figure 2: Overview of the Tri-PvP dataset construction pipeline. Perceptual data are sourced from existing datasets, while propositional data are generated from a sentence template pool via a text renderer and a TTS model. Each resulting sample consists of a vision input (perceptual or propositional), an audio input (perceptual or propositional), a text input, and a question.

Figure 3: Modality bias distribution across five models under four evidence-type conditions, with detailed statistics in Appendix G . Each bar shows the percentage of responses assigned to each label in

\mathcal{Y}

(see Sec. 4.2 for definitions).

	Modality	Position	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
Qwen2.5	Image	Front	51.5	49.4	20.9	12.2
		Mid	57.0	44.8	32.5	15.9
		End	60.2	55.7	45.4	44.0
	Audio	Front	3.0	7.3	2.6	4.5
		Mid	2.6	7.6	1.8	4.3
		End	4.1	8.5	6.5	16.0

Table 1: Bias rate (%) of each modality when placed at Front / Mid / End position of the prompt across evidence-type conditions, with detailed statistics in Appendix G . Bold indicates the position with the maximum value for each (modality, evidence-type) pair.

Panel of Figure 4 — Figure 4: Layer-wise balanced accuracy of linear probes on two models across evidence-type conditions. The dashed line marks the random baseline ( $50\%$ ).

Figure 5: Modality bias distribution (%) under baseline and contrastive decoding (CD) on Gemma4.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Modality	Source
Animal	Image	ImageNet Deng et al. (2009)
Animal	Audio	ESC-50 Piczak (2015)
Emotion	Image	KDEF Calvo and Lundqvist (2008)
Emotion	Audio	CREMA-D Cao et al. (2014)
Environment	Image	Open Images V7 Kuznetsova et al. (2020)
Environment	Audio	ESC-50 Piczak (2015)

Appendix

Table 3: Source datasets used for perceptual images and audios in each domain of Tri-PvP .

	Perc I -Perc A			Perc I -Prop A			Prop I -Perc A			Prop I -Prop A
Model	Conf.	M-Rep.	Ref.	Conf.	M-Rep.	Ref.	Conf.	M-Rep.	Ref.	Conf.	M-Rep.	Ref.
Qwen2.5	48.17	1.83	93.90	91.00	2.84	85.07	84.46	2.67	92.78	75.71	71.55	84.28
MiniCPM	53.49	39.53	2.33	69.77	13.18	10.08	69.23	12.82	3.08	48.34	32.47	2.21
Qwen3	0.00	50.00	0.00	100.00	0.00	0.00	83.33	16.66	33.33	20.00	96.67	0.00
Gemma4	2.56	0.00	94.87	18.18	0.00	78.79	20.18	57.80	33.03	21.68	71.33	25.87
Gemini3	88.00	76.00	0.00	80.00	0.00	0.00	93.75	53.13	3.13	61.47	83.55	1.30

Appendix

Table 4: Finer-grained breakdown of NO_BIAS responses along three independent binary dimensions (Conflict (Conf.), Multi-report (M-Rep.), and Refusal (Ref.)) across models and evidence-type conditions. Values are percentages of NO_BIAS responses within each (model, condition) pair that exhibit each dimension. A single response may exhibit more than one dimension simultaneously.

Bias Type	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
Order: I $\rightarrow$ A $\rightarrow$ T
BIAS_IMAGE	52.7 $\pm$ 2.2	49.7 $\pm$ 2.2	24.8 $\pm$ 1.9	12.7 $\pm$ 1.5
BIAS_AUDIO	2.9 $\pm$ 0.7	7.2 $\pm$ 1.1	2.2 $\pm$ 0.6	5.9 $\pm$ 1.0
BIAS_TEXT	29.9 $\pm$ 2.0	15.0 $\pm$ 1.6	36.2 $\pm$ 2.1	18.5 $\pm$ 1.7
BIAS_IMAGE_AUDIO	0.3 $\pm$ 0.2	0.1 $\pm$ 0.1	0.2 $\pm$ 0.2	0.1 $\pm$ 0.1
BIAS_IMAGE_TEXT	0.8 $\pm$ 0.4	0.3 $\pm$ 0.2	4.1 $\pm$ 0.9	0.2 $\pm$ 0.2

Appendix

Table 5: Modality bias distribution (%) for Qwen2.5 across different evidence-type conditions and input orders. Modality input orders are denoted sequentially (e.g.,

I\rightarrow A\rightarrow T

represents Image followed by Audio, then Text).

Bias Type	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
Order: I $\rightarrow$ A $\rightarrow$ T
BIAS_IMAGE	60.3 $\pm$ 2.1	52.4 $\pm$ 2.2	74.4 $\pm$ 1.9	74.9 $\pm$ 1.9
BIAS_AUDIO	0.9 $\pm$ 0.4	25.4 $\pm$ 1.9	2.7 $\pm$ 0.7	5.4 $\pm$ 1.0
BIAS_TEXT	25.1 $\pm$ 1.9	9.0 $\pm$ 1.3	8.7 $\pm$ 1.2	5.1 $\pm$ 1.0
BIAS_IMAGE_AUDIO	0.1 $\pm$ 0.1	0.2 $\pm$ 0.2	0.2 $\pm$ 0.2	0.3 $\pm$ 0.2
BIAS_IMAGE_TEXT	0.2 $\pm$ 0.2	0.1 $\pm$ 0.1	1.4 $\pm$ 0.5	0.3 $\pm$ 0.2

Appendix

Table 6: Modality bias distribution (%) for Gemma4 across different evidence-type conditions and input orders. Modality input orders are denoted sequentially (e.g.,

I\rightarrow A\rightarrow T

represents Image followed by Audio, then Text).

Bias Type	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
BIAS_IMAGE	73.4 $\pm$ 1.9	79.2 $\pm$ 1.8	40.1 $\pm$ 2.1	68.0 $\pm$ 2.0
BIAS_AUDIO	4.8 $\pm$ 0.9	5.4 $\pm$ 1.0	17.2 $\pm$ 1.7	5.5 $\pm$ 1.0
BIAS_TEXT	9.8 $\pm$ 1.3	8.0 $\pm$ 1.2	37.1 $\pm$ 2.1	22.5 $\pm$ 1.8
BIAS_IMAGE_AUDIO	2.5 $\pm$ 0.7	0.2 $\pm$ 0.2	0.1 $\pm$ 0.1	0.0 $\pm$ 0.0
BIAS_IMAGE_TEXT	0.3 $\pm$ 0.2	0.0 $\pm$ 0.0	0.6 $\pm$ 0.3	0.2 $\pm$ 0.2
BIAS_AUDIO_TEXT	0.0 $\pm$ 0.0	0.3 $\pm$ 0.2	0.1 $\pm$ 0.1	0.7 $\pm$ 0.4

Appendix

Table 7: Modality bias distribution (%) for Qwen3 across evidence-type conditions.

Bias Type	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
BIAS_IMAGE	59.1 $\pm$ 2.2	63.5 $\pm$ 2.1	47.1 $\pm$ 2.2	43.9 $\pm$ 2.2
BIAS_AUDIO	5.9 $\pm$ 1.0	8.5 $\pm$ 1.2	12.3 $\pm$ 1.4	12.8 $\pm$ 1.5
BIAS_TEXT	20.4 $\pm$ 1.8	11.6 $\pm$ 1.4	22.3 $\pm$ 1.8	19.0 $\pm$ 1.7
BIAS_IMAGE_AUDIO	0.3 $\pm$ 0.2	0.3 $\pm$ 0.2	0.0 $\pm$ 0.0	0.6 $\pm$ 0.3
BIAS_IMAGE_TEXT	0.1 $\pm$ 0.1	0.2 $\pm$ 0.2	0.6 $\pm$ 0.3	0.7 $\pm$ 0.4
BIAS_AUDIO_TEXT	0.1 $\pm$ 0.1	0.7 $\pm$ 0.4	0.1 $\pm$ 0.1	1.6 $\pm$ 0.5

Appendix

Table 8: Modality bias distribution (%) for MiniCPM across evidence-type conditions.

Bias Type	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
BIAS_IMAGE	68.9 $\pm$ 2.0	85.1 $\pm$ 1.6	37.6 $\pm$ 2.1	73.6 $\pm$ 1.9
BIAS_AUDIO	4.2 $\pm$ 0.9	2.0 $\pm$ 0.6	17.2 $\pm$ 1.7	1.1 $\pm$ 0.5
BIAS_TEXT	9.2 $\pm$ 1.3	2.7 $\pm$ 0.7	33.6 $\pm$ 2.1	4.6 $\pm$ 0.9
BIAS_IMAGE_AUDIO	7.3 $\pm$ 1.1	0.7 $\pm$ 0.4	1.4 $\pm$ 0.5	2.3 $\pm$ 0.7
BIAS_IMAGE_TEXT	3.2 $\pm$ 0.8	0.6 $\pm$ 0.3	3.7 $\pm$ 0.8	0.7 $\pm$ 0.4
BIAS_AUDIO_TEXT	0.3 $\pm$ 0.2	0.7 $\pm$ 0.4	0.4 $\pm$ 0.3	0.1 $\pm$ 0.1

Appendix

Table 9: Modality bias distribution (%) for Gemini3 across evidence-type conditions.

Bias Type	Method	Perc I - Perc A	Perc I - Prop A	Prop I - Perc A	Prop I - Prop A
BIAS_IMAGE	Baseline	60.30 $\pm$ 2.1	52.4 $\pm$ 2.2	74.4 $\pm$ 1.9	74.9 $\pm$ 1.9
BIAS_IMAGE	CD	47.6 $\pm$ 2.2	31.3 $\pm$ 2.0	39.6 $\pm$ 2.1	38.8 $\pm$ 2.1
BIAS_AUDIO	Baseline	0.9 $\pm$ 0.4	25.4 $\pm$ 1.9	2.7 $\pm$ 0.7	5.4 $\pm$ 1.0
BIAS_AUDIO	CD	1.1 $\pm$ 0.5	16.6 $\pm$ 1.6	1.3 $\pm$ 0.5	5.5 $\pm$ 1.0
BIAS_TEXT	Baseline	25.1 $\pm$ 1.9	9.0 $\pm$ 1.3	8.7 $\pm$ 1.2	5.1 $\pm$ 1.0
BIAS_TEXT	CD	34.0 $\pm$ 2.1	22.5 $\pm$ 1.8	12.4 $\pm$ 1.4	7.3 $\pm$ 1.1

Appendix

Table 10: Detailed bias-type distribution (%) for the baseline and CD (Contrastive Decoding) on Gemma 4 across evidence-type conditions.

Figure 6: An example of Gemma4’s responses under baseline and CD (Prop I -Prop A condition; image label: piano, audio label: flute, text label: clarinet).

Split	$N$	Baseline (%)	CD (%)
Overall
Overall	1142	38.4	37.4
By audio type
Speech	771	41.1	40.1
Sound event	265	35.1	33.6
Music	106	27.4	27.4

Appendix

Table 11: OmniBench multiple-choice accuracy (%) for Gemma 4 under baseline decoding and contrastive decoding (CD). Invalid parses (

N/A

) are counted as incorrect.

Explore similar work

Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
Apr 18, 2026Xinru Yan, Boxi Cao, Yaojie Lu +4Unified Multimodal Models Neural Network Interpretability
University of Chinese Academy of Sciences, Beijing, China · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China
Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts
Sep 22, 2026Zhuoyun Li, Boxuan Wang, Xiaowei Huang +1Multimodal Large Language Models VLM Evaluation
School of Computer Science and Informatics, University of Liverpool, United Kingdom
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team

Apr 18, 2026J/K move · Enter open · S save

Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models

Xinru Yan, Boxi Cao, Yaojie Lu, Hongyu Lin+3

University of Chinese Academy of Sciences, Beijing, China · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China

Unified Multimodal Models Neural Network Interpretability Multimodal Foundation Models Multimodal Model Interpretability Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference

Abstract

Explore similar work

Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models

Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation