For multimodal large language models, when images or speech conflict with accompanying text, measured text reliance can entangle modality preference with evidence position. Earlier studies of text bias often used a fixed evidence order or moved task instructions with the evidence, leaving the contribution of order unclear. In this paper, we use a paired comparison that keeps the instructions and evidence content fixed and swaps only the positions of the two sources to quantify this potential influence. Across vision and speech models, placing an image or recording after conflicting text consistently shifts answers toward its content. We also revisit previous studies and analyze why their experimental settings can lead to misleading conclusions. These findings reveal cross-modal evidence noncommutativity: the same evidence can lead to different judgments when its order changes, and placing perceptual evidence later can increase the model's reliance on its content.
Figures & tables
Figure 1: Everyday requests in which user descriptions conflict with image or speech attachments.
Figure 2: Controlled evidence swaps in vision–text and speech–text. Keeping instructions and content fixed lets us measure how order changes which source the model follows.
Model
P only
P→T−
T−→P
ΔA−
C [95% CI]
Vision: GQA / Visual Genome
Qwen-VL
91.5
28.3
45.0
+16.7
0.65 [0.50, 0.80]
Idefics3
89.6
41.0
67.7
+26.7
3.83 [3.37, 4.29]
Pixtral
91.5
23.5
35.6
+12.1
1.38 [1.03, 1.74]
Vision: VQAv2 / COCO
Qwen-VL
99.2
70.0
85.0
+15.0
1.07 [0.94, 1.21]
Table 1: Main image–text and speech–text results. P denotes perceptual evidence; T− denotes conflicting text. The number is the question selection accuracy (pp). ΔA− is the change of rate. C is the mean log-odds swap score with its 95% confidence interval.
Model
P
T+
PT+
T+P
ΔA+
PT0
T0P
ΔA0
D
Vision: GQA / Visual Genome
Qwen-VL
91.5
98.3
97.5
97.1
-0.4
88.5
90.0
+1.5
+15.2
Idefics3
89.6
98.5
98.8
92.5
-6.3
88.3
86.3
-2.1
+28.8
Pixtral
91.5
98.3
99.0
99.2
+0.2
87.9
90.6
+2.7
+9.4
Vision: VQAv2 / COCO
Qwen-VL
99.2
99.8
100.0
99.8
-0.2
98.8
99.0
+0.2
+14.8
Table 2: Order effects under evidence control. Values show changes in attachment-answer selection when P appears last (pp). D = ΔA− - ΔA0 shows the net influence under evidence relation control. The arrows are omitted. P and T+ mean only use that evidence.
Figure 3: Exploratory Qwen2.5-VL-3B diagnostics on 20 examples: (a) attention moved by warning; (b) instruction placement ablation.
Original warning
Aligned protocol
Task
P→T−
T−→P
ΔA−
P→T−
T−→P
ΔA−
WoV / VQAv2
42.6
28.9
−13.7
14.9
32.6
+17.7
MCR / AQA
1.4
18.4
+17.0
2.4
20.6
+18.2
MCR / SER
0.0
15.1
+15.1
0.1
11.6
+11.5
MCR / VSC
12.7
64.0
+51.3
20.9
65.0
+44.1
Table 3: External benchmarks under original and aligned protocols in pp. P denotes images for Words or Vision (WoV) [ 7 ] and audio for MCR-BENCH (MCR) [ 28 ] . Yellow highlights gains from placing audio last with the original warning retained.
National University of Defense Technology, changsha, China · 2Chinese University of Hong Kong, Shenzhen, China · 3Intelligent Game and Decision Lab, Beijing, China