Beyond Modality Preference: Controlling Multimodal Arbitration under Modality Conflict in MLLMs

Zhaoying Pan; Shenyu Lu; Jun Hui Wong; Junyi Chai; Xiaoqian Wang

cs.CVNov 9, 2026

Beyond Modality Preference: Controlling Multimodal Arbitration under Modality Conflict in MLLMs

Authors: Zhaoying Pan, Shenyu Lu, Jun Hui Wong, Junyi Chai, Xiaoqian Wang

Organizations: Elmore Family School of Electrical and Computer Engineering, Purdue University

Abstract

Multimodal large language models (MLLMs) integrate information across modalities, but become unreliable when those modalities disagree. Under conflicting inputs, MLLMs may favor modalities that are not the most reliable for the task, thus motivating inference-time control of modality arbitration. We identify a small set of attention heads, termed arbitration heads, that causally influence whether predictions follow visual or textual evidence. Building on this, we introduce a lightweight, training-free intervention that biases these heads toward a designated trusted modality at inference time. The identified heads are stable across calibration set sizes and generalize across settings, and steering them enables effective bidirectional control of modality reliance across diverse benchmarks. Our findings identify an effective control mechanism to align MLLM predictions with task-specified modality priorities. The code is available at https://github.com/zhaoyingpan/modality-arbitration.

Figures & tables

Input	Metric	Qwen3	Qwen2.5	LLaVA-OV	InternVL3
$V$	$Acc_{v}$	85.40	84.35	86.80	82.50
$T_{c}$	$Acc_{t}$	87.85	87.30	87.05	85.80
$V+T_{a}$	$Acc_{v}$	94.50	94.45	95.85	93.95
$V+T_{c}$	$Acc_{v}$	47.25	32.70	33.45	30.10
$V+T_{c}$	$Acc_{t}$	42.20	55.25	54.55	56.75

Table 1: Model behavior under unimodal, aligned, and conflicting multimodal inputs. The image is denoted as

I

, the text that aligns or conflicts with the image content is denoted as

T_{a}

and

T_{c}

, respectively.

Acc_{v}

and

Acc_{t}

measure the performance against image-/text-consistent answers.

Input	Metric	Original	Blur-1	Blur-2	Blur-3
$V$	$Acc_{v}$	85.40	84.10	77.35	65.10
$V+T_{c}$	$Acc_{v}$	47.25	40.00	14.10	3.20
$V+T_{c}$	$Acc_{t}$	42.20	49.25	73.95	84.65

Table 2: Effect of degrading the image modality on Qwen3. Image-only accuracy decreases gradually with increasing blur. Under conflicting multimodal input, however, weakening the image causes the model to shift sharply from image-consistent to text-consistent predictions.

Figure 1: Overview of our method. (a) Arbitration head localization: we causally identify attention heads that causally influence modality preference by patching head activations from image-only and text-only inputs into conflicting multimodal inputs and measuring their effects on the answer logit margin. (b) Inference-time intervention: we select the most influential arbitration heads and steer predictions toward a designated modality by adjusting their attention to visual and textual tokens.

Figure 2: Stability of head-wise effects across different sample sizes. Normalized head-wise effects $C^{(\ell,h)}$ for Qwen3-VL-8B, computed using $n$ samples. Despite varying $n$ , high-effect heads consistently concentrate in similar layers and head positions, indicating that the identified pattern is stable even with a small number of samples.

Figure 3: Modality conflict detection for adaptive steering. Top-50 union-logit cosine similarities for samples paired with conflicting or aligned context, where lower similarity indicates stronger modality conflict. The dashed line marks the threshold, selected to maximize conflict-detection accuracy on the calibration set.

Model	Method	MMCD				MC 2				CLASH				CorrelationQA
		Image-P		Text-P		Image-P		Text-P		Image-P		Text-P		Image-P		Text-P
		$Acc_{v}\uparrow$	$Acc_{t}\downarrow$	$Acc_{v}\downarrow$	$Acc_{t}\uparrow$	$Acc_{v}\uparrow$	$Acc_{t}\downarrow$	$Acc_{v}\downarrow$	$Acc_{t}\uparrow$	$Acc_{v}\uparrow$	$Acc_{t}\downarrow$	$Acc_{v}\downarrow$	$Acc_{t}\uparrow$	$Acc_{v}\uparrow$	$Acc_{t}\downarrow$	$Acc_{v}\downarrow$	$Acc_{t}\uparrow$
Qwen3-VL-8B	Base	93.07	6.92	–	–	47.25	42.20	–	–	66.72	28.16	–	–	20.11	51.19	–	–
	CoT	37.88	60.48	–	–	41.65	50.60	–	–	63.30	29.17	–	–	18.12	52.26	–	–
	VCD	95.30	4.70	–	–	53.45	36.30	–	–	70.36	22.81	–	–	21.99	47.04	–	–
	CAP	96.34	3.66	29.50	70.50	61.10	28.45	24.90	64.60	70.83	23.35	51.28	45.23	32.51	35.04	19.37	50.11

Table 3: Comparison across various modality-conflict benchmarks. We report image and text accuracy under both Image-Primary (Image-P) and Text-Primary (Text-P) settings. Under Image-P, higher image accuracy and lower text accuracy are preferred. Under Text-P, lower image accuracy and higher text accuracy are preferred. Best and second-best results are shown in bold and underlined , respectively.

PhD-sec
	Vanilla	CoT	VCD	CAP	MPS	Ours
Image-P $Acc_{v}\uparrow$	69.50	75.60	72.60	69.90	71.70	73.90
Text-P $Acc_{v}\downarrow$	–	–	–	62.00	64.50	58.90
PhD-icc
Image-P $Acc_{v}\uparrow$	64.40	69.10	69.60	59.20	65.70	70.20
Text-P $Acc_{v}\downarrow$	–	–	–	49.50	60.10	58.00

Table 4: Results on the PhD visual hallucination benchmark. We evaluate PhD-sec and PhD-icc, which introduce unrelated or conflicting text. Higher image accuracy is better under Image-Primary. Lower is better under Text-Primary.

Figure 4: Preserving multimodal benefits under sparse modality conflict. The left and right panels evaluate MPS and our method steered towards image and text, respectively. Dotted lines denote unimodal performance. Although unimodal inference avoids modality conflict, it cannot exploit complementary information when the modalities are aligned. Adaptive steering retains this benefit while remaining robust to conflict.

Figure 5: Effect of steering strength $\alpha$ . Increasing $\alpha$ progressively shifts predictions toward the target modality, with performance saturating at moderate steering strengths. $\alpha=0$ denotes the unmodified baseline, and $\alpha=\infty$ corresponds to hard masking of the competing modality.

Figure 6: Effect of the number of arbitration heads (top-k). Steering an increasing number of top-ranked heads progressively shifts predictions toward the target modality, with the effect saturating at moderate values. The top-k of 0 denotes the unmodified baseline.

Intervention	Image-Primary		Text-Primary
	$Acc_{v}\uparrow$	$Acc_{t}\downarrow$	$Acc_{v}\downarrow$	$Acc_{t}\uparrow$
None	47.25	42.20	–	–
Random	47.50	42.30	44.70	44.55
Layer-matched	54.75	34.90	38.50	50.55
Reversed	47.20	42.45	47.20	42.40
All Heads	77.10	6.10	1.05	86.70

Table 5: Ablation study. “Random” randomly selects 30 heads; “Layer-matched” randomly selects each head from the same layers as ours; “Reversed” selects the 30 heads in reverse ranking; “All Heads” intervenes on all attention heads. “Ours w/o suppressing” only increases attention to the trusted modality without suppressing the other modality.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	End-to-end latency	TTFT	Decode latency	Token throughput	Peak GPU memory	Output tokens
Qwen3-VL-8B	Base	0.232	0.134	0.037	14.705	16.500	3.162
	CoT	4.857	0.138	0.036	26.898	16.502	130.129
	VCD	0.637	0.270	0.172	4.873	16.601	2.933
	CAP	0.225	0.136	0.038	13.837	16.502	2.884
	MPS	0.245	0.136	0.039	14.622	16.500	3.314
	Ours	0.275	0.148	0.050	12.300	16.556	3.146

Appendix

Table 6: Efficiency comparison of methods across MLLMs.

Figure 7: Robustness of modality conflict detection using centered logits. Top-50 union-logit cosine similarities after centering each logit vector, for samples paired with conflicting or aligned context. Lower similarity indicates stronger modality conflict. The dashed line marks the conflict-detection threshold calibrated to maximize accuracy on the calibration set.

Figure 8: Robustness of modality conflict detection to calibration-set selection. Top-50 union-logit cosine similarities for samples paired with conflicting or aligned context, using a calibration set generated with random seed 43. The dashed line marks the detection threshold selected to maximize conflict-detection accuracy on the calibration set.

Panel of Figure 9 — Figure 9: Examples of Blur 1-3 on one MC 2 image.

Figure 10: Stability of head-wise effects across different sample sizes for Qwen2.5-VL-7B.

Figure 11: Stability of head-wise effects across different sample sizes for LLaVA-OneVision-7B.

Figure 12: Stability of head-wise effects across different sample sizes for InternVL3-8B.

Figure 13: Robustness to visual degradation.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,120 papers

Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
Jul 29, 2026Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera +2Multimodal Large Language Models Multimodal Model Interpretability
Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom
Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
Sep 1, 2026Jungyeon Lee, Yejin Yoon, Taeuk KimMultimodal Robustness Knowledge Conflicts in Language Models
Hanyang University, Seoul, Republic of Korea
Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucination
May 19, 2026Jinrui Jiang, Zhangtai Wu, Zhen Wu +1Hallucination in Language Models Multimodal Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Jul 29, 2026J/K move · Enter open · S save

Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera, David Watson+1

Faculty of Electrical Engineering, University of Sarajevo, Bosnia and Herzegovina · King’s College London, United Kingdom

Multimodal Large Language Models Multimodal Model Interpretability Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) increasingly support high-stakes decision making by combining complementary information from images and text. While existing explainability methods identify influential image regions or text tokens, they cannot answer a fundamental question: which modality drives a prediction? Consequently, a model may produce the correct output while relying on the wrong source of evidence, masking shortcut learning and unsafe reasoning. We formulate modality attribution as a complementary explainability objective for multimodal foundation models and propose Counterfactual Modality Attribution (CMA), the first framework for quantifying modality-level contributions in MLLMs. CMA generates image-only, text-only, and joint multimodal counterfactuals using coupled diffusion priors and converts them into principled modality attribution scores through a cooperative game-theoretic formulation based on Shapley values. We evaluate CMA on controlled synthetic benchmarks with known ground-truth modality reliance and on a real-world multimodal clinical dataset. CMA correctly identifies the decision-driving modality in 98% of controlled cases and consistently outperforms baselines, revealing failures of cross-modal reasoning that remain invisible to predictive accuracy alone. Our results establish modality attribution as a complementary dimension of explainability beyond feature attribution, providing a principled framework for auditing multimodal foundation models in safety-critical applications.

Abstract

Multimodal Large Language Models

Explore similar work

Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucination