Multimodal Task Interference: A Benchmark and Analysis of History-Target Mismatch in Multimodal LLMs

Masayuki Kawarada; Tatsuya Ishigaki; Hiroya Takamura

cs.CLMar 19, 2026

Multimodal Task Interference: A Benchmark and Analysis of History-Target Mismatch in Multimodal LLMs

Authors: Masayuki Kawarada, Tatsuya Ishigaki, Hiroya Takamura

Organizations: Artificial Intelligence Research Center, AIST

Abstract

Task interference, the performance degradation caused by task switches within a single conversation, has been studied exclusively in text-only settings despite the growing prevalence of multimodal dialogue systems. We introduce a benchmark for evaluating this phenomenon in multimodal LLMs, covering six tasks across text and vision with systematic variation of history-target along three axes: modality mismatch, reasoning mismatch, and answer format mismatch. Experiments on both open-weights and proprietary models reveal that task interference is highly directional: switching from text-only to image-based targets causes severe performance drops, while the reverse transition yields minimal degradation. Interference is further amplified when mismatches co-occur across multiple dimensions, and is driven most strongly by modality differences, followed by answer format, while reasoning requirement shifts cause minimal degradation.

Figures & tables

Figure 1: An illustrative example of multimodal task interference. After processing a conversational history of image captioning tasks, the model is prompted with a text-only question. The sudden task switch causes the model to erroneously expect a visual input, leading to a failure in answering a simple factual question.

Dataset	Task Type	Modality ( $\mathcal{M}$ )	Reasoning ( $\mathcal{R}$ )	Answer Format ( $\mathcal{A}$ )
Rotten Tomatoes	Sentiment Classification	Text-only	No	Classification
MMLU	Multiple-choice QA	Text-only	Yes	Multiple-choice
TweetQA	Open-ended QA	Text-only	No	Generation
VQAv2	Visual QA	Image + Text	No	Short-answer
OK-VQA	Visual QA	Image + Text	Yes	Short-answer
COCO Captions	Image Captioning	Image + Text	No	Generation

Table 1: Overview of the six datasets used in our benchmark, categorized by the three axes of our task interference framework: Modality (

\mathcal{M}

), Reasoning requirement (

\mathcal{R}

), and Answer Format (

\mathcal{A}

N=1
	Modality (%)			Reasoning (%)			Answer Format (%)
	mismatch	match	$\Delta$	mismatch	match	$\Delta$	mismatch	match	$\Delta$
GPT-4.1-mini	-8.89	-7.65	-1.24	-5.35	-11.88	6.53	-8.44	-8.12	-0.32
Gemma-3n	-10.49	-11.48	0.99	-10.84	-10.94	0.10	-11.11	-9.39	-1.72
Qwen3-VL	-9.50	2.65	-12.15 ∗∗∗	-4.29	-5.03	0.74	-5.35	-0.03	-5.31 ∗∗
Pixtral	-6.15	-1.34	-4.81 ∗∗∗	-4.05	-4.42	0.38	-3.83	-6.75	2.91

Table 2: Axis-wise means (%) for mismatch and match groups, and their difference (

\Delta

= mismatch - match). Significance marks on

\Delta

: Welch’s t-test (

{}^{*}p<0.05,\,^{**}p<0.01,\,^{***}p<0.001

Figure 2: A heatmap visualizing the performance drop (relative change in %) across all pairwise combinations of history and target datasets for GPT-4.1-mini with a history length of $N=3$ .

N=1
	Text $\rightarrow$ Image (%)	Image $\rightarrow$ Text (%)
GPT-4.1-mini	-18.65 ∗∗∗	0.88
Gemma-3n	-18.87 ∗∗∗	-2.11 ∗∗∗
Qwen3-VL	-19.84 ∗∗∗	0.85
Pixtral	-12.94 ∗∗∗	0.64
N=3

Table 3: Directional modality transitions. Values are mean relative change (%) for each transition type. Significance markers are based on one-sample t-test against 0 for each transition (

{}^{*}p<0.05,\,^{**}p<0.01,\,^{***}p<0.001

Figure 3: Performance difference (

\Delta

) between mismatch and match conditions across varying history lengths (

N=1,3,5

) for modality, reasoning, and answer format dimensions.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

MIBench: Evaluating LMMs on Multimodal Interaction
Mar 13, 2026Yu Miao, Zequn Yang, Yake Wei +5Multimodal Large Language Models Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China +1
TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
May 20, 2026Andong Hua, Colton Bishop, Igor Mordatch +5Multimodal Large Language Models Modality Gap in VLMs
University of California, Santa Barbara · Google DeepMind
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1

Mar 13, 2026J/K move · Enter open · S save

MIBench: Evaluating LMMs on Multimodal Interaction

Yu Miao, Zequn Yang, Yake Wei, Ziheng Chen+4

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China+1

Multimodal Large Language Models Multimodal Model Evaluation Multimodal Understanding Multimodal Reasoning

Open paper AskPDF arXiv

In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways between modalities are referred to as "multimodal interaction". How well a model handles various multimodal interactions largely characterizes its multimodal ability. In this paper, we introduce MIBench, a comprehensive benchmark designed to evaluate the multimodal interaction capabilities of Large Multimodal Models (LMMs), which formulates each instance as a (con_v , con_t, task) triplet with contexts from vision and text, necessitating that LMMs employ correct forms of multimodal interaction to effectively complete the task. MIBench assesses models from three key aspects: the ability to source information from vision-centric or text-centric cues, and the ability to generate new information from their joint synergy. Each interaction capability is evaluated hierarchically across three cognitive levels: Recognition, Understanding, and Reasoning. MIBench comprises over 10,000 vision-text context pairs spanning 32 distinct tasks. Evaluation of state-of-the-art LMMs show that: (1) LMMs' ability on multimodal interaction remains constrained, despite the scaling of model parameters and training data; (2) they are easily distracted by textual modalities when processing vision information; (3) they mostly possess a basic capacity for multimodal synergy; and (4) natively trained multimodal models show noticeable deficits in fundamental interaction ability. We expect that these observations can serve as a reference for developing LMMs with more enhanced multimodal ability in the future.

Abstract

Multimodal Large Language Models

Explore similar work

MIBench: Evaluating LMMs on Multimodal Interaction

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs