EDMA: Entropy-Driven Multimodal Answering

Emanuele Mezzi; Gertjan Burghouts; Fabio Massacci; Mengyuan Zhang

cs.AIOct 19, 2026

EDMA: Entropy-Driven Multimodal Answering

Authors: Emanuele Mezzi, Gertjan Burghouts, Fabio Massacci, Mengyuan Zhang

Organizations: Vrije Universiteit Amsterdam · TNO · University of Trento

Abstract

Multimodal question answering (MMQA) is based on integrating heterogeneous data sources, selectively leveraging relevant modalities, and ignoring distractors. Existing approaches based on Multimodal Large Language Models (MLLMs) improve performance by splitting the task into intermediate steps, but do not quantify how each modality contributes to reaching the final answer. We argue that quantifying the effect that each modality has on the answering process improves process traceability and final performance. Thus, we propose Entropy-Driven Multimodal Answering (EDMA), which uses logical entropy to formalise multimodal QA as an iterative process of entropy reduction and to quantify the contribution of each modality in distinguishing correct from incorrect candidate answers. This enables the construction of quantifiable answering trajectories, where each step is associated with a measurable entropy reduction. EDMA introduces (i) modality-conditioned partitioning to estimate unimodal entropy reduction, (ii) entropy-driven multimodal fusion to capture complementary information across modalities, and (iii) entropy-based selection of the answering trajectory that minimises the most logical entropy. Evaluated across standard MMQA benchmarks, EDMA outperforms competitive prompting baselines, with average gains of 12.8 F1 and 4.6 LLM-J points over Planning Prompting and 18.6 F1 and 8.3 LLM-J points over Chain-of-Thought. Moreover, EDMA outperforms self-consistency baselines and is competitive with the multi-agent system baseline. Moreover, we show that higher entropy correlates with more false positives and, through controlled trajectory selection, demonstrate that logical entropy serves as an effective, monotonic selector for correct reasoning paths, validating entropy level as a reliable proxy for performance.

Figures & tables

Figure 1: Comparison between CoT and EDMA. While they both structure QA in intermediate steps, EDMA generates quantifiable answering trajectories, allowing for improved performance and quantified modality contribution.

Figure 2: EDMA is composed of four stages: (a) modality prediction and extraction of answer requirements, (b) generation of unimodal partitions and calculation of unimodal logical entropy, (c) generation of multimodal partitions and calculation of multimodal logical entropy, (d) selection of the answering trajectory which minimises the logical entropy and generation of the answer.

	Model
Method	Claude-4.6			Kimi-K2.5			Qwen3			GPT-5.2			Nemotron			Nova-2-Lite			Avg
	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J
MultiModalQA Talmor et al. (2021)
EDMA	82.8	83.5	88.9	80.4	81.0	83.2	79.4	80.9	82.5	80.0	80.3	80.5	79.0	80.5	75.1	79.0	80.4	65.9	80.1	81.1	79.3
PP	61.4	73.1	85.6	64.0	74.0	82.0	60.6	69.4	75.4	56.6	68.4	75.5	51.8	60.0	64.6	50.0	57.8	41.8	57.4	67.1	70.8
CoT	61.4	72.3	87.4	63.8	72.7	79.9	61.0	70.7	76.4	46.2	56.8	64.4	46.4	54.2	57.1	43.4	50.2	53.4	53.7	62.8	69.8

Table 1: EDMA achieves the highest average EM, F1, and LLM-judge scores on MultiModalQA, ManyModalQA across all evaluated models, outperforming best-practice prompting strategies: Direct Prompting (DP), Chain-of-Thought (CoT), and Planning Prompting (PP).

	Model
Method	Claude-4.6			Kimi-K2.5			Nemotron			Nova-2-Lite			Avg
	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J	EM	F1	LLM-J
EDMA	82.8	83.5	88.9	80.4	81.0	83.2	79.0	80.5	75.1	79.0	80.4	65.9	80.3	81.4	78.2
MAMMQA	–	–	83.3	–	–	88.4	–	–	65.7	–	–	65.7	–	–	75.7
CoT-SC	58.5	72.0	84.3	58.5	69.6	76.4	42.5	52.7	61.0	46.5	55.6	63.2	51.5	62.5	71.2
PP-SC	56.5	69.9	82.5	60.5	73.7	80.8	52.5	61.6	60.2	55.5	67.3	59.6	56.3	68.1	70.8

Table 2: Comparison of EDMA with MAMMQA and self-consistency (SC) baselines (

N=6

samples) across EM, F1, and LLM-J metrics.

Figure 3: Qualitative comparison of CoT and EDMA. EDMA answers both questions correctly, isolating information in the unimodal setting before bridging modalities in the multimodal step, while CoT returns incorrect answers.

Panel of Figure 4 — Figure 4: Wrong answers are associated with higher entropy values, as shown by a correlational analysis (a) and a controlled trajectory-selection analysis (b).

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$Q$	Input question
$M$	Set of all modalities
$m\in M$	A specific modality
$M_{s}\subseteq M$	Set of selected starting modalities
$M_{a}$	Modalities under analysis
$m_{t}$	Candidate target modality

Appendix

Table 3: Summary of notation used.

MultiModalQA [ 1 ]
Approach	Model	EM	F1
EDMA	Claude-4.6	82.8	83.5
UniMMQA [ 2 ]	T5-3B [ 43 ]	73.7	80.1
UniRAG [ 34 ]	LLaVA-v1.5-7B [ 44 ]	67.4	71.3
Solar [ 33 ]	T5 [ 43 ] + BLIP [ 45 ]	59.8	66.1
MultiModalQA [ 1 ]	RoBERTa-large [ 46 ] + ViLBERT [ 47 ]	49.3	55.9

Appendix

Table 4: Comparison of EDMA against prior specialized multimodal QA frameworks on MultiModalQA and ManyModalQA datasets.

Model	Approach	Avg. Total Tokens	Avg. # Calls	Avg. Cost ($)	Latency (s)
Nova-2-Lite	CoT	19,775.20	1.00	0.006	$\sim$ 5.04
Nova-2-Lite	DP	19,576.12	1.00	0.006	$\sim$ 3
Nova-2-Lite	EDMA	89,909.88	143.35	0.027	$\sim$ 30
Nova-2-Lite	PP	20,070.06	1.00	0.007	$\sim$ 5
Nova-2-Lite	MAMMQA	79,972.14	7.00	0.024	$\sim$ 10
GPT-5.2	CoT	4,616.74	1.00	0.009	$\sim$ 3

Appendix

Table 5: Average computational cost and latency across models and approaches.

Approach	Avg. Absolute Cost Difference vs. EDMA ($)	Avg. Cost Ratio ( $\frac{\text{EDMA}}{\text{Approach}}$ )	Avg. Latency Ratio ( $\frac{\text{EDMA}}{\text{Approach}}$ )
CoT	0.069	$5.91\times$	$22.83\times$
DP	0.070	$6.40\times$	$45.79\times$
PP	0.068	$5.06\times$	$19.92\times$
MAMMQA	0.011	$1.14\times$	$11.05\times$

Appendix

Table 6: Average cost and latency overhead of EDMA compared with other approaches.

Model	Modality Selection	Criteria Extraction	Intermediate Answers	Multimodal Fusion
Nova-2-Lite	90%	94%	92%	4%
GPT-5.2	96%	98%	96%	6%
Kimi-K2.5	96%	96%	96%	8%
Nemotron	90%	94%	30%	2%
Qwen3	96%	96%	96%	8%
Claude-4.6	96%	98%	96%	8%

Appendix

Table 7: Intermediate-step accuracy for questions incorrectly answered by EDMA.

Model	Modality Selection	Criteria Extraction	Intermediate Answers	Multimodal Fusion
Nova-2-Lite	95%	96%	93%	70%
GPT-5.2	98%	98%	98%	82%
Kimi-K2.5	98%	97%	98%	86%
Nemotron	93%	94%	89%	76%
Qwen3	97%	96%	97%	85%
Claude-4.6	98%	98%	97%	91%

Appendix

Table 8: Intermediate-step accuracy across all questions, both correctly and incorrectly answered.

Model	Metric	EDMA	PP	DP	CoT	MAMMQA
Claude-4.6	EM	82.7 $\pm$ 1.9	57.9 $\pm$ 1.9	56.1 $\pm$ 2.5	57.8 $\pm$ 2.0	–
	F1	83.0 $\pm$ 1.1	70.7 $\pm$ 1.3	68.1 $\pm$ 1.3	70.8 $\pm$ 0.8	–
	LLM-J	88.8 $\pm$ 0.5	85.4 $\pm$ 0.4	82.6 $\pm$ 0.5	87.5 $\pm$ 0.5	82.6 $\pm$ 1.3
Kimi-K2.5	EM	80.7 $\pm$ 1.8	60.2 $\pm$ 2.1	61.9 $\pm$ 0.6	58.1 $\pm$ 3.1	–
	F1	81.3 $\pm$ 0.3	72.4 $\pm$ 1.0	73.7 $\pm$ 0.3	68.7 $\pm$ 2.3	–
	LLM-J	83.3 $\pm$ 0.5	81.7 $\pm$ 0.4	52.0 $\pm$ 0.4	79.3 $\pm$ 0.4	88.3 $\pm$ 0.9

Appendix

Table 9: Mean

\pm

standard deviation of EM, F1, and LLM-judge score (LLM-J) over 10 runs at temperature 0. MAMMQA is only evaluated with LLM-J.

Model	EDMA	EDMA (Random Trajectory)	Difference
GPT-5.2	80.5	45.8	-34.7
Nova-2-Lite	65.9	47.3	-18.6
Kimi-K2.5	83.2	37.1	-46.1
Nemotron	75.1	51.6	-23.5
Qwen3	82.5	48.2	-34.3
Claude-4.6	88.9	53.0	-35.9

Appendix

Table 10: Performance comparison between full EDMA and random trajectory selection.

Model	EDMA	EDMA (Unimodal-only)	Difference
GPT-5.2	80.5	65.5	-15.0
Nova-2-Lite	65.9	48.0	-17.9
Kimi-K2.5	83.2	73.1	-10.1
Nemotron	75.1	56.8	-18.3
Qwen3	82.5	68.4	-14.1
Claude-4.6	88.9	77.8	-11.1

Appendix

Table 11: Performance comparison between full EDMA and unimodal-only partitioning.

Backbone	Metric	EDMA (mean)	PP (mean)	$\Delta$	Sig. ( $t$ -test)	Sig. (Wilcoxon)
Claude-4.6	EM	82.7	57.9	$+24.9$	✓	✓
	F1	83.0	70.7	$+12.4$	✓	✓
	LLM-judge	88.8	85.4	$+3.3$	✓	✓
Kimi-K2.5	EM	80.7	60.2	$+20.5$	✓	✓
	F1	81.3	72.4	$+8.8$	✓	✓
	LLM-judge	83.3	81.7	$+1.6$	✓	✓

Appendix

Table 12: Statistical comparison of EDMA against PP across model backbones.

\Delta

represents the mean score difference (

\text{EDMA}-\text{PP}

). Tick (✓) indicates significance at

p<0.05

Figure 5: Multimodal entropy distribution consistently shows lower values compared to unimodal entropy distributions.

			T-Test		Point-Biserial
Combination direction	$\boldsymbol{\overline{H(Uni)}}$	$\boldsymbol{\overline{H(Multi)}}$	Stat ( $t$ )	Sig. ( $p$ )	Corr. ( $r$ )	Sig. ( $p$ )
Image $\rightarrow$ Table	0.29	0.27	8.84	✓	-0.16	✓
Image $\rightarrow$ Text	0.27	0.21	5.80	✓	-0.60	✓
Table $\rightarrow$ Image	0.25	0.21	6.82	✓	-0.36	✓
Table $\rightarrow$ Text	0.25	0.20	12.16	✓	-0.43	✓
Text $\rightarrow$ Image	0.33	0.27	4.11	✓	-0.47	✓

Appendix

Table 13: Statistical Evaluation of Entropy Reduction via Information Fusion.

\overline{H}

represents mean entropy;

t

represents the t-statistic; Point-Biserial (PB) is the correlation between a binary modality indicator (unimodal vs. multimodal) and entropy values. Negative PB values reflect lower entropy under multimodal representations. Tick (✓) indicates significance at

p<0.05

Modality	$\boldsymbol{\overline{H(TP)}}$	$\boldsymbol{\overline{H(FP)}}$	Stat ( $t$ )	Sig. ( $p$ )	Corr. ( $r$ )	Sig. ( $p$ )
			T-Test		Point-Biserial
Text	0.20	0.24	-8.72	✓	0.26	✓
Image	0.21	0.26	-5.78	✓	0.26	✓
Table	0.21	0.27	-8.66	✓	0.23	✓

Appendix

Table 14: Statistical comparison of entropy in True Positives (TP) vs. False Positives (FP).

\overline{H}

denotes mean entropy;

t

denotes the t-test statistic; PB denotes the point-biserial correlation between entropy and a binary outcome variable (TP = 0, FP = 1). Tick (✓) indicates significance at

p<0.05

	Mean FDR			Friedman Test
Modality	$\boldsymbol{\overline{\text{FDR}}_{\text{Low}}}$	$\boldsymbol{\overline{\text{FDR}}_{\text{Medium}}}$	$\boldsymbol{\overline{\text{FDR}}_{\text{High}}}$	Stat ( $Q$ )	Sig. ( $p$ )
Text	0.081	0.475	0.503	130.72	✓
Image	0.189	0.766	0.782	108.10	✓
Table	0.133	0.459	0.491	40.36	✓

Appendix

Table 15: Statistical comparison across modalities using Mean False Discovery Rates (

\overline{\text{FDR}}

) and the Friedman Test for Related Samples.

Q

denotes the Friedman test statistic. Tick (✓) indicates significance at

p<0.05

	Low vs Medium			Low vs High
Modality	Low	Med.	Sig. ( $p$ )	Low	High	Sig. ( $p$ )
Text	0.081	0.475	✓	0.081	0.503	✓
Image	0.189	0.766	✓	0.189	0.782	✓
Table	0.133	0.459	✓	0.133	0.491	✓

Appendix

Table 16: Pairwise comparisons of Mean False Discovery Rates (

\overline{\text{FDR}}

) across entropy levels within each modality using the Wilcoxon Signed-Rank Test for paired samples. Tick (✓) indicates significance at

p<0.05

Modality Pair	Candidate Bucket	Low Entropy FDR	Medium Entropy FDR	High Entropy FDR
Text	13–19	0.161	0.479	0.480
Text	20–23	0.084	0.359	0.482
Text	24–26	0.071	0.426	0.461
Text $\rightarrow$ Image	13–19	0.250	0.933	1.000
Text $\rightarrow$ Image	20–23	0.267	0.917	1.000
Text $\rightarrow$ Table	13–19	0.200	0.350	0.533

Appendix

Table 17: False Discovery Rate across modality paths, candidate set size buckets, and entropy levels. FDR increases consistently as entropy increases from Low to Medium and High.

System Prompt
You are a Question Answering assistant.
Your task is to determine which modality or combination of modalities is required to answer a given question. You are provided with: text paragraphs, images (with titles), and tables.

Select exactly one of the following labels: • image : the answer can be derived from images alone • text : the answer can be derived from text alone • table : the answer can be derived from tables alone • image_text : the answer requires both images and text • image_table : the answer requires both images and tables • text_table : the answer requires both text and tables • text_table_image : the answer requires text, tables, and images Guidelines: • Use only the provided information • Output exactly one label, with no additional text
User Prompt
Question: {question}

Appendix

Table 18: Prompt used to determine which modalities are required to answer a query.

System Prompt
You are a Question Answering assistant.
You task is to convert natural-language questions into structured query frames.

Given a question, extract a structured representation with the following components: • Topic. Select exactly one topic from the predefined set: {topics} . Choose the most dominant topic; if unclear, use Other . • Expected Answer Type. Identify both: (i) a specific answer type, and (ii) a general entity category. Focus on the type of entity the answer represents. • Target Entity. Identify the main entity referenced in the question. • Constraints. Extract atomic, independently verifiable conditions from the question. Each constraint should include: 1. a constraint type (e.g., relation, award, time, role, qualifier) 2. the exact supporting phrase from the question 3. a normalized version of that phrase
User Prompt
Question: {question_text}

Appendix

Table 19: Prompt used to extract structured answer requirements from natural-language questions.

System Prompt
You are a Question Answering assistant.
Your task is to determine whether a text paragraph (or its title) contains any concept specified in a given answer requirement.
User Prompt
Title: {metadata}
Paragraph: {text}
Answer requirement: {r}

Appendix

Table 20: Prompt to determine the presence of content overlap between a paragraph and an answer requirement.

System Prompt
You are a Question Answering assistant.
Your task is to determine whether an image or its title contains any concept specified by an answer requirement.
User Prompt
Title: {metadata}
Image: {image data}
Answer requirement: {r}

Appendix

Table 21: Prompt to determine the presence of content overlap between an image and an answer requirement.

System Prompt
You are a Question Answering assistant.
Your task is to determine whether a table row contains any concept specified a given answer requirement.
User Prompt
Table Row: {table row}
Answer requirement: {r}

Appendix

Table 22: Prompt to determine the presence of content overlap between a table row and an answer requirement.

System Prompt
You are a Question Answering assistant.
Your task is to extract relevant information from a given paragraph in relation to a question. You are given the question, the paragraph title, and the paragraph. Follow the instructions to extract the relevant information: 1. Step 1. Question analysis: determine whether the paragraph is related to the question using only the provided inputs. 2. Step 2. Evidence extraction: if the paragraph is related to the question, extract the minimal information relevant to answering the question. The description must include only the entity or value that directly connects the paragraph to the question.
User Prompt
Question: {question_text}
Paragraph Title: {title}
Paragraph: {text}

Appendix

Table 23: Prompt used for extracting question-relevant evidence from a paragraph.

System Prompt
You are a Question Answering assistant.
Your task is to extract relevant information from a given image in relation to a question. You are given the question, the image title, and the image. Follow the instructions to extract the relevant information: • Step 1. Question analysis: determine whether the image is related to the question using only the provided inputs. • Step 2. Evidence extraction: if the image is related to the question, produce a short textual description of the image that captures the minimal information relevant to answering the question. The description must include only the entity or value that directly connects the image to the question.
User Prompt
Question: {question_text}
Image Title: {title}
Image: {text}

Appendix

Table 24: Prompt used for extracting question-relevant evidence from an image.

System Prompt
You are a Question Answering assistant.
Your task is to extract relevant information from a given table row in relation to a question. You are given the question and the table row. Follow the instructions to extract the relevant information: • Step 1. Question analysis: determine whether the table row is related to the question using only the provided inputs. • Step 2. Evidence extraction: if the table row is related to the question, produce a short textual description of the table row that captures the minimal information relevant to answering the question. The description must include only the entity or value that directly connects the table row to the question.
User Prompt
Question: {question_text}
Paragraph Title: {title}
Paragraph: {text}

Appendix

Table 25: Prompt used for extracting question-relevant evidence from a table row.

System Prompt
You are a Question Answering assistant.
You are given a question and two paragraphs. Your task is to determine whether any entity, concept, or fact mentioned in one paragraph is also mentioned in the other paragraph.
User Prompt
Question: {question_text}
Title Paragraph 1: {title_1}
Paragraph 1: {content_paragraph_1}

Appendix

Table 26: Prompt to determine the presence of mutual information between two paragraphs.

System Prompt
You are a Question Answering assistant.
You are given a question, one image, and an image title and a textual description.
Your task is to determine whether any entity, concept, or fact mentioned in the textual description is also present in the image or image title.
User Prompt
Question: {question_text}
Text: {content}

Appendix

Table 27: Prompt to check the presence of mutual information between an image and a paragraph.

System Prompt
You are a Question Answering assistant.
You are given a question, paragraph (textual description), a table description, and a table row.
Your task is to determine whether any entity, concept, or fact mentioned in the textual description is also present in the table row. Use the table description to understand the context.
User Prompt
Question: {question_text}
Paragraph: {content}

Appendix

Table 28: Prompt to check the presence of mutual information between a table row and a paragraph.

System Prompt
You are a Question Answering assistant.
You are given a question, a paragraph, the expected answer type. Your task is to determine whether the paragraph contains an entity that completely or partially matches the expected answer type required by the question.
Guidelines: • This is not keyword matching. Perform semantic reasoning using synonyms, paraphrases, and implicit mentions • Only return TRUE if the paragraph contains a specific entity that could fully or partially answer the question. Partial answers are acceptable if relevant. • If the paragraph only provides context without an answer entity, return FALSE . If uncertain, return FALSE . • If TRUE , extract the exact text span from the paragraph without paraphrasing or normalization. If FALSE , return NONE
User Prompt
Question: {question_text}
Expected Answer Type: {answer_class}

Appendix

Table 29: Prompt to check whether a paragraph contains an entity that matches the answer type.

System Prompt
You are a Question Answering assistant.
Your task is to determine whether either the caption or the image contain an entity that matches the expected answer type required by the question. You are given the question, the image title, and an image. You must evaluate both the image title and the image.

Guidelines: • First check whether the caption explicitly contains an entity that satisfies the expected answer type. If not, check whether the image contains a visually identifiable entity that satisfies the answer type. If neither modality satisfies either answer type, return FALSE . • If uncertain, return FALSE . • If evidence comes from the caption, extract the exact text span. If evidence comes from the image, provide a concise visual description. If no match, the returned entity must be NONE . • Critical instructions for the Image: Examine all parts of the image carefully, including background, corners, and small or partially obscured objects. Pay attention to subtle details such as tiny text, small objects, fine patterns, colours, positions, or partially visible entities. Do not rely only on large, obvious objects. Small details may be crucial to answer the question. Only return TRUE for contains if the entity is clearly present in the image. If uncertain, return FALSE .
User Prompt
Question: {question_text}

Appendix

Table 30: Prompt to check if the caption or the image contains an entity that matches the answer type.

System Prompt
You are a Question Answering assistant.
Your task is to determine whether a table row contains a value that directly answers a question.
You are given the question, the expected answer type, the table description, and one table row.

Guidelines: • Use the description of the table to interpret the row correctly • Do not use only literal meaning. Also use synonyms and contextual meaning • Only return TRUE if one of the row’s values directly answers the question • If the row contains related information but no direct answer, return FALSE • If uncertain, return FALSE • If TRUE , extract the exact cell value from the row that contains the answer. If FALSE , return NONE
User Prompt

Appendix

Table 31: Prompt to verify whether a table row contains a value that directly answers a question.

System Prompt
You are a multimodal question answering assistant.
Your task is to answer a given question using three types of provided data sources: text paragraphs, images (with titles), and tables. Carefully review all sources and derive the answer by combining relevant information across modalities.

The final response should be concise and contain only the answer, without any explanation. If the necessary information is not available, output None .
User Prompt
Question: {question_text}

Appendix

Table 32: Direct prompting baseline used in our experiments.

System Prompt
You are a multimodal question answering assistant. Your task is to answer a given question using only the provided information sources: text paragraphs, images (with titles), and tables. You must not rely on any external or prior knowledge. If the provided information is insufficient, output None .

Procedure to follow: • Step 1. Extract information: Identify relevant facts from each modality: text (key statements), images (objects, labels, relationships), and tables (rows, columns, values). • Step 2. Align information: Match entities and concepts that appear across different modalities and explicitly link them. • Step 3. Integrate information: Combine the aligned evidence to form a complete understanding, ensuring that all parts of the question are addressed. • Step 4. Derive the answer: Formulate the final answer, clearly grounded in the combined evidence from the modalities. • Step 5. Verify: Ensure that the answer is fully supported by the provided information and that no relevant modality has been ignored. Output the reasoning separated from the final response. The final response should be concise and contain only the answer, without any explanation.
User Prompt
Question: {question_text}
Images: {images_text}

Appendix

Table 33: Chain-of-Thought (CoT) prompt used in our experiments.

System Prompt
You are a multimodal question answering assistant. Your task is to answer a given question using the provided data sources: text paragraphs, images (with titles), and tables. You must rely only on this information and not use any external knowledge. If the provided information is insufficient, output None .

Before producing the final answer, follow the three phases below: Phase 1: Strategic Planning
Analyse the question and available data sources, and devise a step-by-step plan. The plan should: • Identify which data sources are most relevant • Determine the order in which they should be processed • Specify how to handle potential conflicts across sources Phase 2: Execution
Follow the plan to extract and combine information. Every piece of evidence must be explicitly grounded in a source (e.g., table, image title, or paragraph).
Phase 3: Final Answer

Appendix

Table 34: Planning-based prompting baseline used in our experiments.

Multimodal QA

Multimodal question answering combines language with images, video, audio, or documents to answer questions using evidence across modalities.

See all 195 papers

Sep 8, 2026

Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation

The rapid growth of multimodal data has intensified the need for question answering (QA) systems capable of reasoning across heterogeneous sources such as text, tables, and images. In this paper, we present a comprehensive methodological comparison of three influential frameworks, namely Multimodal Adaptive Extraction (MAE), Solar, and UniMMQA, tracing the evolution of multimodal question answering from modality-adaptive pipelines to fully unified architectures. We examine how each approach models cross-modal interactions, transforms heterogeneous inputs, and performs reasoning, highlighting key design differences in modality representation, reasoning, and answer generation. Our analysis demonstrates a clear shift from explicit modality-specific processing toward unified text-centric formulations enabled by pre-trained language models (PLMs). Empirical comparisons across benchmark datasets show that this transition leads to substantial improvements in both Exact Match (EM) and F1-Scores, with UniMMQA achieving the most consistent and scalable performance. Despite these advances, we identify persistent challenges, including information loss during modality transformation, error propagation in multi-stage pipelines, and limitations in capturing fine-grained cross-modal dependencies. Overall, this study provides a deeper understanding of current design trends and offers insights into the future direction of unified multimodal reasoning systems.

Abdullah Al Shafi

Department of CSE, Khulna University of Engineering & Technology, Bangladesh

Unified Multimodal Models Multimodal QA Multimodal Reasoning

Ask PDF

Jul 1, 2026

MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering

As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in multimodal source documents. To our knowledge, this is the first evaluation dataset designed specifically for multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.

Dang Quang Thien Tran, Quang V. Dang, Vinamra Tyagi +7

University of Massachusetts, Amherst · 2Adobe Research, San Jose

Multimodal QA Document QA Multimodal Model Interpretability

Ask PDF

Apr 28, 2026

M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

We present M

^3

-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (MLLMs) in fine-grained multimodal entity understanding and complex multi-hop reasoning. Unlike existing VQA datasets that focus on coarse-grained categories and simple reasoning over single entities, M

^3

-VQA introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources. It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents, supported by traceable, detailed evidence and a curated multimodal knowledge base. We evaluate 16 leading MLLMs under three settings: without external knowledge, with gold evidence, and with retrieval-augmented input. The poor results reveal significant challenges for MLLMs in knowledge acquisition and reasoning. Models perform poorly without external information but improve markedly when provided with precise evidence. Furthermore, reasoning-aware agentic retrieval surpasses heuristic methods, highlighting the importance of structured reasoning for complex multimodal understanding. M

^3

-VQA presents a more challenging evaluation for advancing the multimodal reasoning capabilities of MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/M3VQA.

Jiatong Ma, Longteng Guo, Yuchen Liu +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Multimodal Understanding Visual Question Answering Multimodal QA Multimodal RAG Knowledge-Based VQA Multimodal Model Evaluation+1

Ask PDF

Abstract

Multimodal QA

Explore similar work

Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation

MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering

M3^33-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering