DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

Saifullah Saifullah; David Dembinsky; Stefan Agne; Andreas Dengel; Sheraz Ahmed

cs.CLOct 27, 2026

DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

Authors: Saifullah Saifullah, David Dembinsky, Stefan Agne, Andreas Dengel, Sheraz Ahmed

Organizations: Smart Data and Knowledge Services (SDS), German Research Center for Artificial Intelligence (DFKI), Trippstadter Str. 122, Kaiserslautern, 67663, Germany. · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Erwin-Schrödinger-Str. 52, Kaiserslautern, 67663, Germany. · DeepReader GmbH, Kaiserslautern, 67663, Germany.

Abstract

Multimodal transformer models have driven unprecedented performance breakthroughs in modern document understanding. However, the inherent black-box nature of these models remains a major obstacle to their safe deployment in high-stakes sectors. Despite significant progress in explainability for vision and language tasks, explainable AI (XAI) research tailored to multimodal document understanding has received little attention. To address this gap, we present \ourframework{}, the first comprehensive explainability framework that adapts existing XAI methods and evaluation metrics to the multimodal (text, layout, and image) document understanding context, while introducing efficient multi-target adaptations of both methods and metrics for domain-specific tasks such as semantic entity recognition. Using \ourframework{}, we systematically evaluate 13 XAI methods across 3 document understanding tasks, 8 datasets, 4 models, and 17 evaluation metrics, and find that only a few methods perform consistently across all evaluation criteria, with Occlusion emerging as the most faithful and interpretable method overall. Through a dedicated meta-evaluation, we observe moderate to strong agreement among several faithfulness and complexity metrics while identifying certain ones that produce unreliable results. Finally, through multiple dataset-wide diagnostic analyses, we uncover key insights into model behavior, including the contribution of individual modalities across tasks and labels, spatial attribution patterns around target entities, and potential dataset biases. Overall, this work serves as both a benchmark and a practical guide for future XAI research in multimodal document understanding. To support reproducibility, we publicly release our code at https://github.com/saifullah3396/docxeval/.

Figures & tables

Figure 1: Overview of the DocXEval framework. Given a dataset and a trained model, the framework (1) extracts modality-specific inputs from each document, (2) defines feature segmentation masks, reference baselines per modality, and targets, (3) computes modality-wise attribution scores via XAI methods, (4) evaluates explanation quality across four XAI evaluation criteria, and (5) conducts a systematic qualitative and quantitative analysis of results across datasets, models, and tasks.

Panel of Figure 2 — Figure 2: Modality-wise model sensitivity to different baselines, shown for LayoutLMv3 [ 2 ] on RVL-CDIP [ 50 ] and FUNSD [ 51 ] . The model is generally less sensitive to the Mean baseline for the image modality on both datasets, whereas for spatial position, it is less sensitive to the Zero Embedding baseline compared to the [PAD] baseline. For the text modality, the model is less sensitive to the [MASK] token compared to other baselines.

Type	Metric	Perturbation Type	Requires Model	Requires Baseline	FS support	MT Support	Chunking
	Completeness ( compl ) ( 35 ) ( $\downarrow$ )	N/A	✓	✓	N/A	✓	$\times$
Axiomatic	Non-Sensitivity ( non-sens ) ( 53 ) ( $\downarrow$ )	Unordered	✓	✓	✓	✓	✓
	Area Over Perturbation Curve ( aopc ) ( 54 ) ( $\uparrow$ )	Ordered	✓	✓	✓	$\times$	✓
	Area Between Perturbation Curves ( abpc ) ( 54 ) ( $\uparrow$ )	Ordered	✓	✓	✓	$\times$	✓
	Sufficiency ( suf ) ( 55 ) ( $\downarrow$ )	Ordered	✓	✓	✓	$\times$	✓
	Sensitivity-N ( sens-n ) ( 56 ) ( $\downarrow$ )	Unordered	✓	✓	✓	✓	✓

Table 1: Overview of the 17 evaluation metrics adapted in this work. Perturbation Type indicates whether the metric requires ordered (features removed in attribution-ranked order), unordered (random subset removal), or no perturbation of the input. FS refers to whether a metric supports computation over modality-wise segmented features. MT indicates whether the metric supports efficient multi-target computation. Chunking indicates whether a metric supports perturbation over feature chunks to improve computational efficiency.

Task	Dataset	Classes	Train	Validation	Test	XAI Eval. Set	Metric
	Tobacco3482 ( 60 )	10	2504	278	700	700	Accuracy
	RVL-CDIP [ 50 ]	16	320000	40000	40000	1000	Accuracy
CLS	DocLayNet ( 61 )	11	69103	6480	4999	1000	Accuracy
	FUNSD [ 51 ]	4	149	—	50	50	F1
	CORD ( 62 )	30	800	100	100	100	F1
	SROIE ( 63 )	5	626	—	347	100	F1

Table 2: Overview of the eight datasets used in this work. For each dataset, we provide the number of classes or entity types, dataset splits, the size of the subset used for XAI evaluation, and the task-specific performance metric.

Task	Dataset	Metric	BERT [ 66 ]	RoBERTa [ 67 ]	LiLT [ 10 ]	LayoutLMv3 [ 2 ]
	Tobacco3482 ( 60 )	Acc. $(\uparrow)$	86.38	87.81	88.53	94.27
	RVL-CDIP [ 50 ]	Acc. $(\uparrow)$	89.70	90.05	91.20	94.12
Classification	DocLayNet ( 61 )	Acc. $(\uparrow)$	97.50	97.90	97.78	98.97
	FUNSD [ 51 ]	F1 $(\uparrow)$	64.69	67.47	76.19	88.09
	CORD ( 62 )	F1 $(\uparrow)$	95.10	96.28	96.85	97.01
	SROIE ( 63 )	F1 $(\uparrow)$	89.63	93.33	94.12	95.28

Table 3: Model performance on individual datasets. Best results are highlighted in bold . Training hyperparameters are reported in Appendix 4 .

Figure 3: Critical difference diagram showing per-metric XAI method rankings across all tasks. Methods connected by a horizontal bar are statistically indistinguishable. Lower ranks indicate better performance. Task-level critical difference diagrams are additionally provided in Appendix 10.1 .

Figure 4: Aggregated mean ranks of XAI methods per evaluation criterion across all tasks, datasets, and models. Lower ranks indicate better performance. Per-task breakdowns are provided in Appendix 10.1 .

Figure 5: Pairwise Spearman correlation between faithfulness metric rankings across all datasets, models, and explainer settings. Per-task correlation matrices are provided in Appendix 10.2 .

Figure 6: Pairwise Spearman correlation between complexity metric rankings across all datasets, models, and explainer settings. Per-task correlation matrices are provided in Appendix 10.2 .

Figure 7: Execution time comparison between iterative (batched) and multi-target attribution computation on CORD (

K=26

) and FUNSD (

K=179

) datasets across two GPUs. Numbers above bars indicate the speedup factor of multi-target over iterative computation. Note the log scale on the y-axis.

Panel of Figure 8 — Figure 8: Mean per-sample-target computation time of each evaluation metric on CLS (single-target) and SER (multi-target) tasks. All results are computed on H100/H200 GPUs. Unordered perturbation metrics benefit significantly from multi-target adaptations, while ordered metrics remain expensive as they cannot be adapted for multi-target computation.

Panel of Figure 9 — Figure 9: Modality-wise explanations of the LayoutLMv3 [ 2 ] model predictions for two randomly selected samples from RVL-CDIP [ 50 ] and DocLayNet ( 61 ) ( CLS task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 10 — Figure 10: Modality-wise explanations of the LayoutLMv3 [ 2 ] model predictions for two randomly selected samples from CORD ( 62 ) and FUNSD [ 51 ] ( SER task). Results are shown for three methods: IG, OCC, and RA, and for each sample, the target entity to be labeled is highlighted in a green bounding box.

Panel of Figure 11 — Figure 11: Modality-wise explanations of the LayoutLMv3 [ 2 ] model predictions for two randomly selected samples from DocVQA ( 65 ) ( QA task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 12 — Figure 12: Modality contributions computed over explanations generated by IG, OCC, and RA for the LayoutLMv3 [ 2 ] model. Results are reported as a dataset-wide mean (left) and as a per-label mean for four randomly selected labels (right). For DocVQA ( 65 ) , contributions are reported separately for the start and end tokens of the predicted answer.

Panel of Figure 13 — Figure 13: Spatial attribution locality scores computed over explanations generated by IG, OCC, and RA for the LayoutLMv3 [ 2 ] model. For each dataset, vertical locality, horizontal locality, and total spread are reported as a dataset-wide mean (left) and as a per-entity-type mean for four randomly selected entity types (right).

Panel of Figure 14 — Figure 14: Top- $k$ word frequency analysis for the LayoutLMv3 [ 2 ] model across five datasets. For each dataset, the most frequently attributed words (top 1% by attribution magnitude) are shown for up to six randomly selected labels. (a–b) CLS task, (c–d) SER task, (e) QA task.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Model	Modality	Learning Rate	Batch Size	Epochs	Optimizer	Weight Decay	Warmup Ratio	Dropout	Early Stopping	Image Size
	BERT [ 66 ]	$\mathcal{T}$ ; $\mathcal{S}$	1.00e-5	32/64	40	Adam	0	1000	0.1	✓	—
	RoBERTa [ 67 ]	$\mathcal{T}$ ; $\mathcal{S}$	1.00e-5	32/64	40	Adam	0	1000	0.1	✓	—
	LiLT [ 10 ]	$\mathcal{T}$ ; $\mathcal{S}$ ; $\mathcal{L}$	1.00e-5	32/64	40	Adam	0	1000	0.1	✓	—
CLS	LayoutLMv3 [ 2 ]	$\mathcal{T}$ ; $\mathcal{S}$ ; $\mathcal{L}$ ; $\mathcal{V}$	1.00e-5	32/64	40	Adam	0	1000	0.1	✓	224 $\times$ 224
	BERT [ 66 ]	$\mathcal{T}$ ; $\mathcal{S}$	2.00e-5	16	100	Adam-W	0.01	100	0.1	✓	—
	RoBERTa [ 67 ]	$\mathcal{T}$ ; $\mathcal{S}$	2.00e-5	16	100	Adam-W	0.01	100	0.1	✓	—

Appendix

Table 4: Training hyperparameters for all model-task combinations used in this work. Modality columns indicate which input modalities each model processes:

\mathcal{T}

(text),

\mathcal{S}

(position),

\mathcal{L}

(layout),

\mathcal{V}

(image).

Method	Hyperparameter	Value
Integrated Gradients (IG) [ 35 ]	n_steps	200
GradientSHAP (GS) [ 24 ]	n_samples	200
DeepSHAP (DS) [ 24 ]	num_baselines	100
Occlusion (OCC) [ 27 ]	sliding_window (token)	8 $\times$ d
Occlusion (OCC) [ 27 ]	stride (token)	4 $\times$ d
Occlusion (OCC) [ 27 ]	sliding_window (image)	16 $\times$ 16 $\times$ 3

Appendix

Table 5: Hyperparameters for the evaluation of XAI methods used in this work. Methods that are not listed have no configurable parameters.

Type	Metric	Hyperparameter	Value
Axiomatic		n_perturbations_per_feature	1
		chunk_size	1%
		$\epsilon_{a}$	1.00e-3
		$\epsilon_{v}$	1.00e-1
	Non-Sensitivity ( ns ) ( $\downarrow$ )	max_features_per_batch	100
Faithfulness		total_feature_bins	200

Appendix

Table 6: Hyperparameters for all XAI metrics used in the evaluation. Metrics that are not listed (e.g., compl , ent-comp , spar ) have no configurable parameters. non-sens and mon-corr share the same hyperparameters as they are computed jointly. Similarly, abpc and suf share hyperparameters with aopc .

Panel of Figure 15 — Figure 15: Per-metric critical difference diagrams for each task. Methods connected by a horizontal bar are not statistically significantly different (Wilcoxon signed-rank test with Holm correction, $\alpha=0.05$ ). Lower ranks indicate better performance.

Panel of Figure 16 — Figure 16: Aggregated mean ranks of XAI methods per evaluation criterion for each task. Lower ranks indicate better performance.

Panel of Figure 17 — Figure 17: Per-task pairwise Spearman correlation between faithfulness metric rankings across respective datasets, models, and explainer settings. On the QA task, Monotonicity ( mon ) ( 28 ) produced undefined values and is therefore excluded.

Panel of Figure 18 — Figure 18: Per-task pairwise Spearman correlation between complexity metric rankings across respective datasets, models, and explainer settings.

Figure 19: aopc curves for all XAI methods across all dataset-model combinations on the CLS task. MoRF (solid) and LeRF (dotted) curves are shown relative to the random baseline.

Figure 20: aopc curves for all XAI methods across all dataset-model combinations on the SER task. MoRF (solid) and LeRF (dotted) curves are shown relative to the random baseline.

Figure 21: aopc curves for all XAI methods across all dataset-model combinations on the QA task. MoRF (solid) and LeRF (dotted) curves are shown relative to the random baseline.

Panel of Figure 22 — Figure 22: Modality-wise explanations of the LayoutLMv3 [ 2 ] model predictions for two randomly selected samples from RVL-CDIP [ 50 ] and DocLayNet ( 61 ) ( CLS task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 23 — Figure 23: Modality-wise explanations of the LayoutLMv3 [ 2 ] model predictions for two randomly selected samples from CORD ( 62 ) and FUNSD [ 51 ] ( SER task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 24 — Figure 24: Modality-wise explanations of the LayoutLMv3 [ 2 ] model predictions for two randomly selected samples from DocVQA ( 65 ) ( QA task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 25 — Figure 25: Modality-wise explanations of the LiLT [ 10 ] model predictions for two randomly selected samples from RVL-CDIP [ 50 ] and DocLayNet ( 61 ) ( CLS task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 26 — Figure 26: Modality-wise explanations of the LiLT [ 10 ] model predictions for two randomly selected samples from CORD ( 62 ) and FUNSD [ 51 ] ( SER task). Results are shown for three methods: IG, OCC, and RA.

Panel of Figure 27 — Figure 27: Modality-wise explanations of the LiLT [ 10 ] model predictions for two randomly selected samples from DocVQA ( 65 ) ( QA task). Results are shown for three methods: IG, OCC, and RA.

Dataset	Model	Explainer	Token	Position	Layout	Image
Tobacco3482	BERT	IG	0.91	0.09	N/A	N/A
		Occ.	0.88	0.12	N/A	N/A
		RA	1.00	N/A	N/A	N/A
	RoBERTa	IG	0.80	0.20	N/A	N/A
		Occ.	0.65	0.35	N/A	N/A
		RA	1.00	N/A	N/A	N/A

Appendix

Table 7: Modality contribution fractions computed for each model across all CLS datasets.

Dataset	Model	Explainer	Token	Position	Layout	Image
FUNSD	BERT	IG	0.88	0.12	N/A	N/A
		Occ.	0.76	0.24	N/A	N/A
		RA	1.00	N/A	N/A	N/A
	RoBERTa	IG	0.46	0.54	N/A	N/A
		Occ.	0.53	0.47	N/A	N/A
		RA	1.00	N/A	N/A	N/A

Appendix

Table 8: Modality contribution fractions computed for each model across all SER datasets.

Dataset	Model	Explainer	Token	Position	Layout	Image
DocVQA	BERT	IG	0.83	0.17	N/A	N/A
		Occ.	0.66	0.34	N/A	N/A
		RA	1.00	N/A	N/A	N/A
	RoBERTa	IG	0.65	0.35	N/A	N/A
		Occ.	0.59	0.41	N/A	N/A
		RA	1.00	N/A	N/A	N/A

Appendix

Table 9: Modality contribution fractions computed for each model on the DocVQA ( 65 ) dataset.

Multimodal Model Interpretability

Multimodal model interpretability studies how AI systems combine and use images, text, audio, video, and other signals, and how to explain their decisions.

See all 130 papers

GridVQA-X: A Framework for Evaluating Multimodal Explainability Methods
Jun 2, 2026Sujay Belsare, Sudarshan Nikhil, Sushant Kumar +2Multimodal Reasoning Multimodal Model Interpretability
1IIIT Hyderabad, India · University of Virginia, USA
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
May 9, 2026Xiang Feng, Jiawei Zhou, Zhangfeng Huang +6Multimodal Understanding LLM Answer Verification
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China · Alibaba Group, Hangzhou, China · Independent Researcher +1
Feature-level Interaction Explanations in Multimodal Transformers
Mar 4, 2026Yeji Kim, Housam Khalifa Bashier Babiker, Mi-Young Kim +1Multimodal Understanding Feature Attribution
University of Alberta, Edmonton, AB, Canada · University of Waikato, Hamilton, New Zealand · University of Alberta, Camrose, AB, Canada

Jun 2, 2026J/K move · Enter open · S save

GridVQA-X: A Framework for Evaluating Multimodal Explainability Methods

Sujay Belsare, Sudarshan Nikhil, Sushant Kumar, Ponnurangam Kumaraguru+1

1IIIT Hyderabad, India · University of Virginia, USA

Multimodal Reasoning Multimodal Model Interpretability Explainability Evaluation Multimodal Model Evaluation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

With the increasing development of Vision-Language Models, it becomes imperative that their predictions are readily explainable to relevant stakeholders. However, the field of explainability has not kept pace with the multimodal surge. While recent Multimodal Explainable AI (MxAI) methods generate explanations to attribute the interaction between different modalities, current evaluation protocols lack the ground truth required to distinguish between true cross-modal reasoning (e.g., spatial composition) and shallow cross-modal shortcuts (e.g., Bag-of-Words attribute matching). It remains unknown whether MxAI methods faithfully capture synergistic interactions or merely hallucinate reasoning on models acting as simple feature detectors. In this paper, we introduce GridVQA-X, the first diagnostic framework specifically designed to evaluate cross-modal explainability. Unlike natural datasets, GridVQA-X leverages a closed-world synthesis logic to generate unique, mathematically guaranteed explanations. We utilize this controlled environment to train paired ground-truth models on identical architectures:

M_{\text{pure}}

, which learns robust spatial-relational reasoning and

M_{\text{spur}}

, which is structurally forced to rely on cross-modal shortcuts. This behavioral divergence creates a rigorous testbed: a faithful explainer must report distinct reasoning pathways for each model. Our findings reveal that widely used methods fail to distinguish between models relying on genuine spatial-relational reasoning and those exploiting cross-modal shortcuts, highlighting a critical gap in capturing true cross-modal synergy and misrepresenting how multimodal models actually make decisions.

Abstract

Multimodal Model Interpretability

Explore similar work

GridVQA-X: A Framework for Evaluating Multimodal Explainability Methods

DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding

Feature-level Interaction Explanations in Multimodal Transformers