Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approaches

Marcin Michał Mirończuk

cs.CLApr 7, 2026

Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approaches

Authors: Marcin Michał Mirończuk

Organizations: National Information Processing Institute, al. Niepodległości 188b, 00-608, Warsaw, Poland

Abstract

Information fusion is used widely to improve document classification by the integration of multiple data sources (multimodal) or representations (multiview). However, the field lacks a unified framework, a quantitative synthesis of its effectiveness, and clear guidance for practitioners. This systematic review addresses these gaps by analysing 139 primary studies. It introduces a formal framework to structure the field, presents the results of a qualitative analysis to identify key trends, and performs a random-effects meta-analysis (to our knowledge, the first focused on document classification) to quantify performance gains. Our meta-analysis reveals that multimodal fusion improves accuracy (mean gain of +5.28 percentage points, $p=0.0016$ ) significantly -- the F1-score effect is directionally positive but statistically non-significant in our primary model. Multiview fusion provides consistent but modest gains for accuracy (+4.67%), F1-score (+3.08%), and recall (all $p<0.05$ ). Critically, our qualitative synthesis uncovers challenges in reproducibility in methodological rigour: only 11.8% (multimodal) and 23.3% (multiview) of the studies use statistical tests to validate their findings, which undermines the reliability of many of their results. This review's primary contributions are a unifying framework, the first quantitative evidence base, and data-driven guidelines. This review concludes that successful information fusion depends not on algorithmic complexity, but on the strategic alignment of the fusion method with the task context and a commitment to more rigorous validation.

Figures & tables

Figure 1: The evolution of information fusion research through the prism of review articles.

Figure 2: A distribution of the studies on document classification reviews.

Ref. & Scope	Dominant fusion paradigm	Methodological focus vs. gap	Systematic protocol	Our added contribution relative to this work
Part 1: Classical sensor & context fusion
[ 153 , 104 ] Multi-sensor perception	Bayesian, Dempster–Shafer (DS), Kalman filtering	Focus: Handling uncertainty/conflict in sensor signals and multi-sensor information fusion. Gap: No treatment of semantic alignment for unstructured text/image data or learned representations.	No	Moves from signal-level uncertainty modelling to semantic-level feature integration for documents; formalises document-classification architectures as information-fusion systems.
[ 117 ] Context/high-level information fusion	JDL framework	Focus: Situational awareness, context exploitation, and high-level information fusion processes. Gap: Does not address the feature-learning pipeline for high-dimensional unstructured document data.	No	Applies high-level context-fusion ideas to different document “views” (such as structure, layout, and textual content) in document-classification pipelines.
[ 135 ] Rough sets	Granular computing, approximation spaces	Focus: Mathematical reducts, approximation operators and rule induction for multi-source information systems. Gap: Limited applicability to deep continuous representations (embeddings) commonly used in multimodal document models.	No	Extends beyond rough-set-based symbolic fusion to a broader representation-learning view that encompasses both classical feature-based and deep neural representations for multimodal and multiview document fusion.
Part 2: Aggregation operators
[ 96 ] Linguistic decision making	Aggregation operators	Focus: Mathematical properties and design of aggregation operators for probabilistic linguistic information in group decision making. Gap: Assumes inputs are decision variables, not raw or feature-level data streams.	No	Extends beyond decision-level aggregation by analyzing the full pipeline that transforms raw document data into fusible feature representations and decision outputs.

Table 1: Comparative analysis of this study against foundational Information Fusion learning surveys.

Figure 3: A graphical representation of the review method for the studies.

Search source	Number of records	Screening of articles policy	Number of articles selected for discussion and citation
Scopus	374	Inclusion criteria: study published in English, study focuses on document classification, study explicitly uses information fusion techniques, article is peer reviewed. Exclusion criteria: Study published in a language other than English, study not related to document classification, study does not use information fusion.	139

Table 2: Search strategy and results for article acquisition, 2025-01-03.

Figure 4: The left plot presents the number of publications per year from 2001 to 2024; the right plot presents the distribution of articles across different publication sources, including journals, books, and conference proceedings.

Figure 5: The left plot presents the number of publications on multimodal, multiview, and both learning approach; the right plot presents the time distribution of articles across those categories.

Figure 6: The progression from raw data, through features and representations, to pattern recognition in document classification tasks. Features are extracted from raw data and encoded into representations, which then enable pattern processes.

	F (Format)	E (Encoding)	M (Meaning)
$R_{\text{abstract}}$	Linear text block	Dense summary encoding	Research overview meaning
$R_{\text{introduction}}$	Multiparagraph structure	Problem–context encoding	Research motivation meaning
$R_{\text{methods}}$	Structured procedure format	Technical detail encoding	Methodological approach meaning
$R_{\text{document}}$	Hierarchical composition of section formats	Integrated section encodings	Complete research contribution meaning

Table 3: An example of hierarchical organisation of representation levels.

Term	Definition	Examples in document classification
Feature	A measurable conceptual property or attribute of the document being measured (e.g. semantic meaning, writing style, topic) that can be used for classification.	Term importance, semantic similarity, document sentiment, readability, topical relevance, authorial style, and structural organisation (e.g. Introduction, Methods, Results, and Discussion (IMRAD) structure)
Representation	A structured, computational encoding of features derived from raw data that enables pattern analysis and the derivation of higher-level features. In other words, the data structure or mathematical object used to encode one or more features.	Basic structures: Scalar (document length); Vector (word embeddings); Matrix (term-document metrics); Tensor (contextual embeddings) Complex structures: Neural networks (BERT embeddings or intermediate activations in the network at different layers); Graph structures (citation networks); Tree structures (section hierarchies); Probabilistic models (topic distributions)
Pattern	A recurring, interpretable structure in representations that supports classification decisions.	Standard section ordering in research articles, characteristic word distributions in scientific material, citation patterns in academic communities, consistent formatting in technical documents

Table 4: Core concepts of pattern recognition in document classification: the progression from features through representations to patterns. Each concept is illustrated with examples that are relevant to document classification tasks.

Figure 7: A multimodal learning framework that illustrates the integration of different data modalities (text, image, audio) through modality-specific processing paths before fusion.

Figure 8: A multiview learning framework that illustrates how different representations of the same data type are processed and combined for classification.

Figure 9: Fusion strategies connected to the

R=(F,E,M)

framework: (a) early fusion combines encodings at the

E

level; (b) late fusion combines classifier outputs at the

RR

level; (c) hybrid fusion integrates features at both

E

and

RR

levels.

Finding	Description	Citations
Performance boost with multimodality	Combining text with other modalities (images, audio, or structured data) consistently improves classification accuracy (e.g. 5–15% gains), particularly in complex domains.	[ 11 ] , [ 52 ] , [ 114 ] , [ 30 ] , [ 64 ] , [ 86 ] , [ 19 ] , [ 90 ] , [ 6 ]
Text dominance in unimodal settings	When individual modalities are compared, text-based classifiers often outperform those that rely solely on other single modalities (e.g. image or audio), particularly for semantic tasks.	[ 95 ] , [ 82 ] , [ 109 ] , [ 99 ] , [ 4 ]
Evolution of fusion strategies	Methodologies for combining modalities have advanced from simple concatenation to sophisticated techniques like attention, late fusion, and novel pooling/interaction architectures.	[ 110 ] , [ 6 ] , [ 77 ] , [ 85 ] , [ 86 ]
Impact of pretrained models	Powerful pretrained language models (PLMs) are highly effective; incremental gains from other modalities can diminish, but techniques like cross-modal contrastive learning & LLM+vision integration show promise.	[ 91 ] , [ 24 ] , [ 149 ] , [ 14 ]
Strengths of multimodal approaches	Multimodal systems can: capture complementary information, offer robustness to noise/missing data, and show potential for cross-domain/lingual generalisation.	[ 7 ] , [ 49 ] , [ 130 ] , [ 83 ] , [ 90 ] , [ 14 ] , [ 32 ] (note: [ 40 ] highlights limits of cross-lingual generalisation)
Weaknesses and limitations	Higher computational complexity, difficulty achieving true synergistic cross-modal interaction, class imbalance issues, and sensitivity to data quality (e.g. OCR errors).	[ 64 ] , [ 12 ] , [ 54 ] , [ 81 ] , [ 113 ] , [ 59 ]

Table 5: Key findings in multimodal document classification.

Challenge	Description	Citations
Multimodal integration & fusion	The effective combination of heterogeneous data types (e.g. text, image, layout, or audio) into a unified representation that captures synergistic information. This includes the selection of appropriate fusion strategies (early, late, hybrid) and the design of robust feature fusion modules.	[ 134 ] , [ 77 ] , [ 42 ] , [ 43 ] , [ 147 ] , [ 34 ] , [ 156 ] , [ 2 ] , [ 14 ] , [ 27 ]
Cross-modal alignment & correspondence	The establishment of meaningful correspondences and semantic relationships between different modalities, particularly when dealing with varying granularities (e.g. image to sentence vs. image to document), implicit relationships, or sparse multimodal signals.	[ 85 ] , [ 86 ]
Data scarcity & dataset creation	The considerable difficulty, cost, and time associated with creating, curating, and annotating large-scale, high-quality multimodal datasets, particularly for specialised domains, low-resource languages, or fine-grained tasks. This often necessitates the creation of in-house datasets.	[ 42 ] , [ 134 ] , [ 114 ] , [ 110 ] , [ 25 ] , [ 101 ] , [ 6 ] , [ 99 ] , [ 142 ] , [ 102 ] , [ 111 ]
Data quality, noise, and bias	Dealing with inherent issues in data such as label noise/errors, OCR inaccuracies, imbalanced class distributions, missing data, informal/mixed language, irrelevant content, and biases embedded in datasets, all of which can hinder model training, performance, and generalisation.	[ 90 ] , [ 40 ] (OCR), [ 59 ] (OCR), [ 11 ] (text extraction), [ 66 ] (OCR), [ 30 ] (quality), [ 83 ] (intrinsic noise), [ 130 ] (noisy data), [ 81 ] (noisy/unbalanced), [ 113 ] (transcription errors), [ 111 ] (quality, imbalance), [ 60 ] (imbalance), [ 29 ] (noise), [ 22 ] (ASR errors)
Low-resource languages & cross-lingual transfer	The adaptation of development of models for languages with limited linguistic resources (corpora, pretrained models) and datasets. The achievement of effective cross-lingual transfer of knowledge from high-resource languages, particularly to syntactically distant ones.	[ 114 ] , [ 32 ] , [ 40 ] , [ 101 ] , [ 6 ] , [ 111 ]
Computational demands & scalability	High computational requirements (CPU/GPU, memory) and extensive training time for complex multimodal models. Challenges in the deployment of large models efficiently, particularly on resource-constrained devices or for real-time applications.	[ 41 ] , [ 115 ] , [ 42 ] (model size), [ 43 ] ( a solution to high computational demand), [ 52 ] , [ 12 ] , [ 64 ] , [ 132 ] , [ 2 ]

Table 6: Key challenges in multimodal document classification.

Finding	Description
Common themes & patterns
Performance superiority of multiview approaches	Across several studies, these methods consistently outperformed traditional singleview methods [ 150 , 17 , 97 , 103 ] .
Integration of complementary information sources	Effectiveness stems from the leveraging of diverse, complementary information from different modalities or views [ 7 , 62 ] .
Efficacy in cross-lingual & cross-cultural applications	Multiview learning holds considerable promise for tasks that involve multiple languages, often by leveraging parallel corpora to learn shared representations [ 108 , 8 , 17 , 51 , 9 ] . For nonparallel or comparable corpora, other methods use machine translation to generate the missing views, thereby creating a multiview setting [ 10 ] .
Synergies with semi-supervised & active learning	Multiview approaches combine effectively with semi-supervised or active learning paradigms to enhance classification with limited labelled data [ 67 , 148 , 61 , 92 , 121 , 122 , 36 , 58 , 46 ] .
Prevalent strengths

Table 7: Key findings in multiview learning for document classification.

Challenge	Description
Data-related challenges
Data scarcity & annotation cost	Insufficient labelled data; annotation is labour-intensive, time-consuming, and requires domain expertise [ 62 , 73 , 3 , 84 , 125 ] .
Data quality & representation	Issues with sparse (particularly short text) data, noise, incompleteness or missing views; informal language on social media; quality of artificially generated views [ 89 , 84 , 65 , 33 , 80 ] .
Class imbalance	Severely skewed class distributions—particularly in domain-specific applications—degrade model performance [ 57 , 7 , 84 ] .
Model architecture & complexity
Optimisation & parameter tuning	Nonconvex objectives, many hyperparameters, and complex structures are challenging to optimise—particularly with limited data [ 145 , 56 , 151 , 17 , 51 ] .

Table 8: Key challenges in multiview document classification.

Figure 10: An overview of evaluation practices in multimodal experiments: The top-left plot presented the frequency distribution of the datasets; the top-right plot presents the frequency distribution of the evaluation metrics; and the bottom plot displays the distribution of the baseline models used for comparison.

Figure 11: An overview of the validation practices in multimodal experiments. The plots depict use of: a held-out test set or cross-validation (top-left); statistical tests (top-right); alternative analyses like Bayesian methods (centre); ablation studies (bottom-left); and the provision of sufficient information or source code for replication (bottom-right).

Metric	Stat.	N	Min	Q1	Median	Mean	Q3	Max	SD
Accuracy
Accuracy	Mean	27	-0.33	1.81	2.70	5.62	5.86	26.87	7.03
Accuracy	Median	27	-1.00	1.65	2.70	5.51	5.52	28.09	7.26
F1-score
F1-score	Mean	16	0.88	1.63	3.31	3.93	4.95	9.38	2.83
F1-score	Median	16	0.75	1.44	3.35	3.91	4.91	9.55	2.89

Table 9: Descriptive statistics for study-level performance differences.

Figure 12: The distribution of performance gains for multimodal over unimodal approaches. The box plots visualise the distribution of study-level performance differences (multimodal–unimodal) across two key metrics. The plots are faceted by the summary statistic used at the study level: mean differences (top panel) and median differences (bottom panel). The horizontal red dashed line at y=0 represents the point of no difference. The black dots depict the individual difference values from each study. The box indicates the interquartile range (IQR), the central line depicts the median of the distribution, and the whiskers extend to 1.5 times the IQR. The number of studies (N) for each metric is presented in Table 9 .

		Meta-analysis (RQ4)			Permutation (RQ1)			Wilcoxon (RQ2)			Cliff’s delta (RQ3)
Metric	N	Est. $\Delta$ [95% CI]	p	Sig.	$d_{Cohen}$	p	Sig.	r [95% CI] (V)	p	Sig.	$d_{Cliff}$ [95% CI]
Accuracy	27	5.28 [2.23, 8.32]	0.0016	**	0.80	<.001		0.99 [0.97, 0.99] (376)	<.001		0.31 [-0.01, 0.57]
F1-score	16	3.29 [-0.0237, 6.6]	0.0511		1.39	<.001		1 [1, 1] (136)	<.001		0.27 [-0.15, 0.60]

Table 10: A definitive summary of statistical analyses on multimodal performance gains. N: Number of studies. Est. (

\Delta

): Mean difference from meta-analysis.

d_{Cohen}

: Cohen’s d effect size.

d_{Cliff}

: Cliff’s delta effect size.

r

: Rank-biserial correlation. Significance for respective p-values: *** p < 0.001, ** p < 0.01, * p < 0.05.

Figure 13: An overview of the evaluation practices in multiview experiments: The top-left plot depicts the frequency distribution of the datasets; the top-right plot presents the frequency distribution of the evaluation metrics; and the bottom plot displays the distribution of the baseline models used for comparison.

Figure 14: An overview of the validation practices in multiview experiments. The plots depict use of: a held-out test set or cross-validation (top-left); statistical tests (top-right); alternative analyses like Bayesian methods (centre); ablation studies (bottom-left); and the provision of sufficient information or source code for replication (bottom-right).

Metric	Stat.	N	Min	Q1	Median	Mean	Q3	Max	SD
Accuracy
Accuracy	Mean	20	-0.20	1.46	2.14	4.34	5.29	26.99	5.99
Accuracy	Median	20	-0.20	1.26	2.35	4.28	5.29	26.99	5.98
F1-score
F1 Score	Mean	17	-6.15	1.92	2.24	3.28	4.60	11.47	3.68
F1 Score	Median	17	-6.15	2.00	2.64	3.20	4.50	12.20	3.75

Table 11: Descriptive statistics for study-level performance differences.

Figure 15: The distribution of performance gains for multiview over singleview approaches. The box plots visualise the distribution of study-level performance differences (multiview–singleview) across four key metrics. The plots are faceted by the summary statistic used at the study level: mean differences (top panel) and median differences (bottom panel). The horizontal red dashed line at y=0 represents the point of no difference. The black dots depict the individual difference value from each study. The box indicates the interquartile range (IQR), the central line depicts the median of the distribution, and the whiskers extend to 1.5 times the IQR. The number of studies (N) for each metric is provided in Table 11 .

		Meta-analysis (RQ4)			Permutation (RQ1)			Wilcoxon (RQ2)			Cliff’s delta (RQ3)
Metric	N	Est. $\Delta$ [95% CI]	p	Sig.	$d_{Cohen}$ :	p	Sig.	r [95% CI] (V)	p	Sig.	$d_{Cliff}$ [95% CI]
Accuracy	20	4.67 [0.37, 8.98]	0.035		0.725	<.001		0.99 [0.97, 1.00] (209)	<.001		0.22 [-0.15, 0.54]
F1-score	17	3.08 [0.15, 6.02]	0.044		0.891	0.002	**	0.82 [0.54, 0.93] (139)	0.002	**	0.16 [-0.24, 0.52]
Precision	5	1.16 [-7.11, 9.43]	0.591		1.093	0.129		0.87 [0.34, 0.98] (14)	0.125		0.12 [-0.61, 0.74]
Recall	6	5.87 [1.55, 10.18]	0.020		1.000	0.032		1.00 [1.00, 1.00] (21)	0.031		0.22 [-0.47, 0.74]

Table 12: A definitive summary of statistical analyses on multiview performance gains. N: Number of studies. Est. (

\Delta

): Mean difference from meta-analysis.

d_{Cohen}

: Cohen’s d effect size.

d_{Cliff}

: Cliff’s delta effect size.

r

: Rank-biserial correlation. Significance for respective p-values: *** p < 0.001, ** p < 0.01, * p < 0.05.

Key conclusion	Implication
A unifying formal framework grounded in Information Fusion theory	The proposed frameworks for representation ( $R$ ), pattern ( $P$ ), and model ( $M$ ) act as a theoretical bridge. They enable researchers to map ad-hoc machine learning architectures (e.g., attention, ensembles) onto rigorous fusion paradigms—such as Bayesian opinion pools and Dempster–Shafer evidential reasoning—facilitating systematic comparison and theoretical validation.
Quantified, but nuanced performance gains	Fusion is not a panacea. Multimodal approaches yield a clear benefit for accuracy (+5.28%, $p=.0016$ ) but a fragile one for F1-score. Multiview approaches demonstrate more consistent, albeit modest, gains across accuracy, F1-score, and recall. The selection of a fusion strategy must be context-dependent.
Challenges in reproducibility	The validity of many reported gains is questionable. With only 11.8% (multimodal) and 23.3% (multiview) of the studies employing statistical tests, and even fewer sharing replication materials, the field suffers from a lack of transparency and statistical validation that hinders cumulative progress.
Strategic guidance over algorithmic superiority	The most successful applications do not hinge on a single ‘best’ algorithm, but on the strategic alignment of the fusion method with the task context (e.g. using complementary vs. redundant information) and the thoughtful integration of powerful pretrained models.

Table 13: Key conclusions and their implications.

Data Fusion

See all 24 papers

Jun 25, 2026

When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction. However, it remains unclear whether these scores influence model decisions or merely correlate with performance. We propose a simple diagnostic to test whether reliability information is used during inference. After training, the model and inputs are fixed while reliability scores are permuted across test examples. If predictions depend on these scores, performance should degrade. Experiments on StressID for stress recognition and CMU-MOSEI for sentiment analysis show that permuting reliability scores leaves performance unchanged despite substantial potential gains from selecting the best modality per example. In positive controls where reliability signals identify the correct modality, the same frozen fusion rules yield significant improvements, indicating that reliability signals influence fused decisions only when they reliably predict unimodal correctness.

Jaden Moon, Arvind Pillai, Andrew Campbell

Dartmouth College, United States

Uncertainty-Aware Multimodal Fusion Multimodal Fusion Multimodal Model Evaluation Modality Reliance in Multimodal Models Data Fusion

Ask PDF

Jul 14, 2026

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose SeRIn (Segregate, Refine, Integrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.

Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2

National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2

Cross-Modal Representation Learning Multimodal Large Language Models Multimodal Sentiment Analysis Multimodal Fusion Data Fusion

Ask PDF

Jun 1, 2026

Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysis

Document type classification in visually rich documents remains challenging, as relevant information is distributed across textual, visual, and layout modalities. To capture this complexity, current approaches rely on diverse multimodal modeling strategies, resulting in heterogeneous architectures that complicate systematic comparison. This variability is also reflected in existing comparative studies, which often rely on heterogeneous evaluation setups, further complicating systematic comparison and making it difficult to assess progress. To address these limitations, this work provides a structured analysis of multimodal design strategies across transformer- and LLM-based architectures, combined with a controlled empirical comparison within a unified experimental framework. Specifically, four representative models (LayoutLMv3, Donut, Qwen3-VL-32B-Instruct, and Qwen3-32B) are evaluated on the RVL-CDIP benchmark to systematically analyze the contributions of text, image, and layout information for document type classification, with a particular focus on contrasting OCR-dependent and OCR-free approaches. The results show that specialized multimodal Transformers outperform LLM-based approaches on visually rich and layout-intensive documents. Image information contributes most strongly to reliable classification, while OCR-derived text provides useful but secondary support. These findings highlight that multimodal processing remains essential for documents with pronounced layout structure. Overall, the study provides a systematic basis for comparing multimodal architectures and offers practical guidance for selecting effective feature combinations and model designs for document type classification.

Catyana Heyne, Jürgen Frikel, Filippo Riccio

OTH Regensburg, Germany

Document Understanding Multimodal Classification Multimodal Model Evaluation Document Image Understanding

Ask PDF

Abstract

Data Fusion

Explore similar work

When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysis