Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding

Haruka Kawasaki; Ryota Tanaka; Kyosuke Nishida

cs.CLApr 6, 2026

Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding

Authors: Haruka Kawasaki, Ryota Tanaka, Kyosuke Nishida

Organizations: Human Informatics Labs., NTT, Inc.

Abstract

Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning over structured layouts. Although recent LVLMs have shown progress on VDU benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.

Figures & tables

Figure 1: Overview of our analysis. We analyze the gap between the internal representations and responses in VDU. For the analysis of internal representations, we employ linear probing and construct classifiers at each layer. For the response, we evaluate the accuracy of text responses.

Figure 2: Examples of linear probing tasks. We use four linear probing tasks covering different aspects of VDU. These include visual attributes recognition, which targets properties such as color and shape; word recognition, which focuses on identifying spelling differences between the word in the image and the query; structure understanding, which asks about the document component highlighted in the image; and figure understanding, which requires reasoning over graphical elements in charts.

Figure 3: Linear probing accuracy at each layer (line plot) and text-response accuracy (horizontal black dotted line). Token types used in linear probing are divided into four categories: image-token , text-token , all-token , and last-token . The vertical axis represents accuracy, and the horizontal axis corresponds to the LLM layers, plotted every two layers.

Metrics	Base	All	Lower	Middle	Upper	L–M	M–U	L $\rightarrow$ M	M $\rightarrow$ L	M $\rightarrow$ U	U $\rightarrow$ M
	(0)	(100)	(20)	(48)	(31)	(69)	(80)	(35)	(35)	(40)	(40)
Visual attributes recognition
Response accuracy ( $\uparrow$ )	67.96	92.29	65.82	93.17	68.60	90.48	91.43	76.55	95.62	89.09	88.49
Linear probing accuracy ( $\uparrow$ )	92.78	96.60	93.24	96.39	93.12	96.16	96.65	96.10	98.12	94.70	98.23
Gap ( $\downarrow$ )	24.82	4.31	27.42	3.22	24.52	5.68	5.22	19.55	2.50	5.61	9.74
Word recognition

Table 1: Response accuracy and linear probing accuracy and the gap between them of models fine-tuned intermediate layers on classification tasks. Bold values indicate the best for each task, and underlined values indicate the second-best. (

\cdot

) represents the effective amount of trainable parameters, normalized such that full-layer tuning for the entire training corresponds to 100. The value reflects both the number of unfrozen parameters and the number of epochs for which they are trained. Accuracy is reported in percentages.

Metrics	Base	All	Lower	Middle	Upper	L–M	M–U	L $\rightarrow$ M	M $\rightarrow$ L	M $\rightarrow$ U	U $\rightarrow$ M
	(0)	(100)	(20)	(48)	(31)	(69)	(80)	(35)	(35)	(40)	(40)
DocVQA
ANLS ( $\uparrow$ )	0.9380	0.9474	0.9379	0.9451	0.9356	0.9430	0.9492	0.9418	0.9386	0.9477	0.9410
Training time	00:00	60:44	30:16	34:52	31:49	37:42	38:25	30:14	29:42	30:16	30:51
InfographicVQA
ANLS ( $\uparrow$ )	0.8352	0.8337	0.8320	0.8341	0.8299	0.8298	0.8393	0.8295	0.8306	0.8349	0.8306

Table 2: Response quality and training time of models fine-tuned intermediate layers on document VQA tasks. ANLS and total training time (minutes and seconds) are reported. For two-step training, training time values represent the sum of the first step (up to the corresponding checkpoint) and the second step. (

\cdot

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 404 papers

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
Jul 8, 2026Miguel Lopez-Duran, Elena Marrero, Julian Fierrez +8VLM Evaluation Document Understanding
BiometricsAI, Universidad Autónoma de Madrid (UAM), Spain
Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding
May 19, 2026Peter El Hachem, Ahmed Nassar, A. Said Gurbuz +2VLM Robustness LLM Prompting
ETH Zurich Rämistrasse, Switzerland · IBM Research Rüschlikon, Switzerland
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University

Jul 8, 2026J/K move · Enter open · S save

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

Miguel Lopez-Duran, Elena Marrero, Julian Fierrez, Marta Robledo-Moreno+7

BiometricsAI, Universidad Autónoma de Madrid (UAM), Spain

VLM Evaluation Document Understanding Visual Question Answering VLM Adaptation Few-Shot Learning

Open paper AskPDF arXiv

Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored. In this study, we present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains: industrial documents of varying type, infographics, and presentation slides. We systematically assess model performance under zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot learning evaluations of knowledge transfer between domains. Our findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides. Although parameter scaling is a dominant factor on performance, supervised finetuning yields higher relative gains in smaller architectures. Furthermore, our cross-domain and few-shot experiments show that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs. Using 50 target domain samples, the models finetuned in DocVQA with datasets of different domains rapidly adapt to the target domain documents, even surpassing their fully supervised counterparts in some cases.

Abstract

Large Vision-Language Models

Explore similar work

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?