When MLLMs miss what humans read: diagnosing and learning to read squint text

Chen Peng; Shan Huang; Jianbo Huang; Linrui Qin; Qibo Sun; Mengwei Xu

cs.CVOct 9, 2026

When MLLMs miss what humans read: diagnosing and learning to read squint text

Authors: Chen Peng, Shan Huang, Jianbo Huang, Linrui Qin, Qibo Sun, Mengwei Xu

Organizations: State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · Tencent Anti-Cheat Expert, China

Abstract

Multimodal LLMs can fail to read text embedded in an image's coarse structure (text a human recovers by squinting) and report that no text is present. We name this class Squint Text and study it as a recognition deficiency, motivated by cases from a production moderation pipeline. We introduce SquintBench, comprising 1076 production-derived moderation images, and SquintCtrl, a controlled synthetic suite for mechanism diagnosis. Frozen-weight analysis shows that hidden text leaves a coarse signal while glyph identity is weakly represented; identity becomes substantially more accessible when the correct spatial regions are supplied, implicating spatial selection as a major bottleneck. Suppressing high-frequency content also improves reading at a fixed visual-token budget, while visual token count may also contribute. These findings motivate Squint OPSD: a frozen teacher receives a downscaled view and the ground-truth transcription as privileged context, and its token-level distributions supervise a student along on-policy trajectories. At inference, the student reads the original image alone. Squint OPSD reaches 88.61 ANLS on SquintBench, versus 26.50 for the unadapted backbone, 24.44 for the best general-purpose model on the original view, and 68.60 at best for an off-the-shelf model under a downscaled view. These results substantially narrow the gap between available visual evidence and its generated read-out without auxiliary views at inference.

Figures & tables

Figure 1: Squint Text , and why the deployed defence does not remove it. (a) Pipeline images whose embedded string a human reads—directly or by stepping back—while GPT, Gemini and Claude each answer “No text in this image” , on the original and the scaled view alike. (b) Downscaling before review recovers only part of the hidden text for every off-the-shelf model.

Figure 2: Three diagnostics on frozen off-the-shelf weights. (a) Restricting the read-out to glyph-bearing tokens improves glyph-identity prediction, supporting spatial selection as a major bottleneck (per-layer expansion in Appendix A ). (b) At a fixed 64-token budget, suppressing high-frequency content raises squint-text ANLS from 29.2 to 67.0 (per- $\sigma$ results in Appendix B ). (c) For the same target glyph 搜, adding a $1/8$ -scale view of the same image shifts the base model’s phonetic predictions toward visually matching glyphs, indicating stronger visual grounding.

	D1	D2	D3
ground-truth SFT	✓	✗	✗
sparse-reward RL	✓	✗	✗
downscaled view at inference	✗	—	—
ground-truth-conditioned teacher	✓	✗	✓
Squint OPSD (view $+$ hint teacher)	✓	✓	✓

Table 1: Candidate solutions under the requirements of § 3.4 : D1 uses original-only inference, D2 uses localization-bearing supervision, and D3 uses a distributional target. A dash denotes a method with no training target. Only Squint OPSD satisfies all three.

Figure 3: Overview of Squint OPSD . The teacher receives the original image, the downscaled view $v=g(x)$ , the task instruction, and the ground-truth output as privileged context. Its token distributions supervise the student along stop-gradient, student-generated trajectories. Gradients flow only through the student logits, and inference uses the original image alone.

	SquintBench	general
		AI2D	DocVQA	OCRBench
Base	26.50	86.66	95.93	879
Base $+1/8$ view (two-view)	58.96	52.36	95.37	834
SFT (full, single)	72.47	85.52	64.73	722
DAPO	73.55	82.12	92.31	728
Squint OPSD	88.61	85.33	92.98	833

Table 2: SquintBench performance and general vision-language capability.

benchmark	cond.	Squint OPSD	Qwen3.5- 9B base	claude- opus-5	gemini- 3.1-pro	gpt-6- astra	qwen3-vl- 235b-think
SquintBench	orig.	88.61	26.50	22.26	18.15	8.33	24.44
	$1/8$ only	49.64	38.22	22.41	27.90	8.13	34.37
	$40{\times}40$ only	76.12	68.60	11.07	44.91	6.26	66.88
IllVQA Illusion	WER $\downarrow$	67.00	178.10	166.50	190.13	119.60	177.97
IllVQA Filtered	WER $\downarrow$	64.87	76.80	83.53	70.00	65.27	91.33
AdvOCR †	orig.	66.7	3.7	3.7	3.7	11.1	3.7

Table 3: Comparison with the base model and off-the-shelf MLLMs on SquintBench and three external hidden-text benchmarks. WER is lower better. The “only” conditions replace the original image with the reduced view, applied before the model preprocessor; further protocol details are in Appendix B.1 .

	configuration	SquintBench
teacher input
	$x$ (neither)	27.96
	$x+h$ (hint only)	41.79
	$x+v$ (privileged view only)	54.68
	$x+v+h$ (default)	88.61
	$x+v_{\rm mis}+h$ (mismatched view) †	78.60

Table 4: Ablations on SquintBench . We vary teacher input, target form, and trainable modules. † The mismatched-view row is paired with a matched-view run under the same setting (84.60), giving a 6.0-point drop.

Figure 4: What changes after training. (a) Example next-token distributions for Base , SFT, and Squint OPSD . (b) Attention concentration on glyph-bearing regions and a causal visual-token embedding knockout. (c) Candidate-distribution structure on 120 matched glyph positions.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Glyph identity is more accessible at the correct locations. Glyph-identity macro-AUC on SquintCtrl-Glyph by encoder block under the three poolings of § 3.2 . random uses the same number of tokens as oracle but remains close to global across depth and script.

script	checkpoint	before	after	$\Delta$
Chinese	Base	0.707	0.700	$-0.007$
Chinese	Squint OPSD	0.989	0.985	$-0.004$
Latin	Base	0.555	0.573	$+0.018$
Latin	Squint OPSD	0.699	0.713	$+0.014$

Appendix

Table 5: Glyph separability before and after the

2{\times}2

patch merger.

	same pipeline		different pipelines
block	Base	Squint OPSD	Base	Squint OPSD
L6	0.964	0.977	0.996	0.996
L12	0.977	0.995	0.995	0.998
L18	0.981	0.995	0.993	0.996
L24	0.967	0.991	0.991	0.995
L27	0.956	0.991	0.984	0.996

Appendix

Table 6: Presence-probe AUC on SquintCtrl-Pair . The controlled columns use the same generation path for both members of each pair.

script	Base	Squint OPSD	$\Delta$
Chinese	0.718	0.975	$+0.257$
Latin	0.612	0.779	$+0.167$

Appendix

Table 7: Glyph-identity macro-AUC at L18 before and after training.

band (c/p)	orig.	$1/8$	$40{\times}40$
$0$ – $1/32$	91.3%	94.3%	96.5%
$1/32$ – $1/16$	6.5%	4.8%	3.3%
$1/16$ – $1/8$	1.7%	0.87%	0.21%
$>1/8$	0.5%	0.08%	0.03%

Appendix

Table 8: Share of non-DC spectral energy by band, in cycles per pixel, for the three SquintBench viewing conditions.

Figure 6: Frequency at constant token count. Hidden-text reading ANLS on all 1076 SquintBench images under Gaussian low-pass filtering. The dashed line is the unfiltered input; the visual-token count is unchanged throughout the sweep.

transform	what it does	ANLS	$\Delta$
CLAHE	raises contrast, spectrum intact	25.91	$-0.53$
bilateral filter	flattens texture, keeps glyph edges	32.22	$+5.79$
posterise ( $3$ levels)	compresses range and texture	37.21	$+10.77$
band-stop	removes mid, keeps low $+$ high	11.22	$-15.22$

Appendix

Table 9: Contrast and band-selective controls on SquintBench , against the unfiltered 26.44 ANLS baseline.

teacher configuration	corrective view $v$	SquintBench
default	$1/8$	88.61
fixed view	$40{\times}40$	88.09
fixed view	$1/4$	78.51
probability mean	$1/8,\ 1/4,\ 40{\times}40$	83.05
max-entropy selection	$1/8,\ 1/4,\ 40{\times}40$	80.51
min-entropy selection	$1/8,\ 1/4,\ 40{\times}40$	74.30

Appendix

Table 10: Teacher-view variants. All Squint OPSD teachers receive

(x,v,q,h)

; only the corrective view

v

or its aggregation is varied. Student inference always uses

(x,q)

only.

Figure 7: Recoverability versus hiding strength. Reading accuracy for the base model and Squint OPSD as conditioning strength varies, over the same 200 scenes. Stars mark strength levels at which the 95% CI of the paired base-vs- Squint OPSD difference excludes zero (paired bootstrap, 100 resamples); unmarked levels are not significant at this threshold, and 0.7 is borderline and is left unmarked. The method provides its clearest benefit in the partially recoverable regime; the two ends saturate for different reasons.

metric	Base	SFT	Squint OPSD
$\rho_{\mathrm{vis}}$ , own valid set	0.079 (254)	0.160 (259)	0.243 (164)
$\rho_{\mathrm{vis}}$ , common set	0.126	0.242	0.286
$\rho_{\mathrm{lang}}$ , common set	0.555	0.334	0.544

Appendix

Table 11: Candidate-distribution structure before and after matching glyph positions across checkpoints. Parentheses give the number of valid positions.

intervention	Base	Squint OPSD
none (baseline)	26.0	90.5
replace glyph tokens, $d{=}0$	19.5	87.0
replace glyph tokens, $d{=}1$	16.0	77.0
replace glyph tokens, $d{=}2$	15.0	71.5
replace random tokens, $d{=}0$	25.0	91.5
replace random tokens, $d{=}1$	25.0	91.0

Appendix

Table 12: Reading accuracy after visual-token embedding replacement. Glyph-region embeddings are replaced by the mean embedding of non-glyph tokens from the same image; the random control replaces an equal number of randomly selected background-token embeddings using the same operation.

d

denotes the number of

3{\times}3

dilations applied to the known glyph mask.

group	sources	prompts
hidden text (synthetic)	AIGC collection 1143, self-synthesised ControlNet 1144,
	FakeTextDetect 1267, minus 65 removed	3489
open-source OCR	TextCaps 564, DocVQA 302, CASIA 150, RCTW 136,
	POIE 56, FUNSD 3, ICDAR13 2	1213
VQA	OCR-VQA	1203
total		5905

Appendix

Table 13: Training mixture, 5905 prompts after removing 65 samples with corrupted annotations. DocVQA contributes train-split samples only; its reported evaluation uses a disjoint test split.

hyperparameter	value
optimizer / learning rate	AdamW ( Loshchilov & Hutter, 2019 ) , $1{\times}10^{-5}$
schedule / warmup	constant / 10 steps
weight decay / grad clip	0.01 / 1.0
prompt batch	56 prompts per step
rollouts per prompt	$n{=}8$ (448 sequences/step)
inner epochs	1

Appendix

Table 14: Main Squint OPSD optimization and rollout settings.

trainable modules	SquintBench	AI2D	DocVQA	OCRBench
LM only; ViT frozen	27.29	86.66	50.39	741
ViT + LM	72.47	85.52	64.73	722

Appendix

Table 15: Effect of visual adaptation under three-epoch supervised fine-tuning.

Panel of Figure 8 — Figure 8: Two examples from each of the four SquintCtrl subsets. The embedded strings are carried in the images’ coarse structure: they are hard to notice at full size but become legible when the page is squinted at or viewed at reduced scale. SquintCtrl-Pair : a text-present image and its text-free counterpart, regenerated through the same ControlNet branch with an all-black conditioning image. SquintCtrl-Glyph : conditioning on the closed vocabularies of ten Chinese glyphs and ten Latin letters. SquintCtrl-Strength : the same type of content under weak versus strong conditioning. SquintCtrl-Read : arbitrary open-vocabulary strings.

Panel of Figure 9 — Figure 9: SquintBench deployment examples that commercial models miss. Four production-derived moderation images, shown together with the answers of two frontier commercial MLLMs and Squint OPSD to the transcription instruction. Both commercial models report that no text is present on the original image, whereas Squint OPSD reads out the embedded promotional strings; the same strings become legible to a human when the page is squinted at or viewed at reduced scale. These examples come from the production-derived portion of SquintBench and contain no personal data.

Explore similar work

SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read
Feb 25, 2026Yibo Peng, Peng Xia, Ding Zhong +6Multimodal Large Language Models Optical Character Recognition
UNC-Chapel Hill · Carnegie Mellon University · University of Michigan +1
Can MLLMs "Read" What is Missing?
Apr 23, 2026Jindi Guo, Chaozheng Huang, Xi FangVLM Evaluation Document Understanding
DP Technology
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1

Feb 25, 2026J/K move · Enter open · S save

SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read

Yibo Peng, Peng Xia, Ding Zhong, Kaide Zeng+5

UNC-Chapel Hill · Carnegie Mellon University · University of Michigan+1

Multimodal Large Language Models Optical Character Recognition Visual Question Answering

Open paper AskPDF arXiv

Despite the rapid advancements in Multimodal Large Language Models (MLLMs), a critical question regarding their visual grounding mechanism remains unanswered: do these models genuinely

read'' text embedded in images, or do they merely rely on parametric shortcuts in the text prompt? In this work, we diagnose this issue by introducing the Visualized-Question (VQ) setting, where text queries are rendered directly onto images to structurally mandate visual engagement. Our diagnostic experiments on Qwen2.5-VL reveal a startling capability-utilization gap: despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting, exposing a deep-seated

modality laziness.'' To bridge this gap, we propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process. By transforming training samples into the VQ format with randomized styles, SimpleOCR effectively invalidates text-based shortcuts, compelling the model to activate and optimize its visual text extraction pathways. Empirically, SimpleOCR yields robust gains without architectural modifications. On four representative OOD benchmarks, it surpasses the base model by 5.4% and GRPO based on original images by 2.7%, while exhibiting extreme data efficiency, achieving superior performance with 30x fewer samples (8.5K) than recent RL-based methods. Furthermore, its plug-and-play nature allows seamless integration with advanced RL strategies like NoisyRollout to yield complementary improvements. Code is available at https://github.com/aiming-lab/SimpleOCR.

Abstract

Explore similar work

SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read

Can MLLMs "Read" What is Missing?

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs