Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Models

Longwei Xu; Feng Feng; Shaojie Zhang; Xin Chen; Hang Li; Anan Du; Hailong Yu; Pei Fu

cs.CVMar 31, 2026

Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Models

Authors: Longwei Xu, Feng Feng, Shaojie Zhang, Xin Chen, Hang Li, Anan Du, Hailong Yu, Pei Fu, +2 more

Organizations: MiLM Plus, Xiaomi Inc

Abstract

Optical Character Recognition (OCR) is increasingly regarded as a foundational capability for modern vision-language models (VLMs), enabling them not only to read text in images but also to support downstream reasoning in real-world visual question answering (VQA). However, practical applications further require reliable text anchors, i.e., accurately grounding queried text to its corresponding spatial region. To systematically evaluate this capability, we introduce TextAnchor-Bench (TABench), a benchmark for fine-grained text-region grounding, which reveals that both general-purpose and OCR-specific VLMs still struggle to establish accurate and stable text anchors. To address this limitation, we propose Q-Mask, a precise OCR framework built upon a causal query-driven mask decoder (CQMD). Inspired by chain-of-thought reasoning, Q-Mask performs causal visual decoding that sequentially generates query-conditioned visual masks before producing the final OCR output. This visual CoT paradigm disentangles where the text is from what the text is, enforcing grounded evidence acquisition prior to recognition and enabling explicit text anchor construction during inference. To train CQMD, we construct TextAnchor-26M, a large-scale dataset of image-text pairs annotated with fine-grained masks corresponding to specific textual elements, encouraging stable text-region correspondences and injecting strong spatial priors into VLM training. Extensive experiments demonstrate that Q-Mask substantially improves text anchoring and understanding across diverse visual scenes.

Figures & tables

Figure 1: Performance comparison of mainstream general-purpose VLMs Bai et al. (2025a) ; Team et al. (2026a) ; DeepMind (2025) ; OpenAI (2025) and OCR-specific VLMs Wei et al. (2026b) on the proposed TABench. 2 2 footnotemark: 2

Figure 2: Comparison of the training paradigm of OCR-specific VLMs. (a) Standard VLMs Achiam et al. (2023) ; Li et al. (2024b) ; Bai et al. (2023) ; Tanaka et al. (2024) ; Wang et al. (2024c) ; Bai et al. (2025a) can recognize text but lack explicit mechanisms to establish reliable text anchors. (b) Existing mask-based methods Wang et al. (2025b) ; Guan et al. (2025) enhance spatial perception but fail to explicitly model text–region correspondence during inference. (c) Q-Mask introduces a causal query-driven mask decoder (CQMD) that explicitly grounds the queried text token within its spatial region prior to recognition.

Figure 3: Overview of the proposed architecture. After a LLM processes the concatenated visual and textual embeddings, the CQMD module extracts only the hidden states of visual tokens ( $\mathbf{H}_{img}$ ) and query tokens ( $\mathbf{H}_{q}$ ). Using cross-attention, Q-Mask predicts spatial masks prior to autoregressive answer generation. The model is trained with the next-token prediction (NTP) loss and a segmentation loss.

Figure 4: Overview of the TextAnchor-26M construction pipeline. We aggregate four data sources: (1) unconstrained scene text mined from large-scale web corpora; (2) academic documents (e.g., arXiv pages); (3) synthetic text rendered by SynthDog with multilingual fonts; and (4) VQA-with-causal-mask samples generated by prompting a VLM on precisely annotated regions. We first obtain transcripts and bounding boxes using expert models or rendering annotations, and then apply stochastic prior injection (SPI) and de-stylized mask rendering to produce unified supervision for Q-Mask training.

Figure 5: Empirical error profile of PPOCR-V5 on scene text. We decouple failures into box-level localization errors and character-level recognition errors. The character error rate is further decomposed into normalized proportions of insertion ( Ins ), deletion ( Del ), and substitution ( Sub ).

Model	Size	$Acc_{R2T}$ (%) $\uparrow$	$F1_{T2R}$ (%) $\uparrow$	Overall (%) $\uparrow$
Gemini 3.0 Pro DeepMind (2025)	Closed	25.85	62.58	44.22
GPT 5.2 OpenAI (2025)	Closed	10.64	0.64	5.64
Kimi K2.5 Team et al. (2026a)	1T	49.54	57.73	53.64
Qwen3.5 Qwen Team (2026)	397B	61.10	72.80	66.95
Qwen3-VL-Instruct Bai et al. (2025a)	235B	60.90	60.40	60.65
DeepSeek OCR2 Wei et al. (2026a)	3B	–	11.66	5.83

Table 1: Evaluation of representative models on TABench . We report Region-to-Text exact-match accuracy (

Acc_{R2T}

) and Text-to-Region localization

F1

(

F1_{T2R}

). Overall is defined as the arithmetic mean of

Acc_{R2T}

and

F1_{T2R}

. “–” indicates the metric is not applicable because the model does not support the corresponding output format. For models that do not support one task direction, the missing metric is counted as 0 when computing Overall.

Model	Size	ChartQA	DocVQA	InfoVQA	TextVQA	OCRBench
Closed-source Generalist Models
Gemini 3.0 Pro DeepMind (2025)	Closed	-	-	57.2	-	940
GPT 5.2 OpenAI (2025)	Closed	57.0	91.7	84.0	72.8	807
Claude-3.5-Sonnet Anthropic (2024a)	Closed	90.8	95.2	74.3	74.1	788
GPT-4o Achiam et al. (2023)	Closed	85.7	92.8	66.4	70.5	736
Claude-3-Opus Anthropic (2024b)	Closed	80.8	89.3	55.6	67.5	694

Table 2: Comprehensive evaluation on downstream multimodal reasoning and fine-grained document understanding benchmarks. By establishing an explicit "Look before Read" spatial prior, our 3B model significantly elevates the baseline performance. Strikingly, our 3B model consistently outperforms heavily engineered OCR-specialized models (e.g., TokenVL-8B, DocOwl) and even rivals or surpasses open-source generalist MLLMs that are more than twice its size (e.g., 7B–8B). Bold indicates the best results among models under 10B parameters.

Table 8

Figure 6: Stage-1 spatial pre-training signals versus Stage-2 downstream performance under a fixed Stage-2 recipe.

Figure 7: Qualitative visualization of Q-Mask. (a) Stage 1 input. (b) Stage 1 output, where the green box indicates the predicted text region. (c) Stage 2 input with the query “What is the transaction date?” . (d) Stage 2 output visualized as a query-conditioned heatmap, highlighting candidate regions before the model generates the answer.

Figure S1 : Representative examples of the 12 categories covered by TABench.

Category	Samples	Ratio	R2T/T2R	Source
SceneText	2,760	50.6%	1,380/1,380	HierText 2,712; ICDAR2015 48
Receipt	560	10.3%	280/280	SVRD 560
Ticket	460	8.4%	230/230	SVRD 371; HierText 89
WarehouseSlip	390	7.2%	195/195	SVRD 390
Report	280	5.1%	140/140	SVRD 139; HierText 141
ChineseDocument	270	5.0%	135/135	CDLA 270

Table S1 : Statistics of TABench. Left: category-wise counts with strict 1:1 parity between R2T and T2R. Right: global source and language distributions.

Figure S2 : Illustration of the two tasks in TABench using representative single-instance examples. Region-to-Text (R2T) asks the model to read the text inside a specified region, while Text-to-Region (T2R) asks the model to localize the region corresponding to a queried text string.

Model family	Supported task(s)	Native coordinate/interface	Adaptation in TABench
Qwen-series	R2T + T2R	$(x_{\min},y_{\min},x_{\max},y_{\max})$ ; absolute or relative varies by version	coordinate conversion only
GLM-4.6V	R2T + T2R	$(x_{\min},y_{\min},x_{\max},y_{\max})$ ; relative coordinates in $[0,1000]$	use native relative-coordinate format
Gemini 3.0 Pro	R2T + T2R	$(y_{\min},x_{\min},y_{\max},x_{\max})$	reorder prompt/parser to native convention
Kimi K2.5	R2T + T2R	normalized coordinates in $[0,1]$	convert GT/predictions to normalized $[0,1]$ format
DeepSeek OCR2	T2R	grounding prompt with <\|ref\|> tags	rewrite natural-language query to native grounding prompt

Table S2 : Model-specific prompt and interface adaptations used in TABench evaluation. We only adapt prompt syntax, coordinate convention, and output parsing to match each model’s native interface.

Figure S3 : A simplified schematic of the pseudo-label generation pipeline for the scene-text subset of TextAnchor-26M. Raw image–text pairs are first processed by two OCR engines (PPOCR-V5 and docTR). Bounding boxes are retained only when the two predictions form a mutually best match with IoU $\geq 0.7$ . For retained candidates, transcripts are further verified by exact text matching. Samples passing both checks are accepted into the agreed subset, while a subset of remaining transcript disagreements is sent to Qwen2.5-VL-72B for final adjudication.

Component	#Instances	Ratio	Usage
Scene text	14.4M	54.0%	Stage-1
Synthetic text	10.8M	40.4%	Stage-1
Academic document pages	1.53M	5.7%	Stage-1
VQA with causal mask	10K	—	Stage-2 only
Total	26.7M	100%

Table S3 : Composition of the TextAnchor-26M training data. The first three components are used in first-stage training, while the VQA with causal mask subset is used only in second-stage training.

Figure S4 : Examples of de-stylized mask rendering. For each pair, we show the original cropped text region and the corresponding rendered mask constructed only from the transcript and bounding box. The resulting mask preserves coarse spatial layout while removing source-specific style, color, texture, and background appearance.

Category	Number	Datasets
Text Recognition	3.2M	CASIA-HWDB2 Liu et al. (2013) , K12_Printing Wiedmann et al. (2025) , ORAND-CAR-2014 Diem et al. (2014) , rendered_text Wendler (2023) , IAM Marti and Bunke (2002) , IIIT5K Mishra et al. (2012) , Imgur5K Krishnan et al. (2021) , VisualMRC Tanaka et al. (2021) , SROIE Huang et al. (2019) , multiple public CAPTCHA datasets, COCO-Text Veit et al. (2016)
Charts / tables	3.9M	ChartQA Masry et al. (2022) , Chart2Text Kantharaj et al. (2022) , CoSyn-400K Yang et al. (2025) , HiTab Cheng et al. (2022) , InfographicVQA Mathew et al. (2022) , LRV-Chart, SciTSR Chi et al. (2019) , SimChart9K Xia et al. (2023) , TabMWP Lu et al. (2022) , VisText Tang et al. (2023) , UniChart Masry et al. (2023) , SynthChartNet Nassar et al. (2025) ; Lysak et al. (2023) , ArxivQA Li et al. (2024a) , DVQA Kafle et al. (2018) , FigureQA Kahou et al. (2017) , FinQA Chen et al. (2021b) , FUNSD Jaume et al. (2019) , MMC Liu et al. (2023) , PlotQA Methani et al. (2020) , RoBUT Zhao et al. (2023) , TAT-QA Zhu et al. (2021) , VQAonBD Baviskar et al. (2021) ; Biten et al. (2019) ; Tito et al. (2021) ; Zheng et al. (2021) , Block-Diagram-Datasets Bhushan and Lee (2022)
Documents	5.7M	DocVQA Mathew et al. (2021) , DocReason25K Hu et al. (2024b) , Sujet Sujet AI (2024) , PDF-VQA Ding et al. (2023) , TAT-DQA Zhu et al. (2022) , DocDownstream Hu et al. (2025) , DocStruct4M Hu et al. (2024b) , Docmatix Hu et al. (2024a) , UReader Ye et al. (2023)
Formula recognition	2.4M	UniMER-1M Wang et al. (2024a) , SynthFormulaNet Nassar et al. (2025) ; Gervais et al. (2024) , HME100K Yuan et al. (2022) , latex_handwritten Wiedmann et al. (2025) , LatexFormulas OleehyO (2023) , Chrome_Writing Wendler (2023)
General OCR VQA	7.1M	EST-VQA Wang et al. (2020) , MTVQA Tang et al. (2025) , ST-VQA Biten et al. (2019) , TextVQA_train Singh et al. (2019) , WebSRC Chen et al. (2021a) , LSVT Sun et al. (2019) , ReCTS Zhang et al. (2019) , RCTW Shi et al. (2017) , ArT Chng et al. (2019) , COCO-Text Veit et al. (2016) , EATEN Guo et al. (2019) , invoices_receipts Cruz and Castelli (2022)
Stage-1 resampled subset	1.0M	A resampled subset drawn from the scene-text, synthetic-text, and academic-document components of TextAnchor-26M

Table S4: Composition of the Stage-2 training corpus.

Panel of Figure 19 — Figure S5: Qualitative comparison of Text-to-Region (T2R) results. Columns correspond to GPT-5.2, Qwen3-VL-235B-A22B-Instruct (Qwen3-VL), Qwen2.5-VL-3B (Baseline), and Q-Mask-3B (Ours). Each row shows the predicted location for a given text query.

Panel of Figure 20 — Figure S6: Representative R2T cases. Q-Mask reads the target text more reliably in small, low-contrast, or densely surrounded regions. In all examples, the queried box is fixed, and the comparison highlights recognition robustness under identical spatial constraints.

Panel of Figure 21 — Figure S7: Qualitative comparison on diverse VQA tasks. We evaluate Q-Mask-3B(Ours) against GPT-5.2, Qwen3-VL-235B-A22B-Instruct(Qwen3-VL), and Qwen2.5-VL-3B(baseline). Labels (a)–(f) correspond to different test cases.

Panel of Figure 22 — Figure S8: Visualization of Q-Mask-3B on representative VQA examples. For each example, we show (a) Input image, (b) Heatmap with Q-Mask-3B, (c) Label Mask.

Explore similar work

Feb 26, 2026

Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models

Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.

Jonathan Steinberg, Oren Gal

Swarms & AI Lab (SAIL), University of Haifa

Vision-Language Models VLM Interpretability

Ask PDF

Feb 25, 2026

SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read

Despite the rapid advancements in Multimodal Large Language Models (MLLMs), a critical question regarding their visual grounding mechanism remains unanswered: do these models genuinely

read'' text embedded in images, or do they merely rely on parametric shortcuts in the text prompt? In this work, we diagnose this issue by introducing the Visualized-Question (VQ) setting, where text queries are rendered directly onto images to structurally mandate visual engagement. Our diagnostic experiments on Qwen2.5-VL reveal a startling capability-utilization gap: despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting, exposing a deep-seated

modality laziness.'' To bridge this gap, we propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process. By transforming training samples into the VQ format with randomized styles, SimpleOCR effectively invalidates text-based shortcuts, compelling the model to activate and optimize its visual text extraction pathways. Empirically, SimpleOCR yields robust gains without architectural modifications. On four representative OOD benchmarks, it surpasses the base model by 5.4% and GRPO based on original images by 2.7%, while exhibiting extreme data efficiency, achieving superior performance with 30x fewer samples (8.5K) than recent RL-based methods. Furthermore, its plug-and-play nature allows seamless integration with advanced RL strategies like NoisyRollout to yield complementary improvements. Code is available at https://github.com/aiming-lab/SimpleOCR.

Yibo Peng, Peng Xia, Ding Zhong +6

UNC-Chapel Hill · Carnegie Mellon University · University of Michigan +1

Multimodal Large Language Models Optical Character Recognition Visual Question Answering

Ask PDF

Sep 12, 2026

Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End

Text-centric Visual Question Answering (VQA) requires reading and reasoning over text embedded in images, a task made substantially harder when images suffer from real-world degradation such as motion blur, low resolution, or compression artifacts. While modular OCR-based pipelines and end-to-end vision-language models are both widely used for this task, their comparative robustness under degraded conditions remains underexplored. We present an empirical study comparing two modular pipelines with SA-DBNet, a custom detector architecture combining ResNet-18 with self-attention spatial modeling and deformable convolutions against an end-to-end vision-language baseline, evaluated on 4013 degraded images with 7000 question-answer pairs. Fine-tuned modular pipelines achieve up to 57.50% exact-match accuracy versus 38.00% for the end-to-end baseline, with domain-specific fine-tuning yielding a gain of up to 29.50 percentage points. Critically, we find that conventional OCR error metrics like Character Error Rate and Word Error Rate are unreliable predictors of downstream VQA performance, as semantic reasoning can compensate for recognition failures when contextual cues are present. These findings highlight the importance of task-aware evaluation for text-centric VQA systems under realistic visual conditions. Codes are available here

Ritali Vatsi, Rachapudi Jagadeesh, Shruti Singh Baghel +3

Indian Institute of Technology Mandi Mandi, India · DIT University Dehradun, India · Indian Institute of Technology Kharagpur Kharagpur, India

Visual Question Answering VLM Robustness VLM Evaluation Optical Character Recognition

Ask PDF