In recent years, Multi-modal Large Language Models (MLLMs) have achieved strong performance in OCR-centric Visual Question Answering (VQA) tasks, illustrating their capability to process heterogeneous data and exhibit adaptability across varied contexts. However, these MLLMs rely on a Large Language Model (LLM) as the decoder, which is primarily designed for linguistic processing, and thus inherently lacks the positional reasoning required for precise visual tasks, such as text spotting and text grounding. Additionally, the extensive parameters of MLLMs necessitate substantial computational resources and large-scale data for effective training. Conversely, text spotting specialists achieve state-of-the-art coordinate predictions but lack semantic reasoning capabilities. This dichotomy motivates our key research question: Can we synergize the efficiency of specialists with the contextual power of LLMs to create a positionally-accurate MLLM? To overcome these challenges, we introduce PositionOCR, a parameter-efficient hybrid architecture that seamlessly integrates a text spotting model's positional strengths with an LLM's contextual reasoning. Comprising 131M trainable parameters, this framework demonstrates outstanding multi-modal processing capabilities, particularly excelling in tasks such as text grounding and text spotting, consistently surpassing traditional MLLMs.
Figures & tables
Figure 1: (a) Represents the specialist model for text spotting, which consists of an image encoder and decoder. The model can only output the text within the entire image along with its corresponding positions. (b) Represents mainstream MLLMs, where the image encoder extracts visual features and the LLM completes various multi-modal tasks. (c) Our proposed method, PositionOCR, employs a text spotting model that is guided by the LLM to accomplish various multi-modal tasks.
Figure 2: Overall framework of PositionOCR. In the first stage, a specialist model is developed, proficient in performing detection and recognition tasks. In the second stage, a Large Language Model (LLM) is introduced to achieve alignment between the two components using data from text spotting, followed by instruction tuning to enable interaction through human language.
Task
Samples
Datasets
Document
1344.5k
IIT-CDIP Lewis et al. (2006) , DocVQA Mathew et al. (2021) , InfoVQA Mathew et al. (2022)
Chart
18.3k
ChartQA Masry et al. (2022)
Scene text
387.1k
Total-Text Ch’ng and Chan (2017) , TextOCR Singh et al. (2021) , ICDAR13 Karatzas et al. (2013) , ICDAR15 Karatzas et al. (2015)
ICDAR2017 MLT Nayef et al. (2017) , HierText Long et al. (2022) , OpenVINO Krylov et al. (2021)
SCUT-CTW1500 Yuliang et al. (2017) , Curve Synthetic Dataset Liu et al. (2020)
Obj. detection
334.3k
COCO Lin et al. (2015) , Objects365 Shao et al. (2019)
Table 1: Details of datasets used in specialist model training.
Domain
Samples
Datasets
Document
4758.8k
DocGenome Xia et al. (2024) , DocVQA Mathew et al. (2021) , InfoVQA Mathew et al. (2022) , KLC Stanisławek et al. (2021)
VisualMRC Tanaka et al. (2021) , DocReason25k Hu et al. (2024) , Sujet-Finance AI (2025) ,
POIE Kuang et al. (2023) , SROIE Huang et al. (2019) , FUNSD Jaume et al. (2019) , TextCaps Sidorov et al. (2020) ,
SynthDoG-en Kim et al. (2022) , Docmatix Laurençon et al. (2024) , OCR-VQA Mishra et al. (2019)
Table
270.9k
WikiTableQuestions Pasupat and Liang (2015) , TabMWP Lu et al. (2023) , DeepForm Svetlichnaya (2020) ,
TableFact Chen et al. (2019) ,TableBench Wu et al. (2024) , TableVQA Kim et al. (2024)
Table 2: Details of datasets used in instruction tuning.
Figure 3: PositionOCR’s visualization results on text granularity include ‘Word”, “Phrase”, “Line”, and “Block”.
Methods
TP
Text Grounding
Block
Line
Phrase
Word
All
Qwen2.5-VL Bai et al. (2025)
7B
0.3
32.2
6.2
20.6
15.0
DocOwl-1.5 Liu et al. (2024c)
8.1B
91.3
85.9
76.4
70.4
80.4
DOGE Zhou et al. (2024)
8B
95.4
84.1
78.4
74.7
82.6
PositionOCR
131M
76.5
89.1
81.9
84.0
83.0
Table 3: The performance of different methods on the text localization evaluation set DocLocal4K. “TP” refers to Trainable Parameters. The [email protected] is used to evaluate the text grounding performance
Methods
Trainable Params
Accuracy @ IoU=0.5
Qwen2.5-VL Bai et al. (2025)
7B
79.92%
PositionOCR
131M
80.31%
Table 4: Performance comparison on the TextVQA-G dataset. The metric used is Accuracy @ IoU=0.5.
Methods
TP
Total-Text
ICDAR2015
None
Full
S
W
G
Qwen2.5-VL Bai et al. (2025)
7B
45.2
51.1
46.4
43.7
42.3
TextMonkey Liu et al. (2024c)
9.7B
61.4
-
-
-
45.1
PositionOCR
131M
66.6
77.4
74.1
71.1
67.7
Table 5: Text spotting results on Total-Text and ICDAR2015 in the VQA task. ‘None’ means lexicon-free. ‘Full’ indicates that we use all the words that appeared in the test set. ‘S’, ‘W’, and ‘G’ represent recognition with ‘Strong’, ‘Weak’, and ‘Generic’ lexicons, respectively. “TP” refers to the number of trainable parameters in the model.
Methods
TP
DocVQA
DeepForm
ChartQA
TabFact
SROIE
POIE
Dessurt Davis et al. (2022)
127M
63.2
-
-
-
-
-
Donut Kim et al. (2022)
176M
67.5
61.6
41.8
54.6
-
-
DocPeida Feng et al. (2023)
7.1B
47.1
-
46.9
-
21.4
39.9
DocOwl Ye et al. (2023a)
7.3B
62.2
42.6
57.4
67.6
1.7
2.5
UReader Ye et al. (2023b)
7.1B
65.4
49.5
59.3
67.6
-
-
QwenVL Bai et al. (2023)
9.6B
65.1
65.7
-
-
-
Table 6: Comparison with MLLMs and OCR-free methods on various types of image understanding tasks. All evaluation benchmarks use the officially designated metrics. “TP” refers to the number of trainable parameters in the model.
Methods
Refclef
Refcoco
Refcoco+
Refcocog
Test
Val
TestA
TestB
Val
TestA
TestB
Val
Test
Val
CITE Plummer et al. (2018)
34.13
-
-
-
-
-
-
-
-
-
MMI Mao et al. (2016)
-
-
64.9
54.5
-
54.0
42.8
-
-
-
Speaker Yu et al. (2017)
-
-
67.6
55.1
-
55.8
43.4
-
-
-
PFOS Sun et al. (2022)
67.9
-
81.4
73.1
78.4
72.4
55.2
65.8
67.6
67.8
OFA-L* Wang et al. (2022)
-
-
83.6
76.3
79.9
76.0
61.7
68.2
67.5
67.5
Table 7: Results of object grounding on Refclef, Refcoco, Refcoco+, and Refcocog.
Methods
Total-Text
ICDAR2015
None
Full
S
W
G
TOSS Tang et al. (2022)
65.1
74.8
65.9
59.6
52.4
SPTS Peng et al. (2022)
74.2
82.4
77.5
70.2
65.8
SPTS-v2 Liu et al. (2023)
75.5
84.0
82.3
77.7
72.6
InstructOCR Duan et al. (2024b)
77.1
84.1
82.5
77.1
72.1
PositionOCR
76.6
83.4
80.5
78.2
74.5
Table 8: The text spotting results of the specialist model on the Total-Text and ICDAR2015 datasets.
Despite the rapid advancements in Multimodal Large Language Models (MLLMs), a critical question regarding their visual grounding mechanism remains unanswered: do these models genuinely read'' text embedded in images, or do they merely rely on parametric shortcuts in the text prompt? In this work, we diagnose this issue by introducing the Visualized-Question (VQ) setting, where text queries are rendered directly onto images to structurally mandate visual engagement. Our diagnostic experiments on Qwen2.5-VL reveal a startling capability-utilization gap: despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting, exposing a deep-seated modality laziness.'' To bridge this gap, we propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process. By transforming training samples into the VQ format with randomized styles, SimpleOCR effectively invalidates text-based shortcuts, compelling the model to activate and optimize its visual text extraction pathways. Empirically, SimpleOCR yields robust gains without architectural modifications. On four representative OOD benchmarks, it surpasses the base model by 5.4% and GRPO based on original images by 2.7%, while exhibiting extreme data efficiency, achieving superior performance with 30x fewer samples (8.5K) than recent RL-based methods. Furthermore, its plug-and-play nature allows seamless integration with advanced RL strategies like NoisyRollout to yield complementary improvements. Code is available at https://github.com/aiming-lab/SimpleOCR.
Yibo Peng, Peng Xia, Ding Zhong +6
UNC-Chapel Hill · Carnegie Mellon University · University of Michigan +1
Vision-Language Models (VLMs) have achieved impressive results on general vision-language tasks, yet they suffer from hallucination, imprecise localization, and prohibitive computational cost when applied to dedicated OCR scenarios. This paper presents PP-OCRv6, a lightweight OCR system that combines architectural innovation with data-centric optimization. PP-OCRv6 redesigns the backbone, detection neck, and recognition neck around a unified MetaFormer-style building block with structural reparameterization, decoupling spatial token mixing from channel mixing and supporting both tasks through task-specific stride configurations. Three model tiers (medium, small, tiny) share the same block primitives, covering deployment scenarios from server to edge. On our in-house benchmarks, PP-OCRv6_medium achieves 83.2% recognition accuracy and 86.2% detection Hmean, outperforming PP-OCRv5_server by +5.1% and +4.6% respectively while surpassing Qwen3-VL-235B, GPT-5.5, and Gemini-3.1-Pro with orders of magnitude fewer parameters. The tiny tier achieves 3.9× faster inference than PP-OCRv5_mobile on Intel Xeon CPU while maintaining comparable accuracy.
Scene text spotting requires high-precision alignment between textual recognition and spatial localization. While visual-token grounding has emerged as a promising formulation for Multimodal Large Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances. To address this, we propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token and then recovers geometry via full-image refinement. To accurately identify this anchor without oracle labels, we introduce Single-Patch Selective Optimization (SPaSO), a reinforcement learning framework that optimizes discrete visual-token selection using patch-level rewards. To further improve representation robustness and localization precision, we introduce Directional Embedding Alignment (DEA) to suppress unstable norm bias by decoupling feature magnitude and direction, and Patch-Enhanced Decoding (PED) to fuse the routed anchor with language semantics and cross-attend over the full-image feature map for geometry-aware boundary regression beyond coordinate-space surrogates. Extensive experiments demonstrate that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs. Code is available at https://github.com/eeNickTang/SPaTS.
Rui Tang, Wentao Yang, Peirong Zhang +4
South China University of Technology · Guangzhou, China · HiThink Research +1