While modern Vision-Language Models (VLMs) have been remarkably successful in general scene text recognition, they consistently struggle with the long-tail distribution of text that is degraded by motion blur or low resolution. These limitations are particularly apparent in egocentric applications like smart glasses, where text is often captured in noisy environments. In this paper, we introduce the BSTRB (Blurry Scene Text Recognition Benchmark), a novel benchmark of 8,398 noisy image samples with high-fidelity annotations derived from 1,699 real-world video sequences. To support this, we present Hawkeye, a VLM fine-tuned on BSTRB to read and reason about illegible text by leveraging contextual priors and pixel-level alignment. Our results show that \name matches much larger VLMs on BSTRB overall and exceeds them on text at or below 20 pixels, in a single pass suited to real-time deployment. We characterize the resolution at which current VLMs begin to fail.
Figures & tables
Figure 1: Each video filmer walks backwards from the object, so the segmented object from the scene yields the same text from legible to illegible (median text height 28.0 px to 10.9 px). Ground truth is annotated on the initial reference frames and propagated to later frames.
Figure 2: BSTRB composition and text density. Panel (a) is over the clips carrying each label (1,449 object/text, 1,559 scene) of 1,699 collected; see Tab. 1 . BSTRB is denser than every benchmark except HierText and RIMES.
Figure 3: Transcription benchmark results, BoW F1. Columns: 20 API models (by mean), Qwen3-VL bases, and Hawkeye. Rows: 10 OCR datasets and held-out BSTRB , grouped by ground-truth length, with n and mean ground-truth words. Boxed: best per row. GPT-6 Astra is best on 4 of 12 columns; Hawkeye (Qwen3-VL-8B) is best on BSTRB -vid.
Stage
Unit
Video
Photo
Collected
clips / photos
1,699
944 + 511
Retained after filtering
clips
1,413
—
Train (automatic labels)
frames (videos)
6,734 (1,133)
700
Validation (verified)
frames (videos)
333 (56)
36
Held-out test (verified)
frames (videos)
1,331 (224)
241
Table 1: BSTRB counts at each pipeline stage. Within a surviving clip, each tracked object is sampled at 6 intervals ( Sec. 4 ).
Figure 4: Hawkeye 4B and 8B − their Qwen3-VL bases: paired differences in BoW F1 with 95% CIs, clustered by video [ 27 ] ; filled: significant after Holm. Right of each panel: the difference (bold if significant) and base → Hawkeye BoW F1.
Figure 5: Dedicated OCR engines compared to VLMs on BSTRB . Google Vision is competitive only on the sharp reference frames.
Figure 6: SoTA proprietary VLMs for OCR on Snellen charts. Performance drops at 5px.
Figure 7: BoW F1 on held-out BSTRB video frames (1,331 frames, 224 videos) by mean text height, split at 20 px with a separate y-scale on each side. Left: Hawkeye (solid) vs. its base (dashed, same colour). Right: Hawkeye vs. the strongest API VLMs. Equal-count bins ( n=98 – 111 ), bootstrap 95% CI bands.
Figure 8: Study pages shown to Prolific participants, in survey order; identifying details (study account, e-mail address, upload links) are redacted. Video study ( BSTRB ): (a) instructions with the demonstration video, (b) reminders and steps, (c) upload page. Photo pilot: (d) instructions; (e) a correct and (f) an incorrect example pair, each with its explanation; (g) privacy acknowledgement; (h) upload page; (i) reference pair and reminder after the quiz; (j) example photo gallery; (k–m) the three quiz pairs, each answered yes or no.
Category
Dataset
Primary Task
Train
Test
Scene Text
HierText [ 24 ]
Hierarchical Text
–
1,000
Total-Text [ 7 ]
Curved Text
–
255
COCO-Text [ 38 ]
Scene Text Detection
–
500
ICDAR [ 16 ]
In-the-wild Text
–
200
TextZoom [ 39 ]
Super-Resolution
–
67
Occluded [ 42 ]
Occlusion Robustness
–
22
Table 2: Training and evaluation data. Hawkeye is fine-tuned on BSTRB video frames only; BSTRB photos and the public sets were tried on validation and excluded ( Sec. 10 ), so every other row is evaluation-only. BSTRB test rows are the held-out, human-verified split; a further 369 validation images (333 frames from 56 videos, 36 photos; 372 prompt rows) were used only to select the recipe and checkpoints.
res_video
res_photo
Units annotated
282
277
accepted unchanged
222
119
edited
58
158
flagged (excluded)
2
0
Units in benchmark
280
277
Evaluation items
1664
277
Table 3: Composition of the human-verified BSTRB test set. A res_video unit is one video, annotated once for the whole video and applied to its 5–6 sampled frames; a res_photo unit is one image. Every unit carries two independent annotations: A1 from the automatic GPT-5 annotator and A2 from the human expert annotator, who worked from the frames without editing A1’s output in place.
Exact agr. (%)
CER mean
CER med.
WER mean
WER med.
Agr. mean
res_video
80.0
0.058
0.000
0.065
0.000
0.961
res_photo
43.7
0.481
0.333
0.491
0.393
0.583
Table 4: Inter-annotator agreement on BSTRB between A1, the automatic GPT-5 annotator, and A2, the human expert annotator. Agreement is computed after the normalisation used for all BSTRB scoring. CER and WER are directional, taking A1 as hypothesis and A2 as reference, and are clipped at 1.0 so that units whose A1 label is near-empty cannot dominate the mean; Agr. is the symmetric normalised edit similarity 1−d(A1,A2)/max(∣A1∣,∣A2∣) , which does not depend on which annotator is treated as the reference.
res_video
res_photo
Median A1 transcript (chars)
242
102
Median A2 transcript (chars)
246
342
Units with A1 < 30 chars
13
73
Units where A2 >3× A1
1
101
Units where A2 <31 A1
0
0
Table 5: Transcript length under A1 (automatic) against A2 (human). Disagreement is near-exclusively additive : no unit in either split was shortened by more than a factor of three under A2, while 101 of 277 res_photo units more than tripled in length. A1 under-transcribed dense and non-Latin text rather than hallucinating text that was not present, so the disagreement is dominated by omission rather than by substitution.
Figure 9: BSTRB degradation characteristics over 8,889 per-frame records. Laplacian variance rises only because it is scale-dependent.
Figure 10: 40 of the 211 held-out test clips at t=0.15 , one frame per object. Note the variance in text density and style.
Figure 11: Transcription track, word inclusion rate; layout as in Fig. 3 . GPT-6 Astra is best on 5 of 12 columns; Hawkeye (Qwen3-VL-8B) is best on BSTRB -vid. The best model per column is the same as under BoW F1 on 8 of 12 columns.
Figure 12: Fig. 3 recomputed with output formatting normalized (markdown, preamble and bullets stripped identically for every model). No API request failed, so no sample is dropped. Normalization changes the score of 3–5% of each model’s outputs and moves no model’s mean by more than 0.07pp; the best model per column is unchanged on 12 of 12 columns.
Figure 13: Short-answer QA track scored with answer containment, for the preliminary 4B model ( Sec. 5 ) and API models; it was not run for the final models.
Figure 14: 37 objects captured with smart glasses outside the BSTRB distribution (screens, signage, documents, outdoors, lab, Chinese characters): Hawkeye 8B against its base Qwen3-VL-8B.
Figure 15: Error composition and what fine-tuning costs, for Hawkeye at each size and its Qwen3-VL base.
Figure 16: How each model fails, on held-out BSTRB video frames at ≤ 20 px (filled) and the 10 general OCR sets (open), 95% CIs clustered by video. Left: outputs that ran to the token cap (repetition loops). Middle: empty outputs. Right: over the remaining outputs, the share of words absent from the reference, an upper bound on hallucination. At ≤ 20 px, Hawkeye 4B loops on 7.2% of frames against 14.3% for its base, and Hawkeye 8B on 5.2% against 8.5%. Once loops are removed, Hawkeye’s unsupported-word share differs from its base’s by 4B +0.1, 8B +3.3 pp (intervals overlap at every size). API models rarely loop but return nothing instead, up to 14.5% of frames (GPT-5).
BSTRB video ≤ 20 px
BSTRB video > 20 px
BSTRB photo
General OCR (10 sets)
Uns. %
Ins.
Loop %
Uns. %
Ins.
Loop %
Uns. %
Ins.
Loop %
Uns. %
Ins.
Loop %
Qwen3-VL-4B
26.5
0.10
14.3
13.9
0.13
5.9
29.9
0.15
11.6
30.1
0.37
17.1
Hawkeye 4B
26.6
0.10
7.2
12.6
0.11
2.2
26.2
0.18
9.1
26.0
0.43
8.0
Qwen3-VL-8B
21.4
0.07
8.5
10.7
0.11
4.1
20.1
0.12
9.5
25.4
0.39
11.1
Hawkeye 8B
24.7
0.10
5.2
10.8
0.11
1.9
24.3
0.23
5.0
27.0
0.53
6.0
GPT-5
6.3
0.04
0.0
5.9
0.04
0.0
17.7
0.04
0.0
21.9
0.16
0.0
Table 6: Hallucination and output failures. Loop: outputs that ran to the token cap. Over the remaining non-empty outputs, Uns.: mean share of output words absent from the reference (an upper bound on hallucination), and Ins.: mean Levenshtein insertions per reference word. Hawkeye and its base at each size, and the three API models with the fewest unsupported words on BSTRB video ≤ 20 px. All models: Fig. 16 .
Figure 17: Reading trace on a web page photographed with smart glasses: peak attention patch per generated token, in decoding order (sinks excluded); green regions hold a peak, pink regions none. Base Qwen3-VL-4B transcribes one post title; Hawkeye 4B reads the sidebar and the post (BoW F1 .12 vs .84).
Figure 18: Attention of Hawkeye and its Qwen3-VL base at 4B (160 frames) and 8B (159 frames) on held-out BSTRB video frames, same input for both arms (bicubic × 3, 1 MP cap), last four layers’ attention to image patches averaged over generated tokens; bars are image means (light: base, dark: Hawkeye), whiskers the paired 95% bootstrap CI of the difference. Content-free “sink” patches (mean attention > 50 × the median patch) are reported separately and excluded from the other measures. Hawkeye − base, 4B / 8B: sink mass -0.032 ( dz =-0.33) / -0.042 ( dz =-0.5); entropy -0.014 ( dz =-0.29) / +0.009 ( dz =0.23); top-1% mass +0.020 ( dz =0.28) / -0.009 ( dz =-0.14, n.s.).
Figure 19: Reading traces: the attention peak of each generated token (sink patches excluded), darker for later tokens, for base and Hawkeye at 4B and 8B on the two analysed frames with the largest Hawkeye 4B gain and loss in BoW F1 (selection rule fixed in advance; the same frames are shown for 8B).
Figure 20: Reading traces of base Qwen3-VL-8B and Hawkeye 8B on two smart-glasses captures, same conventions as Fig. 17 (4B on the Reddit page). The base model lists the section headings of the Amazon page (5 of 35 text regions) and one post title of the Reddit page (1 of 47); Hawkeye reads the tile labels (16 regions) and the sidebar and post (20 regions). Hand-picked among captures where Hawkeye’s transcription beats its base’s at both sizes; over all captures the share of text regions visited does not differ ( Sec. 16 ). Regions: detector and OCR boxes merged into lines, plus tile labels added by hand.
Figure 21: Attention at individual decoding steps on the Reddit capture of Figs. 17 and 20 . Each panel is the attention map (last four layers, mean over heads, sinks excluded; brighter = more) at the step that emits the word in its title, zoomed to the word; the cross marks the peak and the box the word as located by OCR (green: peak on the word, pink: not). Words are content words in the model’s own output, spread over decoding and preferring those located by OCR. The bases spend their few steps on a preamble and one title, with the peak often beside the word; Hawkeye’s peak tracks the word it is emitting through the post body.
Figure 22: BSTRB examples, with median text height. Note, these examples in the paper are downsampled and not full-resolution.
Figure 23: Transcriptions comparing the base Qwen3-VL-4B and the finetuned Hawkeye on the same frame. Green words appear in the ground truth, red do not; transcripts are truncated to four lines. The last row is an example of a failure case in Hawkeye .
Figure 24: Illustrative held-out BSTRB frames (hand-picked among frames where Hawkeye beats its base at both 4B and 8B). Mean attention over the generated tokens (last four layers, mean over heads, attention sinks excluded; brighter = more attention), each map scaled to its own maximum; F1 is BoW F1 of the scored output. On the cereal box and the book cover the bases stay on the large print, while Hawkeye also attends to the small print it then transcribes (the lower panel text; the “Beginner Books” badge). On the back cover Hawkeye spreads over the paragraph and reads it, while base Qwen3-VL-4B falls into a repetition loop and base Qwen3-VL-8B answers that no text is visible ( Fig. 25 ).
Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45% to 58.51%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.
Yuxing Cheng, Yuan Wu, Yi Chang
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Vision-language models (VLMs), despite their success in optical character recognition (OCR) tasks, are vulnerable to typographic attacks and have a fragile structure for images with multiple text layers. In this study, the DecoyBench dataset was created using the Decoy Font method. The dataset consists of 300 images, each containing text with sharp contour lines superimposed on another text with soft shading. Six recent closed-source models from three different model families were evaluated using this dataset under two different prompting conditions (naive and guided) and at two different resolutions (512×512 and 64×64). A validation study showed that human participants could read both text layers with high accuracy. In contrast, the models, with most variants and both prompting methods, read the contour text with near-human accuracy at high resolution, but almost never fully extracted the shading text. At low resolution, the contour text could not be read by either the models or humans, while the shading text could be extracted with high accuracy. The findings indicate that the evaluated VLMs exhibit a consistent behavioral limitation when processing typographic structures containing multiple spatial frequency layers.
Mert İncidelen, Yamen Kashkash, Asya Berker +1
Department of Artificial Intelligence and Data Engineering, Fırat University, Elazığ, Türkiye · Department of Software Engineering, Fırat University, Elazığ, Türkiye
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.