Multimodal Large Language Models (MLLMs) have recently made strong progress in visual--linguistic understanding. However, their performance on text-centric video reasoning remains highly sensitive to input quality. Real-world user-provided videos often contain motion blur, compression artifacts, noise, and low-resolution text, which impair reliable text reading and downstream reasoning. Whether MLLMs can robustly read and reason about real-world scene text under diverse quality conditions remains a fundamental open question. We introduce ClearText-Video (CTVid), a large-scale, scene-text-aware benchmark for studying text-centric video understanding under controlled quality variation. CTVid contains 4,639 real-world text-rich egocentric videos, 550K+ frames, 1.6M human-verified scene-text annotations, and 220K+ spatial/temporal question--answer pairs in Chinese and English. For each high-quality video, CTVid provides content-matched Degraded-Quality and Restored-Quality variants, supporting two task families: Text-Centric Video Restoration and Multi-Quality VideoQA. We evaluate 18 representative restoration methods and 16 state-of-the-art MLLMs on CTVid. The results show that visual enhancement does not guarantee textual fidelity or downstream reasoning gains: blur is more damaging than low resolution, restored videos can alter the textual evidence used by MLLMs, and OCR-only pipelines remain far below direct multimodal reasoning. CTVid exposes the gap between video restoration and text-grounded understanding, providing a rigorous foundation for restoration-aware, quality-robust text-centric video systems.
Figures & tables
Figure 1: Overview of the ClearText-Video (CTVid) Benchmark . CTVid is built from bilingual (Zh/En) egocentric videos with scene-text boxes, transcripts, captions, and spatial/temporal QA pairs. Its key design is a quality-controlled video triplet: each content instance is evaluated as High-Quality (HQ), Degraded-Quality (DQ), and Restored-Quality (RQ), allowing direct analysis of how degradation and restoration change the textual evidence available to MLLMs. Unlike restoration-only datasets or single-quality TextVQA benchmarks, CTVid links Text-Centric Video Restoration (image SR, video SR, and deblurring) with Multi-Quality VideoQA, testing whether models can preserve, recover, and reason over the same scene text across changing video quality.
Task
Dataset
Venue
GT Resolution
Caption
Text in Video
Content Language
Videos/Img
Video Source
Video Super-Resolution
VideoLQ [ 6 ]
CVPR 2022
640 × 480
✗
✗
✗
0.05K/4.9K
Online
SPMCS [ 47 ]
ICCV 2017
1920 × 1080
✗
✗
✗
0.03K/0.93K
Manual
Vimeo-90k [ 62 ]
IJCV 2019
448 × 256
✗
✗
✗
4.28K/0.6M
Online
RealVSR [ 65 ]
ICCV 2021
1024 × 512
✗
✗
✗
0.5K/2.5K
Manual
YouHQ [ 75 ]
CVPR 2024
1920 × 1080
✗
✗
✗
37K/-
Online
MVSR4 × [ 51 ]
CVPRW 2023
1920 × 1080
✗
✗
✗
0.2K/2K
Manual
Table 1: Comparison of ClearText-Video with representative video datasets. “Text in Video” indicates whether textual content appears in video frames, and “Content Language” reports the languages of embedded text.
Figure 2: Example videos and their annotated questions from the ClearText-Video benchmark. Note: [BBOX] denotes bounding-box coordinates, visualized as red detection boxes in the images.
Figure 3: Overview of the annotation pipeline.
Input Type
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
DISTS ↓
CLIPIQA ↑
NIQE ↓
MUSIQ ↑
MANIQA ↑
FID ↓
P det ↑
R det ↑
F1 det ↑
Acc rec ↑
NED rec ↓
Image-based
Real-ESRGAN [ 52 ]
27.85
0.8748
0.1638
0.1032
0.4790
4.877
64.65
0.5718
35.00
89.06
57.78
70.09
34.82
53.51
SwinIR [ 28 ]
28.18
0.8788
0.1613
0.1012
0.5029
4.877
65.73
0.5705
32.15
88.78
58.47
70.50
36.76
54.96
SeeSR [ 57 ]
27.89
0.8529
0.1824
0.1161
0.6471
5.121
69.92
0.6196
27.66
87.27
68.72
76.89
35.08
54.95
OSEDiff [ 56 ]
25.67
0.8289
0.1946
0.1151
0.6219
4.852
69.11
0.6168
32.67
87.59
66.35
75.50
34.25
54.06
S3Diff [ 68 ]
26.05
0.8033
0.1617
0.09611
0.5764
4.517
64.52
0.5799
29.16
86.72
69.52
77.17
38.42
58.03
AdcSR [ 7 ]
25.84
0.8239
0.2024
0.124
0.6448
4.941
69.82
0.6083
37.52
88.84
57.72
69.98
32.02
51.26
Table 2: Text-Centric Visual Super-Resolution Benchmark. We report both general perceptual metrics and text-aware metrics. Text-aware metrics include precision (P det% ), recall (R det% ), and H-mean (F1 det% ) for detection, and accuracy (Acc rec% ) and normalized edit distance (NED rec% ) for recognition. The best and second-best results for each metric are highlighted in red and blue , respectively.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
DISTS ↓
CLIPIQA ↑
NIQE ↓
MUSIQ ↑
MANIQA ↑
FID ↓
P det ↑
R det ↑
F1 det ↑
Acc rec ↑
NED rec ↓
MIMO-UNet+ [ 11 ]
29.26
0.8657
0.2053
0.1393
0.3324
6.483
46.02
0.4760
47.15
88.58
65.02
74.99
51.75
69.54
NAFNet [ 8 ]
28.21
0.8716
0.2461
0.1749
0.3487
7.614
48.35
0.4253
48.65
90.81
66.37
76.69
50.77
68.30
Restormer [ 67 ]
31.38
0.8863
0.1904
0.1349
0.3404
6.552
47.31
0.4864
41.50
88.28
67.08
76.23
54.99
71.67
Stripformer(GoPro) [ 49 ]
31.13
0.8832
0.1754
0.1191
0.3348
6.386
48.65
0.4955
38.78
88.42
66.69
76.03
55.05
72.26
Stripformer(RealBlur-J) [ 49 ]
27.04
0.8655
0.1837
0.1182
0.3657
6.240
54.44
0.4922
32.36
89.18
69.99
78.43
58.93
76.51
Stripformer(RealBlur-R) [ 49 ]
27.36
0.8667
0.1905
0.1166
0.3396
6.490
50.41
0.4668
33.46
89.45
69.71
78.36
58.57
75.71
Table 3: Text-Centric Video Deblurring Benchmark. We use the same metrics as the Super-Resolution Benchmark above. Best and second-best results are highlighted in red and blue , respectively.
Figure 4: Statistics of the CTVid dataset. The left panels summarize text carrier types and scene categories, while the right panels show word-frequency visualizations for video captions and frame-level text annotations.
Figure 5: Qualitative results on the Text-Centric Video Restoration benchmarks on the CTVid test set. Results are shown for both English and Chinese text instances. The examples include video super-resolution and deblurring outputs, highlighting differences in text fidelity and legibility across methods.
HQ
DQ-Low_res [ 34 ]
DQ-Blur [ 34 ]
RQ-DOVE [ 10 ]
RQ-MIMO [ 11 ]
RQ-S3DIFF [ 68 ]
Model
Acc ↑
OC ↓
UAcc ↑
Abs
Acc ↑
OC ↓
UAcc ↑
Abs
Acc ↑
OC ↓
UAcc ↑
Abs
Acc ↑
OC ↓
UAcc ↑
Abs
Acc ↑
OC ↓
UAcc ↑
Abs
Acc ↑
OC ↓
UAcc ↑
Abs
GPT-5.4 [ 38 ]
60.00
15.00
55.00
0.00
53.33
10.00
58.33
6.70
56.67
8.33
60.00
18.30
48.33
15.00
55.00
3.30
55.00
10.00
58.33
6.70
53.33
13.33
53.33
0.00
GPT-5.4-mini [ 37 ]
46.67
6.67
65.00
0.00
48.33
18.33
51.67
0.00
43.33
5.00
63.33
3.30
46.67
3.33
66.67
0.00
41.67
11.67
56.67
0.00
45.00
8.33
56.67
0.00
Claude-Sonnet-4.6 [ 2 ]
70.00
21.67
66.67
0.00
63.33
20.00
70.00
0.00
60.00
20.00
70.00
13.30
65.00
18.33
70.00
0.00
70.00
20.00
70.00
3.30
61.67
16.67
71.67
0.00
Gemini-2.5-pro [ 12 ]
71.67
16.67
65.00
0.00
65.00
8.33
71.67
1.70
60.00
13.33
66.67
23.30
65.00
13.33
68.33
1.70
63.33
20.00
56.67
10.00
70.00
11.67
65.00
0.00
Gemini-2.5-flash [ 12 ]
60.00
21.67
61.67
1.70
65.00
10.00
61.67
1.70
48.33
10.00
76.67
30.00
66.67
10.00
66.67
3.30
60.00
16.67
61.67
11.70
61.67
13.33
58.33
1.70
Table 4: Text-Centric VideoQA for Spatial Understanding. We evaluate accuracy (Acc), uncertainty-aware accuracy (UAcc), overconfidence ratio (OC), and abstention rate (Abs) across six input-quality conditions: HQ (High-Quality), DQ-Low_res [ 34 ] (low-resolution degradation), DQ-Blur [ 34 ] (blur degradation), RQ-DOVE [ 10 ] (video-based SR), RQ-MIMO [ 11 ] (blur restoration), and RQ-S3DIFF [ 68 ] (image-based SR). “Qwen2.5-VL-7B-SFT” denotes Qwen2.5-VL-7B instruction-tuned on the ClearText-Video training set. For Acc, OC, and UAcc, the best and second-best results in each column are highlighted in red and blue , respectively. All reported values are percentages (%).
Figure 6: Accuracy (Acc) and uncertainty-aware accuracy (UAcc) radar plots for multiple MLLMs across six video-quality conditions. The top plot reports Acc, and the bottom plot reports UAcc.
Accuracy%( ↑ )
Models
HQ
DQ-Blur [ 34 ]
RQ-MIMO [ 11 ]
RQ-S3DIFF [ 68 ]
RQ-DOVE [ 10 ]
Gemini-2.5-flash [ 12 ]
51.55
51.20
52.12
51.32
53.04
Gemini-2.5-pro [ 12 ]
45.83
50.00
51.39
50.00
48.61
GPT-4o-mini [ 36 ]
50.06
50.52
50.29
50.13
49.94
Claude-Sonnet-4.6 [ 2 ]
60.02
59.45
59.34
58.88
60.37
Qwen2.5-VL-7B [ 3 ]
50.86
50.06
50.40
50.29
50.63
Table 5: Text-Centric Video Question Answering for Temporal Understanding. We report accuracy across five video-quality conditions. The best and second-best results in each quality condition are highlighted in red and blue , respectively, across all models.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sunburst visualization of the ClearText-Video test set composition. From the center outward, the rings represent video source (offline vs. online), language (EN vs. CHN), and lighting condition (artificial vs. daylight). Offline videos constitute 74% of the test set, with a balanced distribution of English and Chinese clips and a mix of artificial and natural lighting. The online portion (26%) is evenly split between English and Chinese and is almost entirely captured under artificial lighting.
Dataset
#QA
#Boxes / Trans.
Spatial QA
Temporal QA
Bilingual/ Multilingual
Dense Frame-Text
Paired HQ/DQ/RQ
Rest. Eval.
TextVQA [ 44 ]
45,336
OCR tokens #boxes: NA
✗
✗
✗
✗
✗
✗
NewsVideoQA [ 19 ]
8,672
OCR tokens + subtitles
✗
✓
✗
✗
✗
✗
EgoTextVQA [ 76 ]
7,064
OCR stats only #boxes: NA
Partial
✓
✗
✗
✗
✗
ViTXT-GQA [ 77 ]
2,055
52,494 boxes 2,227 segments
✓
✓
✗
Partial
✗
✗
M4-ViteVQA [ 73 ]
25,123
OCR tokens avg. 56.92/video
Partial
✓
✗
✗
✗
✗
MME-VideoOCR [ 43 ]
2,000
NA
✓
✓
Partial
✗
✗
✗
Appendix
Table 6: Comparison with text-centric video benchmarks. “#Boxes / Trans.” denotes text bounding boxes, OCR tokens, or transcripts when explicitly provided. “Dense Frame-Text” indicates dense frame-level text annotations rather than sparse or question-grounded evidence. “Partial” indicates related but incomplete support.
Figure 8: Example videos and their annotated questions from the ClearText-Video benchmark. Note: [BBOX] denotes bounding-box coordinates, visualized as the red detection boxes in the images. The examples span transportation, commercial, and public-service scenes in both English and Chinese environments, illustrating the multilingual and text-centric diversity of our benchmark. This diversity enables the evaluation of model generalization across varied real-world video scenarios.
Figure 9: An example text instance showing its trajectory, speed, and acceleration over time.
Figure 10: Temporal variation of the bounding-box area for the same text instance.
Figure 11: Failure modes on RQ inputs. Restoration may hallucinate or distort text, while MLLMs may further fail through localization bias, reasoning errors, or temporal aggregation errors.
HQ
LR
Blur
DOVE
MIMO
S3Diff
Avg. Acc ↑
47.73
44.59
41.69
45.21
44.02
44.79
Avg. Δ HQ
–
-3.14
-6.05
-2.52
-3.72
-2.95
Qwen3-VL-8B Acc ↑
50.56
45.43
44.46
47.70
46.91
46.64
Qwen3-VL-8B Δ HQ
–
-5.13
-6.10
-2.86
-3.65
-3.92
Appendix
Table 7: Quality-type comparison on spatial QA. Avg. Acc is averaged over the 16 MLLMs in the main spatial VideoQA table.
Input quality
OCR+Qwen2.5-7B
Qwen2.5-VL-7B
Acc ↑
UAcc ↑
OC ↓
Acc ↑
UAcc ↑
OC ↓
HQ
23.3
66.0
14.6
51.21
58.69
10.89
DQ-Low_res
22.7
66.7
13.5
40.23
61.67
3.99
DQ-Blur
22.3
66.6
13.7
41.72
62.66
9.19
RQ-DOVE
22.9
64.5
17.7
46.58
59.97
10.79
RQ-MIMO
22.2
61.9
20.1
46.10
59.89
10.79
Appendix
Table 8: OCR+LLM vs. direct MLLM on spatial QA. OCR+Qwen2.5-7B uses OCR text only, without image/video input.
Figure 12: Result visualization on ClearText-Video for text-centric spatial VideoQA. This example shows a Chinese sign in which the question asks the model to select the option that correctly matches the text inside a specified bounding box. Most higher-performing models identify the correct phrase consistently across quality conditions, while weaker models confuse visually similar Chinese characters or repeatedly select an incorrect option. This example illustrates that even when the text is visually prominent, robust spatial text understanding still depends on precise character-level discrimination rather than coarse recognition.
Figure 13: Result visualization on ClearText-Video for text-centric spatial VideoQA. This example shows an outdoor transit sign in which the question asks whether a set of words, including “Food”, “Do Not”, “on bus”, and “Bring”, appears inside the specified bounding box. Many models correctly verify the presence of the target phrases across quality conditions, but some models consistently reject the correct answer or become unstable under degraded and restored inputs. This case illustrates that spatial VideoQA requires not only text recognition, but also accurate grounding of multiple words within the queried region.
Figure 14: Result visualization on ClearText-Video for text-centric spatial VideoQA. This example shows an English warning sign where the question requires completing the missing substring in “HIGH ____ TAGE”. Most proprietary MLLMs consistently recover the correct missing text across different quality conditions, while several open-source models either output the full word instead of the missing span, predict only a single character, or hallucinate visually plausible but incorrect fragments. The results highlight the difficulty of fine-grained text completion under blur, low resolution, and restoration artifacts.
Figure 15: Qualitative results on the Text-Centric Video Restoration benchmark on the CTVid test set. Results are shown for both English and Chinese text instances. The examples include video super-resolution and deblurring outputs, highlighting differences in text fidelity and legibility across methods.
Figure 16: Result visualization on ClearText-Video for text-centric temporal VideoQA. This example asks for the overall motion direction of the text “ ↓ 363-378Gates” across the video clip. Although the ground-truth motion is consistently leftward under all quality conditions, most models incorrectly predict that the text stays roughly in place, suggesting substantial reliance on static keyframe cues in this example. Only a few models predict the correct motion direction, while their predictions still become unstable under degraded or restored inputs, showing that robust temporal motion reasoning over scene text remains challenging for current MLLMs.
Figure 17: Result visualization on ClearText-Video for text-centric temporal VideoQA. This example asks where the text “NO SUPRISES!” appears for most of the clip. The ground-truth answer is consistently the bottom-left region, but many models confuse it with the bottom-right or center regions, and some predictions vary across different quality conditions. This example illustrates that reliably estimating the dominant spatial position of a text instance over time requires more than recognizing the text in a single frame, as models must aggregate its location across the full temporal sequence.
Figure 18: Result visualization on ClearText-Video for text-centric temporal VideoQA. This example asks about the relation between the text “PHARMACY” and the frame boundary. Although the correct answer is that the text touches or runs off an edge, most models incorrectly classify it as fully inside or merely close to an edge, especially under degraded and restored inputs. This case highlights the difficulty of boundary-aware temporal text reasoning, where models must precisely track the relation between a text instance and the frame boundary rather than relying on coarse recognition alone.
System Prompt
Fill-in Question Prompt
Your task is to answer fill-in questions based on image input. Input Format: An image with {size} and a fill-in question string, e.g., “C____T”. Output Format: A single-line JSON object: {"answer": "<only the missing characters>", "reasoning": "<a brief explanation, 1 sentence max>"} Rules: Only output the missing characters, not the full word. Keep the reasoning short and image-based. Do not infer from common sense or spelling patterns, because the answer may not follow normal spelling. Only answer based on what is actually visible in the image. The blank, represented as “____”, can correspond to any number or type of characters, including letters, digits, Chinese characters, or symbols. Example: Question: What letter is missing in “C__T”? Answer: {"answer": "a", "reasoning": "C-a-T forms CaT as shown in the image."} Respond only with a JSON object. Do not include anything else.
Multi-choice Question Prompt
You are a vision-language assistant specialized in text recognition. Your task is to answer multiple-choice questions based on image content. Input Format: An image with {size} and a question string with options, e.g., “Which letter is missing in G__FT? Choices: A. A B. B C. I D. H”. Output Format: A single-line JSON object: {"answer": "<only the correct letter(s)>", "reasoning": "<a brief explanation, 1 sentence max>"} Rules: Only output the letter(s) corresponding to the correct option. Do not repeat the full option. Keep the reasoning short and image-based. The answer can contain multiple letters, e.g., “A,C”, if multiple options are correct. Do not infer from common sense or spelling patterns. Only answer based on what is actually visible in the image. Example: Question: Which word(s) is located within the area […]? Choose the correct option. A. Apple B. App C. Able D. Aple Answer: {"answer": "A,B", "reasoning": "I can see both ‘Apple’ and ‘App’ in the bounding box area."} Respond only with a JSON object. Do not include anything else.
True-False Question Prompt
You are a vision-language assistant specialized in text recognition. Your task is to answer true/false questions based on the content of the image. Input Format: An image with {size} and a true/false question string, e.g., “The word in the image is DOG. True or False?”. Output Format: A single-line JSON object: {"answer": "True" or "False", "reasoning": "<a brief explanation, 1 sentence max>"} Rules: The answer must be strictly “True” or “False”. Keep the reasoning short and image-based. Do not infer from common sense or spelling patterns. Only answer based on what is actually visible in the image. Example: Question: The word in the image is DOG. True or False? Answer: {"answer": "False", "reasoning": "The image shows DoG, not DOG."} Respond only with a JSON object. Do not include anything else.
Appendix
Table 9: System prompt used for evaluating spatial video question answering.
System Prompt
You are a helpful assistant that can answer questions about a video. Video context: The original video has {total_frames} frames in total{fps_clause}. Original frame resolution: {original_size}. Frames are resized to {resized_size} before being shown to you. Any coordinates or bounding boxes mentioned in the questions are based on the original resolution {original_size}. You are shown {n_sampled} frames uniformly sampled from this video. The frames are given in chronological order. The k-th frame you see, 1-based, corresponds to original frame index sampled_indices[k-1], where sampled_indices = {sampled_indices_str}. When the question refers to a specific frame index or timestamp, reason about which of your sampled frames is closest to it. You only have access to the listed sampled frames. Answer vocabulary — geometric question types: Each question about a specific target text uses one of the fixed option sets below. Pick the option that best matches what you observe across the sampled frames. Position regions: “center (middle of the frame)”: the central ∼ 1/3 of the frame on both the horizontal and vertical axes; “top-left (upper-left area)”: upper half of the frame, left of centre; “top-right (upper-right area)”: upper half of the frame, right of centre; “bottom-left (lower-left area)”: lower half of the frame, left of centre; “bottom-right (lower-right area)”: lower half of the frame, right of centre. Position change amount: “stays in roughly the same place (barely shifts)”: the text barely moves; displacement is negligible; “moves a moderate amount (shifts across part of the frame)”: noticeable shift — text crosses a portion of the frame; “moves a large distance (shifts across much of the frame)”: large displacement — text traverses most of the frame. Overall motion direction: “stays roughly in place”: negligible net displacement throughout; “moves left”: predominantly leftward drift; “moves right”: predominantly rightward drift; “moves up”: predominantly upward drift; “moves down”: predominantly downward drift; “moves diagonally”: significant movement on both horizontal and vertical axes simultaneously; “changes direction partway”: reverses along its dominant axis mid-clip, e.g., moves right then left. Text size level: “small (a minor part of the frame)”: text occupies a small fraction of the frame, roughly <2% ; “medium (a noticeable block of the frame)”: clearly visible, moderate fraction, roughly 2 – 6% ; “large (a major part of the frame)”: covers a substantial portion of the frame, roughly 6 – 15% ; “very large (dominates the frame)”: fills most of the frame, roughly >15% . Scale change: “stays about the same size”: size is roughly constant throughout the clip; “gets larger”: text grows noticeably from the start to the end; “gets smaller”: text shrinks noticeably from the start to the end; “size fluctuates (goes up and down)”: size oscillates non-monotonically during the clip. Edge relation: “fully inside, away from the edges”: text stays well clear of all frame borders throughout; “comes close to an edge”: text approaches a border but does not touch it; “touches or runs off an edge”: text reaches or extends beyond a frame border. Answer the question based on the video. Directly return the answer with no extra text.
Appendix
Table 10: System prompt used for evaluating temporal video question answering.
It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.
Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang +8
University of British Columbia · Vector Institute · Etude AI +4
Despite remarkable progress in multimodal understanding, current MLLMs still exhibit limitations in video text understanding, particularly when semantics emerge through the integration of temporally distributed textual cues across multiple frames. This perception challenge fundamentally differs from static image text understanding, yet existing datasets fail to capture: the vast majority of questions remain answerable from single frames, inadequately reflecting real-world video text comprehension demands. To address this, we present ViTexQA, a large-scale video-text QA dataset, and FrameThinker for robust multi-frame temporal reasoning. We build ViTexQA via a quality-controlled Chain-of-Thought (CoT) annotation pipeline boosted with temporal constraints; all its QA pairs demand cross-frame text fusion to solve, enforcing true temporal reliance. FrameThinker adopts two-stage training for explicit temporal modeling: CoT-Guided Supervised Fine-Tuning (SFT) generates frame-aware reasoning chains, followed by Temporally-grounded Reinforcement Learning (RL) optimized with multi-frame coherence rewards. Evaluations show our method outperforms SOTA baselines on ViTexQA, lifting ROUGE-L by 6.3%.
Video text-based visual question answering (Video TextVQA) aims to answer questions by reasoning over visual textual content appearing in videos. Despite the strong multimodal video understanding capabilities of recent Video-LLMs, their performance on existing Video TextVQA benchmarks remains limited. To better understand this gap, we conduct an upper-bound analysis through frame-wise question answering, counting a sample as correct if any frame yields the right answer, which significantly outperforms direct video-based inference and reveals a substantial performance gap. The results suggest that the primary bottleneck lies in the localization of key question-relevant evidence, rather than in reasoning capacity itself. Building on this insight, we propose a question-guided agent framework that explicitly anchors the relevant keyframes before answering. The approach operates effectively in a training-free setting and consistently surpasses direct video inference. With additional supervised fine-tuning (SFT) and reinforcement learning (RL), it achieves an average improvement of +12.12 in accuracy and +11.15 in ANLS across benchmarks, establishing new state-of-the-art results. Our study underscores the critical role of explicit keyframe anchoring for advancing Video TextVQA. The code will be publicly released.
Haibin He, Maoyuan Ye, Jing Zhang +2
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan, Hubei, China