Test-time reinforcement learning can adapt vision-language models (VLMs) to unlabeled target data, but its effectiveness is fundamentally limited by the reliability of self-generated learning signals. To assess the reliability of consensus-based learning signals, we analyze VLM test-time reinforcement learning across diverse VQA datasets and model sizes, revealing two limitations. First, gains from consensus-based test-time training largely come from answer normalization rather than content correction. Second, many initial VLM responses are incorrect due to the model's limited ability to jointly use the image and the question; consensus rewards derived from these outputs may preserve the resulting grounding errors rather than correct them. Motivated by these, we propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation. TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants, using the resulting token-level likelihood changes to estimate dependence on each input. It combines image and question dependence with calibrated confidence to construct a response-level reward that favors jointly grounded responses, and uses the token-level signals to assign greater positive policy credit to tokens supported by both inputs. This design favors responses that are jointly grounded in the image and the question and sufficiently confident, rather than merely popular. Experiments across eight VQA datasets and multiple VLM sizes show that TTIQ achieves the best average performance at every model scale. It further generalizes across VLM families, while models trained on one dataset improve performance on unseen datasets without further training.
Figures & tables
Figure 1: Analysis of where the gains of TTRV ( Singh et al., 2026 ) come from across the eight-task evaluation. In (a), an exact-match correction is considered form-only when the Base response already contains an acceptable ground-truth span; 72.3% of the corrections fall into this category. In (b), TTRV improves exact match across model scales, while ground-truth answer containment decreases.
Figure 2: Left: Incorrect pre-update responses across eight tasks, grouped by dependence on the question only, the image only, neither input, or both inputs. Numbers indicate the percentage lacking joint image–question dependence. Right: Example predictions from Base, TTRV, and TTIQ , together with the ground-truth answer.
Figure 3: Overview of TTIQ . Given an unlabeled image–question pair, the VLM samples multiple responses. Each response is teacher-forced under image-ablated and question-ablated inputs to estimate token-level image and question dependence. These signals are used to construct a response-level reward and token-level credit assignment, followed by a test-time GRPO update.
Method
InfoVQA
TextVQA
GQA-Rel.
GQA-Attr
ST-VQA
MathVerse-VD
COCO-QA-N
CLEVR-H
Mean
LLaMA-3.2-11B
46.36
62.59
2.76
14.72
61.55
8.72
36.38
24.19
32.16
LLaVA-1.5-7B
15.48
53.43
41.54
56.30
43.06
14.58
63.80
36.33
40.57
Qwen2.5-VL-3B
66.06
81.87
37.41
55.61
71.25
28.65
73.75
80.60
61.90
Qwen2.5-VL-7B
70.62
84.40
33.69
47.42
76.69
32.16
78.79
80.70
63.06
Qwen3-VL-2B
62.30
80.52
34.21
48.91
69.17
27.08
72.80
71.69
58.34
Qwen3-VL-4B
70.07
81.83
41.00
54.93
71.56
30.73
74.99
79.98
63.14
Table 1: Comparison with all test-time reinforcement learning baselines. Best and second-best results within each backbone block are shown in bold and underlined.
Backbone
Method
Target dataset
Avg.
Source: GQA-Attr
GQA-Rel
COCO-N
MathV-VD
IVL3-2B
Base
26.08
67.53
25.91
39.84
IVL3-2B
TTRV
29.46
72.72
26.82
43.00
IVL3-2B
TTIQ
30.34
73.31
26.95
43.53
IVL2.5-4B
Base
17.25
36.01
28.91
27.39
IVL2.5-4B
TTRV
30.00
71.12
29.17
43.43
Table 2: Cross-dataset generalization of TTIQ. Bold values indicate the best result for each setting.
Method
InfoVQA
GQA-Attr
GQA-Rel
COCO-N
Avg.
n
3,268
1,746
3,873
2,735
Qwen2.5-VL-3B
Base
66.06
55.61
37.41
73.75
58.21
TTRV
65.97
55.44
37.31
73.49
58.05
TTIQ
66.03
55.90
37.70
74.19
58.45
Qwen2.5-VL-7B
Table 3: Generalization of TTIQ to Qwen2.5-VL.
Method
Reward
Credit
InfoVQA
TextVQA
GQA-Attr
GQA-Rel.
Avg.
Reference methods
Base
57.89
73.55
45.59
17.25
48.57
TTRV
answer frequency
58.35
74.76
55.50
31.06
54.92
Reward components
Image only
I
full response
59.49
78.71
56.13
36.33
57.66
Question only
Q
full response
57.65
78.41
56.93
23.96
54.24
Table 4: Component ablation on InternVL2.5-4B. Bold indicates the best result in each column over the entire table.
Figure 4: Sensitivity to confidence strength and its limiting behavior. (a) Accuracy gain over TTRV with InternVL2.5-4B; hollow markers indicate the best strength for each task. (b) Relationship between the raw-rollout correct answer rate and the relative effect of confidence on incorrect and correct responses.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Task focus
Public labeled source split
Unlabeled train
Evaluation
InfoVQA
Infographic reading and reasoning
Official test split
20
3,268
TextVQA
Scene-text question answering
Validation split
20
4,980
GQA-Relation
Visual relations
Test-dev balanced, Relation subset
20
3,873
GQA-Attribute
Visual attributes
Test-dev balanced, Attribute subset
20
1,746
ST-VQA
Scene-text question answering
Public train split
20
26,054
MathVerse-VD
Vision-dominant mathematical reasoning
Test-mini, Vision-Dominant subset
20
768
Appendix
Table 5: Dataset statistics for the experiments. The 20 training answers are withheld from every test-time training method.
Train n
Method
InfoVQA
TextVQA
GQA-Rel.
COCO-QA-N
Avg.
20
TTRV
58.31
74.73
31.00
69.91
58.49
TTIQ
59.69
79.45
37.12
75.03
62.82
50
TTRV
59.03
77.12
35.43
73.56
61.29
TTIQ
58.85
79.14
37.33
73.71
62.26
100
TTRV
57.94
77.86
38.20
72.81
61.70
TTIQ
58.47
80.43
37.36
74.16
62.60
Appendix
Table 6: Exact match accuracy (%) with 20, 50, and 100 unlabeled training examples. Bold marks the better method for each training size and task.
Figure 5: Effect of training-set size with InternVL2.5-4B. (a) Macro accuracy over InfoVQA, TextVQA, GQA-Relation, and COCO-QA-N. (b) Mean training time on four GH200 GPUs. Training time includes only the update loop; model setup and evaluation are excluded.
Method
Extra TF
Sec./update
8 updates
Peak GB/GPU
TTRV
0
21.9
2.92 min
76.84
TTIQ
2
35.8
4.78 min
76.84
Appendix
Table 7: Matched training-only efficiency on InternVL2.5-4B and InfoVQA with four GH200 GPUs.
Task
Question
Base
TTRV
TTIQ
GT
GQA-Rel.
What vehicle is in front of the red vehicle?
A car.
Car.
SUV
suv
GQA-Rel.
What are the jars sitting on top of?
Tea cups.
Cups.
Stove.
stove
GQA-Attr
How clean are the pots?
The pots are clean.
Clean.
Dirty.
dirty
GQA-Attr
Which side of the image is the cyclist on?
Left.
Left
Right.
right
ST-VQA
What number is this bus known as?
Private bus.
Private.
23
23
ST-VQA
What does the advertisement say behind the woman’s head?
Stretchy shery
Stretchy
Stretch
Stretch
Appendix
Table 9: Representative form-to-content correction chains from the final InternVL3-2B outputs. TTRV changes the surface form or length of the Base response but retains the same central incorrect answer, whereas TTIQ changes the answer content to an acceptable ground truth.
Figure 6: Qualitative examples of image–question dependence before test-time training. Each example compares two responses sampled for the same image–question input. Incorrect responses either follow visual content without sufficient question alignment (top) or exhibit weak image dependence (bottom), while the correct responses depend more strongly on both inputs.
Test-time reinforcement learning enables vision-language models (VLMs) to adapt using unlabeled inputs. However, repeated sampling under fixed visual conditions can reinforce shared perceptual errors, while sequence-level rewards fail to isolate visual perception the foundational bottleneck that anchors multimodal reasoning risking the degradation of pre-trained reasoning capabilities. We propose TTRSD, a test-time reinforcement learning framework combining multi-view answer-level self-distillation with visual contrastive token selection. A shared policy aggregates teacher predictions across original, cropped, and downsampled views into an answer distribution. Student trajectories generated from the original image receive rewards based on the support for their final answers in this distribution. To allocate this feedback precisely toward perceptual bottlenecks, we compare the log-probabilities of the same sampled tokens under original and visually ablated inputs while holding their textual prefixes fixed, selecting visually sensitive positions for policy-gradient updates. TTRSD separates update direction, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher. With only 20 unlabeled adaptation samples, TTRSD improves performance across seven benchmarks and three VLMs, raising InternVL3-2B's MMMU accuracy from 35.79% to 49.32%(+13.53%), demonstrating cross-dataset generalization while preserving inherent reasoning integrity.
Shuning Wang, Zhiheng Wu, Xun Zhou +8
Zhejiang University · Hong Kong University of Science and Technology · Harbin Institute of Technology +1
Test-time scaling (TTS) methods have proven highly effective for LLMs, yet their application to vision-language models (VLMs) remains relatively underexplored. Existing VLM TTS methods largely require open-weight model access or expensive repeated sampling, and are evaluated primarily on multimodal mathematical and scientific reasoning benchmarks rather than natural-image understanding. In this paper, we propose Test-Time Hinting, a method that improves VLM performance with a single call to the target VLM and requires only black-box API access, which makes it broadly applicable to frontier closed-weight models. Our method is motivated by the observation that VLM errors tend to cluster around recurring failure patterns that are shared across models and predictable from the input alone. We therefore train a lightweight hint generator model to predict, for a given test input, which "hint" should be prepended to the prompt, providing targeted contextual or procedural guidance that steers the VLM away from its characteristic failure modes without revealing the answer. We show that Test-Time Hinting significantly improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks and that these gains generalize to unseen benchmarks and VLMs without retraining the hint generator.
Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.
Ziqian Zhang, Junjie Liao, Zhining Zhang +1
National University of Singapore · Beijing Normal University · Peking University +1