Label-free test-time reinforcement learning adapts vision--language models using rewards derived from their own predictions. However, misinterpreted visual evidence and sensitivity to textual formulation can produce a consistent but incorrect consensus that majority voting reinforces. Cross-view aggregation can expose such errors, but image and text transformations may themselves degrade task-relevant evidence to different extents. We introduce \method, a reliability-aware harmonic consensus framework for multimodal test-time reinforcement learning. \method constructs a 2×2 grid combining original and perturbed images with direct-solving and reframing modes, selects pseudo-labels through weighted harmonic aggregation, and adapts modality-level weights using batch-averaged, temporally smoothed shifts in pseudo-label confidence as a heuristic reliability signal. We formulate consensus as a weighted χ2-barycenter problem whose unique minimizer is the normalized weighted harmonic aggregate, without requiring correct-answer invariance. An exact support--dispersion decomposition characterizes when harmonic aggregation preserves or corrects arithmetic rankings, while a sensitivity analysis establishes how modality reweighting changes pairwise ranking margins, informing interpretation of the heuristic updates. Across six benchmarks and three Qwen3-VL model scales, \method outperforms TTRV in 16 of 18 benchmark--model settings on adaptation instances and 17 of 18 settings on held-out instances under matched estimation and optimization rollout counts. These results demonstrate improved target-task adaptation and transfer to unseen instances without further updates.
Figures & tables
Figure 1: Overview of TIMBRE. Four views combine original or perturbed images with direct-solving or reframe-and-solve prompts. Weighted harmonic consensus selects the pseudo-label y∗ . Cross-view shifts in confidence assigned to y∗ update modality-level reliability factors, which determine view weights for the next step (dashed arrow). For each instance, rollout rewards are pooled and jointly normalized to compute GRPO advantages.
Visual Mathematical Reasoning
Multi-Discipline
Method
MathVista
MathVision
MathVerse
DynaMath
WeMath
MMMU-Pro (Science)
Qwen3-VL-2B-Instruct
Before RL
66.3 / 63.1
26.9 / 28.5
39.8 / 42.8
53.3 / 53.6
62.3 / 60.7
50.3 / 46.4
- TTRV
72.8 / 66.7
28.7 / 27.5
46.7 / 47.8
58.7 / 57.2
71.4 / 66.3
50.6 / 43.0
- Timbre (ours)
74.2 / 68.8
29.2 / 28.3
47.0 / 49.4
60.6 / 59.1
73.8 / 69.6
50.6 / 45.8
Qwen3-VL-4B-Instruct
Table 1: Adaptation and held-out performance (pass@1, %). Each entry reports Val-TTT / Val : performance on adaptation instances / held-out instances. Bold and underline indicate the best and second-best distinct scores, respectively, within each model size, benchmark, and evaluation setting.
Figure 2: Pseudo-label accuracy during test-time adaptation. Comparison between Timbre and TTRV on DynaMath with Qwen3-VL-8B-Instruct. Curves are smoothed using a time-weighted exponential moving average for visualization. Ground-truth answers are used only for post-hoc evaluation and are unavailable during adaptation. Timbre maintains higher pseudo-label accuracy over most later adaptation steps, indicating improved reliability of the generated training signal.
Method
MathVista
DynaMath
Before RL
77.0 / 74.2
65.7 / 66.4
- Weighted arithmetic aggregation
80.1 / 75.4
68.2 / 67.6
- Uniform harmonic aggregation
79.5 / 75.0
69.2 / 67.5
- Weighted harmonic aggregation ( Timbre )
81.1 / 75.5
69.2 / 67.7
Table 2: Ablation of consensus aggregation strategies on Qwen3-VL-4B. Each entry reports Val-TTT / Val pass@1 (%). All variants use the same optimization settings.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Hugging Face source and configuration
Adaptation
Held-out
Total
MathVista
AI4Math/MathVista ( testmini )
500
500
1,000
MathVerse
AI4Math/MathVerse ( testmini , Text Lite)
500
278
778
MathVision
MathLLMs/MathVision (adaptation: test ; held-out: testmini )
Table 3: Data sources and split sizes for label-free adaptation and held-out evaluation.
Hyperparameter
Value
Training and optimization
Maximum prompt length
6,320 tokens
Maximum response length
8,192 tokens
Training batch size
32
Training epochs
16
PPO mini-batch size
4
Appendix
Table 4: Hyperparameters used in our experiments. M and K denote per-view rollout counts for Timbre .
Figure 3: Reliability-aware view weighting dynamics. The top row shows the evolution of the learned harmonic-consensus weights for the four image–text views, and the bottom row reports the corresponding per-view pseudo-label accuracy. Each column corresponds to the same view vit , enabling comparison between assigned consensus influence and empirical correctness during adaptation. The first index denotes the image state (original or transformed), and the second index denotes the text mode (direct solving or reframing).
Test-time reinforcement learning enables vision-language models (VLMs) to adapt using unlabeled inputs. However, repeated sampling under fixed visual conditions can reinforce shared perceptual errors, while sequence-level rewards fail to isolate visual perception the foundational bottleneck that anchors multimodal reasoning risking the degradation of pre-trained reasoning capabilities. We propose TTRSD, a test-time reinforcement learning framework combining multi-view answer-level self-distillation with visual contrastive token selection. A shared policy aggregates teacher predictions across original, cropped, and downsampled views into an answer distribution. Student trajectories generated from the original image receive rewards based on the support for their final answers in this distribution. To allocate this feedback precisely toward perceptual bottlenecks, we compare the log-probabilities of the same sampled tokens under original and visually ablated inputs while holding their textual prefixes fixed, selecting visually sensitive positions for policy-gradient updates. TTRSD separates update direction, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher. With only 20 unlabeled adaptation samples, TTRSD improves performance across seven benchmarks and three VLMs, raising InternVL3-2B's MMMU accuracy from 35.79% to 49.32%(+13.53%), demonstrating cross-dataset generalization while preserving inherent reasoning integrity.
Shuning Wang, Zhiheng Wu, Xun Zhou +8
Zhejiang University · Hong Kong University of Science and Technology · Harbin Institute of Technology +1
Vision-language models transfer well in zero-shot settings, but at deployment the visual and textual branches often shift asymmetrically. Under this condition, entropy-based test-time adaptation can sharpen the fused posterior while increasing error, because an unreliable modality may still dominate fusion. We study this failure mode through a majorization view of multimodal posteriors and cast adaptation as a constrained de-mixing problem on the fused prediction. Based on this view, we propose MG-MTTA, which keeps the backbone frozen and updates only a lightweight gate or adapter. The objective combines fused-posterior entropy minimization with a reliability-aware gate prior built from anchor-based modality consistency and cross-modal conflict. Our analysis gives conditions under which entropy reduction preserves the correct ranking and a threshold that characterizes modality-dominance failure. On the ImageNet-based benchmark, MG-MTTA improves top-1 accuracy from 57.97 to 66.51 under semantics-preserving textual shift and from 21.68 to 26.27 under joint visual-textual shift, while remaining competitive in the visual-only benchmark. These results show that multimodal test-time adaptation should control modality reliability, not just prediction entropy.
Lixian Chen, Yanhui Chen, Junyi Lin
Guangdong University of Technology · Guangzhou, China
Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy with Qwen2.5-7B, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000x fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability. These results demonstrate that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.
Naveen Vakada, Mingyuan Li, Shaoxiong Ji
University of Turku, Finland · ELLIS Institute Finland