Timbre: Reliability-aware harmonic consensus for label-free multimodal test-time rexinforcement learning

Haotian Lyu; Haoyu Wang; Ru Wang; Feng Ji; Ziwen Li; Jiaxian Guo; Mingming Gong

cs.AIOct 29, 2026

Timbre: Reliability-aware harmonic consensus for label-free multimodal test-time rexinforcement learning

Authors: Haotian Lyu, Haoyu Wang, Ru Wang, Feng Ji, Ziwen Li, Jiaxian Guo, Mingming Gong

Organizations: University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence · The University of Tokyo · Google

Abstract

Label-free test-time reinforcement learning adapts vision--language models using rewards derived from their own predictions. However, misinterpreted visual evidence and sensitivity to textual formulation can produce a consistent but incorrect consensus that majority voting reinforces. Cross-view aggregation can expose such errors, but image and text transformations may themselves degrade task-relevant evidence to different extents. We introduce \method, a reliability-aware harmonic consensus framework for multimodal test-time reinforcement learning. \method constructs a $2\times2$ grid combining original and perturbed images with direct-solving and reframing modes, selects pseudo-labels through weighted harmonic aggregation, and adapts modality-level weights using batch-averaged, temporally smoothed shifts in pseudo-label confidence as a heuristic reliability signal. We formulate consensus as a weighted $χ^2$ -barycenter problem whose unique minimizer is the normalized weighted harmonic aggregate, without requiring correct-answer invariance. An exact support--dispersion decomposition characterizes when harmonic aggregation preserves or corrects arithmetic rankings, while a sensitivity analysis establishes how modality reweighting changes pairwise ranking margins, informing interpretation of the heuristic updates. Across six benchmarks and three Qwen3-VL model scales, \method outperforms TTRV in 16 of 18 benchmark--model settings on adaptation instances and 17 of 18 settings on held-out instances under matched estimation and optimization rollout counts. These results demonstrate improved target-task adaptation and transfer to unseen instances without further updates.

Figures & tables

Figure 1: Overview of TIMBRE. Four views combine original or perturbed images with direct-solving or reframe-and-solve prompts. Weighted harmonic consensus selects the pseudo-label $y^{*}$ . Cross-view shifts in confidence assigned to $y^{*}$ update modality-level reliability factors, which determine view weights for the next step (dashed arrow). For each instance, rollout rewards are pooled and jointly normalized to compute GRPO advantages.

Qwen3-VL-2B-Instruct
	Visual Mathematical Reasoning					Multi-Discipline
Method	MathVista	MathVision	MathVerse	DynaMath	WeMath	MMMU-Pro (Science)
Before RL	66.3 / 63.1	26.9 / 28.5	39.8 / 42.8	53.3 / 53.6	62.3 / 60.7	50.3 / 46.4
- TTRV	72.8 / 66.7	28.7 / 27.5	46.7 / 47.8	58.7 / 57.2	71.4 / 66.3	50.6 / 43.0
- Timbre (ours)	74.2 / 68.8	29.2 / 28.3	47.0 / 49.4	60.6 / 59.1	73.8 / 69.6	50.6 / 45.8
Qwen3-VL-4B-Instruct

Table 1: Adaptation and held-out performance (pass@1, %). Each entry reports Val-TTT / Val : performance on adaptation instances / held-out instances. Bold and underline indicate the best and second-best distinct scores, respectively, within each model size, benchmark, and evaluation setting.

Figure 2: Pseudo-label accuracy during test-time adaptation. Comparison between Timbre and TTRV on DynaMath with Qwen3-VL-8B-Instruct. Curves are smoothed using a time-weighted exponential moving average for visualization. Ground-truth answers are used only for post-hoc evaluation and are unavailable during adaptation. Timbre maintains higher pseudo-label accuracy over most later adaptation steps, indicating improved reliability of the generated training signal.

Method	MathVista	DynaMath
Before RL	77.0 / 74.2	65.7 / 66.4
- Weighted arithmetic aggregation	80.1 / 75.4	68.2 / 67.6
- Uniform harmonic aggregation	79.5 / 75.0	69.2 / 67.5
- Weighted harmonic aggregation ( Timbre )	81.1 / 75.5	69.2 / 67.7

Table 2: Ablation of consensus aggregation strategies on Qwen3-VL-4B. Each entry reports Val-TTT / Val pass@1 (%). All variants use the same optimization settings.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Hugging Face source and configuration	Adaptation	Held-out	Total
MathVista	AI4Math/MathVista ( testmini )	500	500	1,000
MathVerse	AI4Math/MathVerse ( testmini , Text Lite)	500	278	778
MathVision	MathLLMs/MathVision (adaptation: test ; held-out: testmini )	500	304	804
DynaMath	DynaMath/DynaMath_Sample (adaptation: variant 1; held-out: variant 2)	465	465	930
WeMath	We-Math/We-Math ( testmini )	500	500	1,000
MMMU-Pro (Science)	MMMU/MMMU_Pro (standard four-option, Science)	198	93	291

Appendix

Table 3: Data sources and split sizes for label-free adaptation and held-out evaluation.

Hyperparameter	Value
Training and optimization
Maximum prompt length	6,320 tokens
Maximum response length	8,192 tokens
Training batch size	32
Training epochs	16
PPO mini-batch size	4

Appendix

Table 4: Hyperparameters used in our experiments.

M

and

K

denote per-view rollout counts for Timbre .

Panel of Figure 3 — Figure 3: Reliability-aware view weighting dynamics. The top row shows the evolution of the learned harmonic-consensus weights for the four image–text views, and the bottom row reports the corresponding per-view pseudo-label accuracy. Each column corresponds to the same view $v_{it}$ , enabling comparison between assigned consensus influence and empirical correctness during adaptation. The first index denotes the image state (original or transformed), and the second index denotes the text mode (direct solving or reframing).

Test-Time Adaptation of VLMs

Test-time adaptation of vision-language models uses unlabeled deployment data to improve predictions under domain shifts without retraining.

See all 80 papers

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models
Sep 27, 2026Shuning Wang, Zhiheng Wu, Xun Zhou +8Test-Time Adaptation of VLMs VLM Distillation
Zhejiang University · Hong Kong University of Science and Technology · Harbin Institute of Technology +1
Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift
Apr 27, 2026Lixian Chen, Yanhui Chen, Junyi LinVLM Robustness Test-Time Adaptation
Guangdong University of Technology · Guangzhou, China
Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces
Sep 16, 2026Naveen Vakada, Mingyuan Li, Shaoxiong JiFine-Tuning Test-Time Adaptation of VLMs
University of Turku, Finland · ELLIS Institute Finland

Sep 27, 2026J/K move · Enter open · S save

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models

Shuning Wang, Zhiheng Wu, Xun Zhou, Chongyang Cui+7

Zhejiang University · Hong Kong University of Science and Technology · Harbin Institute of Technology+1

Test-Time Adaptation of VLMs VLM Distillation RL for VLMs

Open paper AskPDF arXiv

Test-time reinforcement learning enables vision-language models (VLMs) to adapt using unlabeled inputs. However, repeated sampling under fixed visual conditions can reinforce shared perceptual errors, while sequence-level rewards fail to isolate visual perception the foundational bottleneck that anchors multimodal reasoning risking the degradation of pre-trained reasoning capabilities. We propose TTRSD, a test-time reinforcement learning framework combining multi-view answer-level self-distillation with visual contrastive token selection. A shared policy aggregates teacher predictions across original, cropped, and downsampled views into an answer distribution. Student trajectories generated from the original image receive rewards based on the support for their final answers in this distribution. To allocate this feedback precisely toward perceptual bottlenecks, we compare the log-probabilities of the same sampled tokens under original and visually ablated inputs while holding their textual prefixes fixed, selecting visually sensitive positions for policy-gradient updates. TTRSD separates update direction, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher. With only 20 unlabeled adaptation samples, TTRSD improves performance across seven benchmarks and three VLMs, raising InternVL3-2B's MMMU accuracy from 35.79% to 49.32%(+13.53%), demonstrating cross-dataset generalization while preserving inherent reasoning integrity.

Abstract

Test-Time Adaptation of VLMs

Explore similar work

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models

Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces