Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM Reasoning

Hongbin Lin; Sizhe Zou; Juangui Xu; Xinyue Xu; Zhuoli Ouyang; Stanislav Kolchin; Chengwei Qin; Zhongxiang Dai

cs.AINov 6, 2026

Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM Reasoning

Authors: Hongbin Lin, Sizhe Zou, Juangui Xu, Xinyue Xu, Zhuoli Ouyang, Stanislav Kolchin, Chengwei Qin, Zhongxiang Dai,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · CISPA Helmholtz Center for Information Security · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology ·

Abstract

Reasoning traces improve vision-language models on complex multimodal tasks, but forcing every input to follow a long reasoning process leads to overthinking and unnecessary inference cost. We study adaptive fast/slow thinking for VLMs, where one model answers directly for simple inputs and reasons step by step when needed. Fast answers and slow traces often come from distinct behavior policies, making a single-reference offline objective misweight one mode. We propose DUAL-REFERENCE BOLT (DR-BOLT), which distinguishes the desired fast/slow mixture reference from the sampling distribution and corrects their mismatch in reward-weighted training. Fast/slow partition competition explains routing in the optimal target; an efficient source-wise correction approximates the exact objective when response formats are well separated. Matched Qwen3-VL comparisons use identical candidate banks to isolate the objective from slow-teacher quality. DR-BOLT improves score-length trade-offs over the evaluated offline and online baselines on ScienceQA and three held-out transfer benchmarks, with a 24.1% cross-benchmark slow ratio and a 2.90-fold end-to-end training speedup over reproduced GRPO. These results support adaptive reasoning in the studied two-reference setting, while broader robustness remains open. Our code is available at https://github.com/HongbinLin3589/DR-BOLT.

Figures & tables

Dataset	GRPO	FAST-GRPO [ 31 ]	TON [ 26 ]	DualMindVLM [ 13 ]	DR-BOLT (Ours)
Dataset	Score / Length
ScienceQA (IID)	$0.962/201.0$	$0.964/37.6$	$0.963/35.9$	$0.872/87.2$	$\mathbf{0.967/20.4}$
MathVision (transfer)	$0.463/2498.2$	$0.527/1245.7$	$0.488/1505.8$	$0.302/446.0$	$\mathbf{0.606/1058.1}$
DynaMath (transfer)	$0.769/1402.2$	$0.782/512.4$	$0.775/495.5$	–	$\mathbf{0.799/370.6}$
MMMU-Pro (transfer)	$0.498/2006.4$	$0.540/892.6$	$0.519/747.3$	–	$\mathbf{0.581/385.7}$
Training Type	online RL / GRPO	online RL / GRPO	online RL / GRPO	online RL / GRPO	offline dual-ref BOLT

Table 1: Our reproductions of adaptive-thinking methods and GRPO. ScienceQA uses same-benchmark training; the other datasets measure transfer after ViRL39K training. Cells report

\text{Score}/\text{Length}

in words. Runtime includes candidate generation or online rollouts and all training stages; memory is peak per GPU. Speedup is relative to GRPO.

Dataset	Qwen3-Instruct	Qwen3-Thinking	Mix SFT	Reward SFT	Single-ref Fast	Single-ref Slow	DR-BOLT
Dataset	Score / Length
ScienceQA (IID)	$0.943/7.2$	$0.954/288.3$	$0.955/100.1$	$0.956/89.3$	$0.945/9.3$	$0.955/317.7$	$\mathbf{0.967/20.4}$
MathVision (transfer)	$0.244/395.8$	$0.368/5775.3$	$0.406/3777.6$	$0.421/3802.9$	$0.301/474.7$	$0.439/5901.8$	$\mathbf{0.606/1058.1}$
DynaMath (transfer)	$0.613/205.5$	$0.654/2696.7$	$0.730/1896.1$	$0.741/1675.0$	$0.627/241.4$	$0.752/2022.9$	$\mathbf{0.799/370.6}$
MMMU-Pro (transfer)	$0.278/265.3$	$0.377/3082.3$	$0.472/1352.6$	$0.484/1105.9$	$0.412/443.3$	$0.408/2575.1$	$\mathbf{0.581/385.7}$
Slow Ratio (%)	$0.0$	$99.8$	$67.2$	$60.4$	$1.9$	$97.4$	$24.1$

Table 2: Matched internal comparisons. Cells report

\text{Score}/\text{Length}

in words; ScienceQA and cross-benchmark results use separately trained checkpoints. Slow ratio (%) aggregates only MathVision, DynaMath, and MMMU-Pro.

Figure 1: Difficulty-wise response-Length distributions. Each group contains the response-Length distributions for DR-BOLT , FAST-GRPO, and GRPO. DR-BOLT stays short on easy examples and increases reasoning Length on harder examples, while GRPO-style training remains much longer.

Figure 2: Score–Length dynamics on ScienceQA. Solid lines show accuracy and dashed lines show response Length.

Figure 3: Partition-based routing calibration. Bars show empirical fast rate and markers show predicted fast probability.

Method	ScienceQA	MathVision	DynaMath	MMMU-Pro	Slow ratio (%)
MR-DPO	$0.964/31.8$	$0.568/1398.4$	$0.787/506.9$	$0.557/528.6$	$29.4$
Cond-RW	$0.960/48.6$	$0.513/1810.7$	$0.761/714.8$	$0.521/689.5$	$39.7$
DR-BOLT	$0.967/20.4$	$0.606/1058.1$	$0.799/370.6$	$0.581/385.7$	$24.1$

Table 3: Controlled offline alternatives on identical candidate banks. Cells report score / response words. ScienceQA uses a separate checkpoint; slow ratio aggregates the three transfer benchmarks.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$x=(I,Q)$	Multimodal input consisting of image $I$ and text query $Q$ .
$y$	Model response.
$y_{\mathrm{f}}$	Fast response, usually a concise direct answer.
$y_{\mathrm{s}}=(c,a)$	Slow response containing reasoning trace $c$ and final answer $a$ .
$z\in\{\mathrm{f},\mathrm{s}\}$	Latent response mode: fast or slow.
$\pi_{\theta}(y\|x)$	Trainable VLM policy.

Appendix

Table 4: Notation used in the paper.

Figure 4: Cross-likelihood heatmaps on the main path. The left panel reports mean per-token log-likelihood; the right panel reports mean sequence log-likelihood. Diagonal entries correspond to in-mode likelihoods, while off-diagonal entries measure cross-mode likelihoods.

	$y_{\mathrm{f}}$ $(\bar{L}=6.21)$	$y_{\mathrm{s}}$ $(\bar{L}=276.64)$
$\pi_{\mathrm{f}}$	$\mathbf{-0.181/-1.192}$	$-1.434/\mathbf{-274.380}$
$\pi_{\mathrm{s}}$	$-4.514/\mathbf{-19.465}$	$\mathbf{-0.190/-66.764}$

Appendix

Table 5: Cross-likelihood matrix on the main path. Each entry reports mean per-token log-likelihood / mean sequence log-likelihood in nats.

Direction	Mechanism	Length dependence
$\pi_{\mathrm{f}}(y_{\mathrm{s}})$	Length-compounded decay. The fast policy assigns slightly lower probability to many reasoning tokens; over long traces this compounds into an exponentially small sequence ratio. Empirically, the per-token mismatch is amplified over $\bar{L}\approx 277$ tokens.	Strong: $r(L,\Delta_{\mathrm{seq}})=-0.902$ .
$\pi_{\mathrm{s}}(y_{\mathrm{f}})$	Prefix bottleneck. The slow policy tends to start with a reasoning-format prefix such as <think> ; direct fast answers start with answer tokens instead. The mismatch is therefore concentrated near the first token rather than accumulated over length.	Weak: $r(L,\Delta_{\mathrm{seq}})=+0.226$ .

Appendix

Table 6: Different mechanisms behind the two cross-mode likelihood gaps.

Figure 5: Cross-likelihood heatmaps on the ablation path. The slow policy still assigns low likelihood to direct fast answers, indicating that the prefix bottleneck is a structural property of the checkpoint. A small fraction of contaminated fast samples weaken the fast-pool purity.

Figure 6: Correction weights in the balanced setting $\alpha=q=0.5$ . Both $\log w_{\mathrm{theory}}$ and $\log w_{\mathrm{exact}}$ collapse to zero, so correlation is undefined. The single-reference diagnostic $\log w_{\mathrm{single}}$ still shows a strong bimodal structure.

Samples	Mean	Std	Min	Max
$\log w_{\mathrm{single}}\mid y_{\mathrm{f}}$ $(n=958)$	$+18.27$	$4.76$	$-7.96$	$+29.49$
$\log w_{\mathrm{single}}\mid y_{\mathrm{s}}$ $(n=1000)$	$-207.62$	$91.96$	$-769.90$	$-101.92$

Appendix

Table 7: Statistics of

\log w_{\mathrm{single}}=\log\pi_{\mathrm{f}}(y|x)-\log\pi_{\mathrm{s}}(y|x)

in the balanced setting.

Figure 7: Correction weights on the main path with $\alpha=0.5,q=0.3$ . Since $\alpha\neq q$ , $\log w_{\mathrm{theory}}$ is non-constant and can be compared with $\log w_{\mathrm{exact}}$ .

Figure 8: Correction weights on the main path with $\alpha=0.5,q=0.7$ . The agreement between $w_{\mathrm{theory}}$ and $w_{\mathrm{exact}}$ is strongest in this setting.

Figure 9: Correction weights on the ablation path with $\alpha=0.5,q=0.3$ . The equivalence degrades because the ablation fast pool contains reasoning-style contaminated samples.

Metric	main, $q=0.3$	main, $q=0.7$	ablation, $q=0.3$
Pooled $r(w_{\mathrm{theory}},w_{\mathrm{exact}})$	$0.9877$	$\mathbf{0.9902}$	$0.9730$
$\|\mathrm{err}\|>0.01$	$1.5\%$	$1.2\%$	$2.2\%$
Max $\|\mathrm{err}\|$	$0.847$	$0.847$	$0.847$
Fast margin std	$4.76$	$4.76$	$\mathbf{7.87}$
Fast margin min	$-7.96$	$-7.96$	$\mathbf{-66.38}$

Appendix

Table 8: Equivalence between exact correction and mode-wise correction. The main path shows strong agreement, while the ablation path degrades due to contaminated fast samples.

Figure 10: Implicit KL trajectory on the main path under $(\alpha,q)=(0.5,0.7)$ . The mode-wise and exact buckets remain nearly indistinguishable throughout training.

Panel of Figure 11 — Figure 11: Training loss and gradient norm on the main path. The mode-wise and exact corrections have nearly identical optimization dynamics, while the full mode-wise implementation yields a smoother gradient profile.

Bucket	Step 25	Step 50	Step 100	Step 500	Final Loss	Grad Peak
Mode-wise	$0.64$	$7.75$	$8.08$	$\mathbf{11.28}$	$0.154$	$47.0$
Exact	$0.61$	$7.77$	$8.06$	$\mathbf{11.27}$	$0.153$	$46.5$
Single-ref	$0.67$	$7.76$	$8.02$	$10.69$	$0.142$	$46.7$
Mode-wise-full	$0.62$	$7.23$	$8.36$	$10.39$	$0.151$	$\mathbf{28.8}$

Appendix

Table 9: Optimization dynamics on the main path. KL values are sequence-level implicit KL in nats.

Figure 12: Implicit KL trajectory on the ablation path. The relative differences between buckets remain similar to the main path, but the absolute optimization scale is much larger due to first-token mismatch.

Dataset	Setting	Metric
ScienceQA [ 15 ]	IID: train on ScienceQA, test on ScienceQA	Accuracy
MathVision [ 27 ]	Transfer: train on ViRL39K, test on MathVision	Accuracy
DynaMath [ 41 ]	Transfer: train on ViRL39K, test on DynaMath	Accuracy
MMMU-Pro [ 39 ]	Transfer: train on ViRL39K, test on MMMU-Pro	Accuracy

Appendix

Table 10: Datasets and evaluation protocol.

Hyperparameter	Value / Definition
Target fast prior $\alpha$	$0.5$
Empirical fast sampling ratio $q$	$0.7$
Fast candidates $N_{\mathrm{f}}$	$7$ per input
Slow candidates $N_{\mathrm{s}}$	$3$ per input
BOLT temperature $\beta$	$1.0$
Format reward weight $\gamma$	$0.1$

Appendix

Table 11: Reward and correction hyperparameters used in the main experiments.

Item	Value
Framework	PyTorch
Fine-tuning method	LoRA weighted SFT
Optimizer	AdamW
Training steps	$500$ for white-box optimization diagnostics; full benchmark runs use the same budget across matched baselines
Per-device batch size	$1$
Gradient accumulation	$8$

Appendix

Table 12: Training configuration for reproduced internal experiments.

Correction	MathVision	DynaMath	MMMU-Pro	Slow Ratio
No $\kappa$	$0.260/518.2$	$0.626/308.1$	$0.314/329.9$	$0.13$
Exact $\kappa$	$0.623/1023.2$	$0.804/325.7$	$0.594/382.0$	$22.0$
Mode-wise $\kappa$	$0.606/1058.1$	$0.799/370.6$	$0.581/385.7$	$24.1$

Appendix

Table 13: Downstream ablation of behavior-reference correction under

\alpha=0.5,q=0.7

. Each benchmark cell reports

\text{Score}/\text{Length}

. Slow Ratio is reported in percentage points.

Change from default	Score	Length	Calibration slow ratio (%)
Default	$0.967$	$20.4$	$51.4$
$\alpha=0.3$	$0.968$	$25.5$	$71.1$
$\alpha=0.7$	$0.964$	$15.2$	$31.2$
$q=0.3$	$0.965$	$21.4$	$53.8$
$q=0.5$	$0.966$	$20.8$	$52.2$
$\beta=0.5$	$0.966$	$17.8$	$45.1$

Appendix

Table 14: One-factor ScienceQA sensitivity study. Length is mean response words. Slow ratio is measured on the 100-prompt repeated-decoding calibration subset used for Figure 3 , not the full ScienceQA test set.

$\lambda$	MathVision	MMMU-Pro	Cross-benchmark slow ratio (%)
$0$	$0.609/1396.3$	$0.578/512.8$	$31.8$
$0.025$	$0.614/1208.6$	$0.585/445.2$	$27.7$
$0.05$	$0.606/1058.1$	$0.581/385.7$	$24.1$
$0.10$	$0.589/811.4$	$0.561/302.7$	$17.4$

Appendix

Table 15: Shared length-penalty sweep after ViRL39K training. Cells report score / mean response words; slow ratio aggregates MathVision, DynaMath, and MMMU-Pro.

Method	ScienceQA	MathVision	DynaMath	MMMU-Pro	Slow ratio (%)
Two-policy router	$0.958/64.7$	$0.401/2018.5$	$0.681/827.8$	$0.416/1054.6$	$28.9$
Thinking prompt router	$0.953/252.4$	$0.366/5108.7$	$0.652/2396.4$	$0.375/2748.9$	$87.6$
DR-BOLT	$0.967/20.4$	$0.606/1058.1$	$0.799/370.6$	$0.581/385.7$	$24.1$

Appendix

Table 16: Router controls. ScienceQA and transfer results use separately trained systems. Cells report score / mean response words; slow ratio aggregates the three transfer benchmarks.

Dataset	Answer span	Parsing outcome
ScienceQA	B. New Hampshire	Leading-option rule; B matches ground truth B.
ScienceQA	B after a 168-word slow response	Reasoning is discarded; single-letter rule yields B.
ScienceQA	Neither.	No option is recovered; zero correctness and format reward.
MathVision	Unboxed arithmetic-sequence derivation	Last numeric literal is 60; matches ground truth 60.
MathVision	A for a multiple-choice item	Letter path yields A; matches the correct option.
MMMU-Pro	E, H	Multiple options; no unique answer, so zero correctness and format reward.

Appendix

Table 17: Representative parsing cases from generated responses.

Role	Checkpoint and revision
Fast reference	Qwen/Qwen3-VL-8B-Instruct 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b
Slow reference and initialization	Qwen/Qwen3-VL-8B-Thinking 92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b
FAST setup verification	xiaowenyi/FAST-8B-Preview 45aa28a69976f14f295f34d4bde25099a84cf1e5

Appendix

Table 18: Model snapshot revisions.

Item	Value
LoRA	Rank 8, scaling factor 64, dropout 0.05, no bias.
Target modules	q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , down_proj .
AdamW	Learning rate $5\times 10^{-6}$ , weight decay $0.1$ , moments $(0.9,0.95)$ .
Schedule	Cosine decay with warm-up ratio $0.1$ .
Batch	Four GPUs, per-device batch 1, gradient accumulation 8; effective batch 32 response sequences.
Precision / attention	bf16 / SDPA; key-value cache disabled during training.

Appendix

Table 19: Additional shared optimization and decoding configuration.

Method	Time (h)	Included stages
GRPO	$202.4$	Online rollout generation and policy updates.
FAST-GRPO	$176.6$	Online rollout generation and policy updates under the released FAST recipe.
TON	$248.9$	Thought-dropout SFT, online rollouts, and policy updates.
DualMindVLM	–	Runtime is not reported.
DR-BOLT	$69.7$	Fast/slow candidate generation, correctness and format rewards, weight construction, and policy optimization.

Appendix

Table 20: Runtime provenance on four A100-80GB GPUs. Times are wall-clock hours from our reproduced runs.

Method	Score	Length
Fast reference	$0.531$	$13.9$
Slow reference	$0.548$	$1846.1$
DR-BOLT	$0.546$	$452.2$

Appendix

Table 21: 3DSRBench results on the constructed held-out test partition. Length is mean response words.

Reasoning for VLMs

See all 78 papers

Balanced Thinking: Improving Chain of Thought Training in Vision Language Models
Mar 19, 2026Shaked Perek, Ben Wiesel, Avihu Dekel +2PEFT of VLMs VLM Reasoning
IBM Research
Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning
Oct 11, 2026Quanlin Chen, Xiuwei Chen, Kun Xiang +5Multimodal Large Language Models Multimodal Reasoning
Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy +2
Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation
Mar 27, 2026Yiming Ren, Yujiu Yang, Junjie WangVLM Adaptation VLM Reasoning
Tsinghua University

Mar 19, 2026J/K move · Enter open · S save

Balanced Thinking: Improving Chain of Thought Training in Vision Language Models

Shaked Perek, Ben Wiesel, Avihu Dekel, Nimrod Shabtay+1

IBM Research

PEFT of VLMs VLM Reasoning Vision-Language Models Supervised Fine-Tuning Curriculum Learning

Open paper AskPDF arXiv

Multimodal reasoning in vision-language models (VLMs) typically relies on a two-stage process: supervised fine-tuning (SFT) and reinforcement learning (RL). In standard SFT, all tokens contribute equally to the loss, even though reasoning data are inherently token-imbalanced. Long <think> traces overshadow short but task-critical <answer> segments, leading to verbose reasoning and inaccurate answers. We propose SCALe (Scheduled Curriculum Adaptive Loss), which explicitly separates supervision over reasoning and answer segments using dynamic, length-independent weighting. Unlike vanilla SFT, which overweights the <think> segment, SCALe-SFT gradually shifts the focus from <think> to <answer> throughout training via a cosine scheduling policy, encouraging concise and well-grounded reasoning. We evaluate SCALe across diverse benchmarks and architectures. Results show that SCALe consistently improves accuracy over vanilla SFT and matches the performance of the full two-phase SFT + GRPO pipeline while requiring only about one-seventh of the training time, making it a lightweight yet effective alternative. When combined with GRPO, SCALe achieves the best overall performance, highlighting its value both as a standalone method and as a strong foundation for reinforcement refinement.

Abstract

Reasoning for VLMs

Explore similar work

Balanced Thinking: Improving Chain of Thought Training in Vision Language Models

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning

Mitigating the Reasoning Tax in Vision-Language Fine-Tuning with Input-Adaptive Depth Aggregation