Learning When to Think: Dual-Reference Offline Optimization for Adaptive VLM Reasoning
Authors: Hongbin Lin, Sizhe Zou, Juangui Xu, Xinyue Xu, Zhuoli Ouyang, Stanislav Kolchin, Chengwei Qin, Zhongxiang Dai, Yao Shu
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · CISPA Helmholtz Center for Information Security · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong, Shenzhen
Reasoning traces improve vision-language models on complex multimodal tasks, but forcing every input to follow a long reasoning process leads to overthinking and unnecessary inference cost. We study adaptive fast/slow thinking for VLMs, where one model answers directly for simple inputs and reasons step by step when needed. Fast answers and slow traces often come from distinct behavior policies, making a single-reference offline objective misweight one mode. We propose DUAL-REFERENCE BOLT (DR-BOLT), which distinguishes the desired fast/slow mixture reference from the sampling distribution and corrects their mismatch in reward-weighted training. Fast/slow partition competition explains routing in the optimal target; an efficient source-wise correction approximates the exact objective when response formats are well separated. Matched Qwen3-VL comparisons use identical candidate banks to isolate the objective from slow-teacher quality. DR-BOLT improves score-length trade-offs over the evaluated offline and online baselines on ScienceQA and three held-out transfer benchmarks, with a 24.1% cross-benchmark slow ratio and a 2.90-fold end-to-end training speedup over reproduced GRPO. These results support adaptive reasoning in the studied two-reference setting, while broader robustness remains open. Our code is available at https://github.com/HongbinLin3589/DR-BOLT.
Figures & tables
Dataset
GRPO
FAST-GRPO [ 31 ]
TON [ 26 ]
DualMindVLM [ 13 ]
DR-BOLT (Ours)
Score / Length
ScienceQA (IID)
0.962/201.0
0.964/37.6
0.963/35.9
0.872/87.2
0.967/20.4
MathVision (transfer)
0.463/2498.2
0.527/1245.7
0.488/1505.8
0.302/446.0
0.606/1058.1
DynaMath (transfer)
0.769/1402.2
0.782/512.4
0.775/495.5
–
0.799/370.6
MMMU-Pro (transfer)
0.498/2006.4
0.540/892.6
0.519/747.3
–
0.581/385.7
Training Type
online RL / GRPO
online RL / GRPO
online RL / GRPO
online RL / GRPO
offline dual-ref BOLT
Table 1: Our reproductions of adaptive-thinking methods and GRPO. ScienceQA uses same-benchmark training; the other datasets measure transfer after ViRL39K training. Cells report Score/Length in words. Runtime includes candidate generation or online rollouts and all training stages; memory is peak per GPU. Speedup is relative to GRPO.
Dataset
Qwen3-Instruct
Qwen3-Thinking
Mix SFT
Reward SFT
Single-ref Fast
Single-ref Slow
DR-BOLT
Score / Length
ScienceQA (IID)
0.943/7.2
0.954/288.3
0.955/100.1
0.956/89.3
0.945/9.3
0.955/317.7
0.967/20.4
MathVision (transfer)
0.244/395.8
0.368/5775.3
0.406/3777.6
0.421/3802.9
0.301/474.7
0.439/5901.8
0.606/1058.1
DynaMath (transfer)
0.613/205.5
0.654/2696.7
0.730/1896.1
0.741/1675.0
0.627/241.4
0.752/2022.9
0.799/370.6
MMMU-Pro (transfer)
0.278/265.3
0.377/3082.3
0.472/1352.6
0.484/1105.9
0.412/443.3
0.408/2575.1
0.581/385.7
Slow Ratio (%)
0.0
99.8
67.2
60.4
1.9
97.4
24.1
Table 2: Matched internal comparisons. Cells report Score/Length in words; ScienceQA and cross-benchmark results use separately trained checkpoints. Slow ratio (%) aggregates only MathVision, DynaMath, and MMMU-Pro.
Figure 1: Difficulty-wise response-Length distributions. Each group contains the response-Length distributions for DR-BOLT , FAST-GRPO, and GRPO. DR-BOLT stays short on easy examples and increases reasoning Length on harder examples, while GRPO-style training remains much longer.
Figure 2: Score–Length dynamics on ScienceQA. Solid lines show accuracy and dashed lines show response Length.
Figure 3: Partition-based routing calibration. Bars show empirical fast rate and markers show predicted fast probability.
Method
ScienceQA
MathVision
DynaMath
MMMU-Pro
Slow ratio (%)
MR-DPO
0.964/31.8
0.568/1398.4
0.787/506.9
0.557/528.6
29.4
Cond-RW
0.960/48.6
0.513/1810.7
0.761/714.8
0.521/689.5
39.7
DR-BOLT
0.967/20.4
0.606/1058.1
0.799/370.6
0.581/385.7
24.1
Table 3: Controlled offline alternatives on identical candidate banks. Cells report score / response words. ScienceQA uses a separate checkpoint; slow ratio aggregates the three transfer benchmarks.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
x=(I,Q)
Multimodal input consisting of image I and text query Q .
y
Model response.
yf
Fast response, usually a concise direct answer.
ys=(c,a)
Slow response containing reasoning trace c and final answer a .
z∈{f,s}
Latent response mode: fast or slow.
πθ(y∣x)
Trainable VLM policy.
Appendix
Table 4: Notation used in the paper.
Figure 4: Cross-likelihood heatmaps on the main path. The left panel reports mean per-token log-likelihood; the right panel reports mean sequence log-likelihood. Diagonal entries correspond to in-mode likelihoods, while off-diagonal entries measure cross-mode likelihoods.
yf(Lˉ=6.21)
ys(Lˉ=276.64)
πf
−0.181/−1.192
−1.434/−274.380
πs
−4.514/−19.465
−0.190/−66.764
Appendix
Table 5: Cross-likelihood matrix on the main path. Each entry reports mean per-token log-likelihood / mean sequence log-likelihood in nats.
Direction
Mechanism
Length dependence
πf(ys)
Length-compounded decay. The fast policy assigns slightly lower probability to many reasoning tokens; over long traces this compounds into an exponentially small sequence ratio. Empirically, the per-token mismatch is amplified over Lˉ≈277 tokens.
Strong: r(L,Δseq)=−0.902 .
πs(yf)
Prefix bottleneck. The slow policy tends to start with a reasoning-format prefix such as <think> ; direct fast answers start with answer tokens instead. The mismatch is therefore concentrated near the first token rather than accumulated over length.
Weak: r(L,Δseq)=+0.226 .
Appendix
Table 6: Different mechanisms behind the two cross-mode likelihood gaps.
Figure 5: Cross-likelihood heatmaps on the ablation path. The slow policy still assigns low likelihood to direct fast answers, indicating that the prefix bottleneck is a structural property of the checkpoint. A small fraction of contaminated fast samples weaken the fast-pool purity.
Figure 6: Correction weights in the balanced setting α=q=0.5 . Both logwtheory and logwexact collapse to zero, so correlation is undefined. The single-reference diagnostic logwsingle still shows a strong bimodal structure.
Samples
Mean
Std
Min
Max
logwsingle∣yf(n=958)
+18.27
4.76
−7.96
+29.49
logwsingle∣ys(n=1000)
−207.62
91.96
−769.90
−101.92
Appendix
Table 7: Statistics of logwsingle=logπf(y∣x)−logπs(y∣x) in the balanced setting.
Figure 7: Correction weights on the main path with α=0.5,q=0.3 . Since α=q , logwtheory is non-constant and can be compared with logwexact .
Figure 8: Correction weights on the main path with α=0.5,q=0.7 . The agreement between wtheory and wexact is strongest in this setting.
Figure 9: Correction weights on the ablation path with α=0.5,q=0.3 . The equivalence degrades because the ablation fast pool contains reasoning-style contaminated samples.
Metric
main, q=0.3
main, q=0.7
ablation, q=0.3
Pooled r(wtheory,wexact)
0.9877
0.9902
0.9730
∣err∣>0.01
1.5%
1.2%
2.2%
Max ∣err∣
0.847
0.847
0.847
Fast margin std
4.76
4.76
7.87
Fast margin min
−7.96
−7.96
−66.38
Appendix
Table 8: Equivalence between exact correction and mode-wise correction. The main path shows strong agreement, while the ablation path degrades due to contaminated fast samples.
Figure 10: Implicit KL trajectory on the main path under (α,q)=(0.5,0.7) . The mode-wise and exact buckets remain nearly indistinguishable throughout training.
Figure 11: Training loss and gradient norm on the main path. The mode-wise and exact corrections have nearly identical optimization dynamics, while the full mode-wise implementation yields a smoother gradient profile.
Bucket
Step 25
Step 50
Step 100
Step 500
Final Loss
Grad Peak
Mode-wise
0.64
7.75
8.08
11.28
0.154
47.0
Exact
0.61
7.77
8.06
11.27
0.153
46.5
Single-ref
0.67
7.76
8.02
10.69
0.142
46.7
Mode-wise-full
0.62
7.23
8.36
10.39
0.151
28.8
Appendix
Table 9: Optimization dynamics on the main path. KL values are sequence-level implicit KL in nats.
Figure 12: Implicit KL trajectory on the ablation path. The relative differences between buckets remain similar to the main path, but the absolute optimization scale is much larger due to first-token mismatch.
Dataset
Setting
Metric
ScienceQA [ 15 ]
IID: train on ScienceQA, test on ScienceQA
Accuracy
MathVision [ 27 ]
Transfer: train on ViRL39K, test on MathVision
Accuracy
DynaMath [ 41 ]
Transfer: train on ViRL39K, test on DynaMath
Accuracy
MMMU-Pro [ 39 ]
Transfer: train on ViRL39K, test on MMMU-Pro
Accuracy
Appendix
Table 10: Datasets and evaluation protocol.
Hyperparameter
Value / Definition
Target fast prior α
0.5
Empirical fast sampling ratio q
0.7
Fast candidates Nf
7 per input
Slow candidates Ns
3 per input
BOLT temperature β
1.0
Format reward weight γ
0.1
Appendix
Table 11: Reward and correction hyperparameters used in the main experiments.
Item
Value
Framework
PyTorch
Fine-tuning method
LoRA weighted SFT
Optimizer
AdamW
Training steps
500 for white-box optimization diagnostics; full benchmark runs use the same budget across matched baselines
Per-device batch size
1
Gradient accumulation
8
Appendix
Table 12: Training configuration for reproduced internal experiments.
Correction
MathVision
DynaMath
MMMU-Pro
Slow Ratio
No κ
0.260/518.2
0.626/308.1
0.314/329.9
0.13
Exact κ
0.623/1023.2
0.804/325.7
0.594/382.0
22.0
Mode-wise κ
0.606/1058.1
0.799/370.6
0.581/385.7
24.1
Appendix
Table 13: Downstream ablation of behavior-reference correction under α=0.5,q=0.7 . Each benchmark cell reports Score/Length . Slow Ratio is reported in percentage points.
Change from default
Score
Length
Calibration slow ratio (%)
Default
0.967
20.4
51.4
α=0.3
0.968
25.5
71.1
α=0.7
0.964
15.2
31.2
q=0.3
0.965
21.4
53.8
q=0.5
0.966
20.8
52.2
β=0.5
0.966
17.8
45.1
Appendix
Table 14: One-factor ScienceQA sensitivity study. Length is mean response words. Slow ratio is measured on the 100-prompt repeated-decoding calibration subset used for Figure 3 , not the full ScienceQA test set.
λ
MathVision
MMMU-Pro
Cross-benchmark slow ratio (%)
0
0.609/1396.3
0.578/512.8
31.8
0.025
0.614/1208.6
0.585/445.2
27.7
0.05
0.606/1058.1
0.581/385.7
24.1
0.10
0.589/811.4
0.561/302.7
17.4
Appendix
Table 15: Shared length-penalty sweep after ViRL39K training. Cells report score / mean response words; slow ratio aggregates MathVision, DynaMath, and MMMU-Pro.
Method
ScienceQA
MathVision
DynaMath
MMMU-Pro
Slow ratio (%)
Two-policy router
0.958/64.7
0.401/2018.5
0.681/827.8
0.416/1054.6
28.9
Thinking prompt router
0.953/252.4
0.366/5108.7
0.652/2396.4
0.375/2748.9
87.6
DR-BOLT
0.967/20.4
0.606/1058.1
0.799/370.6
0.581/385.7
24.1
Appendix
Table 16: Router controls. ScienceQA and transfer results use separately trained systems. Cells report score / mean response words; slow ratio aggregates the three transfer benchmarks.
Dataset
Answer span
Parsing outcome
ScienceQA
B. New Hampshire
Leading-option rule; B matches ground truth B.
ScienceQA
B after a 168-word slow response
Reasoning is discarded; single-letter rule yields B.
ScienceQA
Neither.
No option is recovered; zero correctness and format reward.
MathVision
Unboxed arithmetic-sequence derivation
Last numeric literal is 60; matches ground truth 60.
MathVision
A for a multiple-choice item
Letter path yields A; matches the correct option.
MMMU-Pro
E, H
Multiple options; no unique answer, so zero correctness and format reward.
Appendix
Table 17: Representative parsing cases from generated responses.
Multimodal reasoning in vision-language models (VLMs) typically relies on a two-stage process: supervised fine-tuning (SFT) and reinforcement learning (RL). In standard SFT, all tokens contribute equally to the loss, even though reasoning data are inherently token-imbalanced. Long <think> traces overshadow short but task-critical <answer> segments, leading to verbose reasoning and inaccurate answers. We propose SCALe (Scheduled Curriculum Adaptive Loss), which explicitly separates supervision over reasoning and answer segments using dynamic, length-independent weighting. Unlike vanilla SFT, which overweights the <think> segment, SCALe-SFT gradually shifts the focus from <think> to <answer> throughout training via a cosine scheduling policy, encouraging concise and well-grounded reasoning. We evaluate SCALe across diverse benchmarks and architectures. Results show that SCALe consistently improves accuracy over vanilla SFT and matches the performance of the full two-phase SFT + GRPO pipeline while requiring only about one-seventh of the training time, making it a lightweight yet effective alternative. When combined with GRPO, SCALe achieves the best overall performance, highlighting its value both as a standalone method and as a strong foundation for reinforcement refinement.
The introduction of long chain-of-thought (CoT) reasoning and external tools has substantially improved the performance of multimodal large language models (MLLMs) on general and reasoning tasks. However, existing fast-slow reasoning paradigms mainly regulate textual reasoning length, while visual computation introduced by external tools or visual tokens is often treated as an auxiliary component. This separation prevents MLLMs from jointly optimizing visual perception and textual reasoning. To address this issue, we propose L2T (adaptive Look, flexible Thought), a unified reasoning framework that jointly optimizes the form and depth of computation. L2T first discretizes tool-produced visual states into visual tokens, bringing visual perception and textual reasoning into a shared autoregressive space. Based on this unified representation, L2T introduces a modality-aware routing reward to select appropriate reasoning modes and a difficulty-aware fast-slow strategy to adapt reasoning depth according to task complexity. Experiments on general perception and reasoning-intensive tasks demonstrate the effectiveness of L2T. Compared with explicit tool-calling approaches, L2T maintains competitive performance while achieving 1.7--3.6 times higher throughput.
Quanlin Chen, Xiuwei Chen, Kun Xiang +5
Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy +2
Supervised fine-tuning (SFT) on visual instruction data often improves perceptual capabilities in vision-language models (VLMs) while degrading reasoning performance, creating a persistent reasoning tax during post-training. We investigate whether this degradation is related to disrupted access to depth-wise representations, and find that even fixed cross-depth aggregation substantially restores reasoning, suggesting that preserved cross-depth access is an important missing factor in VLM fine-tuning. Building on this observation, we propose Input-Adaptive Depth Aggregation (IADA), a lightweight mechanism that makes cross-depth retrieval input-adaptive, modality-aware, and efficiently parameterized through a low-rank bottleneck. On Qwen3-VL-2B, IADA improves the average reasoning score by 9.5 points and the average perception score by 3.3 points over LoRA-only fine-tuning with only 0.14M additional parameters, with the strongest gains appearing in parameter-efficient low-rank settings.