Efficient chain-of-thought (CoT) reasoning in vision-language models (VLMs) requires reducing both intermediate reasoning steps and their execution cost. Latent CoT methods compress textual rationales into continuous thoughts, yet their full-precision (FP) models retain substantial storage and computation costs. Lowering numerical precision could reduce these remaining costs, yet direct quantization severely degrades reasoning accuracy, particularly with 4-bit weights and activations. Operator and perturbation analyses reveal key sensitivity and direction-dependent predictive effects of continuous-thought errors, motivating preservation of information retrieval and subsequent reasoning beyond reconstruction accuracy. We therefore introduce QL-VLM, a dynamics-aware Quantization framework for efficient Latent reasoning in Vision-Language Models (VLMs). First, Dynamics-Guided Quantization (DGQ) uses nonuniform codebooks that preserve rotary-key structure, mitigating quantization-induced distortion in vision-language information retrieval. Second, Dynamics-Preserving Distillation (DPD) aligns continuous thoughts, latent-query readouts, and predictions with a full-precision teacher, maintaining predictive consistency through subsequent low-precision reasoning. On ScienceQA-IMG with Qwen2-VL-2B-Instruct, 4-bit QL-VLM achieves 90.43% accuracy and improves baseline by 40.33%, with 16.64x fewer bit-operations and 6.40x lower measured latency than explicit CoT.
Figures & tables
Figure 1: Accuracy-efficiency trade-off on ScienceQA-IMG with Qwen2-VL-2B-Instruct. Bubble area denotes inference cost (TBitOps per answer), and arrows show gains over matched-precision baselines. The inset compares generated tokens (solid) and latent steps (hatched).
Figure 2: Overview of QL-VLM. (a) Dynamics-Preserving Distillation aligns the quantized student’s continuous thoughts, latent-query readouts, and predictions with a frozen full-precision teacher. (b) Compared with Cartesian key quantization, Dynamics-Guided Quantization reconstructs rotary key pairs using grouped nonuniform codebooks and a calibration-based predictive metric at the same bit width.
Figure 3: Quantization sensitivity of Qwen2-VL-2B-Instruct on ScienceQA. (a) Module-level comparison between FP and 4-bit Linear layers, attention operands, or both. (b) Activation-level comparison with 4-bit Linear layers: the precision of Q (query), K (key), V (value), or P (attention probability) is individually increased from 4 to 8 bits, while the remainder stays at 4 bits.
Figure 4: Prediction sensitivity of Qwen2-VL-2B trained with CODI on ScienceQA. (a) KL between unperturbed/perturbed predictions versus sample-specific quadratic estimates. (b) Spearman correlation with measured KL for local/fixed estimates, computed separately at each perturbation magnitude. Shading indicates 95% bootstrap confidence intervals over 8 held-out examples.
Method
#B
ScienceQA
A-OKVQA
M 3 CoT
Qwen2-VL-2B-Instruct
Explicit CoT
16
70.50
71.32
42.52
Latent CoT
16
79.97
79.91
44.16
Q-VLM [ 25 ]
4
-
-
-
LLM-QAT [ 13 ]
4
-
-
-
Baseline
4
50.10
34.10
20.40
Table 1: Generated-answer accuracy (%) on supervised vision-language CoT reasoning. Each dataset uses its own FP-trained initialization (gray-shadowed rows). Explicit CoT denotes CoT-SFT on ScienceQA-IMG and A-OKVQA, and Multimodal-CoT on M 3 CoT. #B specifies the corresponding bit width defined in Sec. 4.1 . The best results are in bold . “-” indicates failure to converge under the specified training budget.
Method
#B
MMStar
MMMU-MC
Qwen2-VL-2B-Instruct
Explicit CoT
16
40.50
31.20
Latent CoT
16
43.28
32.24
Baseline
4
30.11
24.19
QL-VLM
4
44.27
31.18
Baseline
8
37.26
28.79
Table 2: Zero-shot accuracy (%) with a fixed ScienceQA-IMG-trained source checkpoint. #B specifies the corresponding bit width defined in Sec. 4.1 . The best results are in bold .
Figure 5: Inference efficiency on ScienceQA-IMG with (a) Qwen2-VL-2B-Instruct and (b) Qwen2.5-VL-7B-Instruct, measured on an NVIDIA RTX PRO 6000 GPU. Bars show inference time; annotations report generated tokens and continuous thoughts. Speedups are relative to explicit CoT within each model.
Configuration
Correspondence
Acc. (%)
Cartesian baseline
Eqs. ( 5 )-( 6 )
50.10
Cartesian K + DPD
Eqs. ( 14 )-( 15 )
52.69
+ Polar encoding
Eq. ( 10 ), fixed grids
60.20
+ Nonuniform codebooks
Learn ρg,μ,ψg,ν
64.28
+ Axis grouping
Axis index a in g
70.11
+ Source grouping
Source index ξ in g
72.69
Table 3: Progressive DGQ analysis on ScienceQA-IMG at 4-bit Qwen2-VL-2B-Instruct. After enabling DPD, each row cumulatively adds the indicated design choice. Acc. denotes accuracy.
Variant
λroll
λread
λKD
2B
7B
Cartesian baseline
0
0
1
50.10
63.31
DGQ only
0
0
1
55.89
68.05
+Lroll
0.05
0
1
69.72
79.79
+Lread
0
0.05
1
72.74
82.13
+Lroll+Lread ( QL-VLM )
0.05
0.05
1
90.43
93.78
Table 4: DPD ablations on ScienceQA-IMG at 4-bit setting. DGQ is enabled except in the Cartesian baseline. 2B (Qwen2-VL-2B-Instruct) and 7B (Qwen2.5-VL-7B-Instruct) report accuracy (%).
Figure 6: Loss-weight sensitivity on ScienceQA-IMG at 4-bit. Each curve varies one coefficient while fixing the others. The horizontal axis shows the ratio to its default: λkey=0.001 , λroll=0.05 , and λread=0.05 . Dashed lines denote default.
T
Latent CoT
Baseline
QL-VLM
Steps
CR
2
63.21
18.17
45.15
6.0 + 2
7.04
4
68.12
22.46
53.61
6.03 + 4
5.61
6
79.97
50.10
90.43
5.86 + 6
4.75
8
81.26
55.23
91.01
5.93 + 8
4.04
12
82.94
56.79
91.65
5.86 + 12
3.15
Table 5: Latent-budget trade-off on ScienceQA with Qwen2-VL-2B-Instruct. Steps and CR denote decoding length and compression ratio of QL-VLM relative to explicit CoT.
Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3× throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment. Code and data are available at: https://github.com/ForeverBlue816/GRACE.
Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Qualcomm AI Research, Amsterdam, the Netherlands +1
Vision-Language Models often struggle with complex visual reasoning due to the visual information loss in textual CoT. Existing methods either add the cost of tool calls or rely on localized patch-based embeddings that are insufficient to extract semantics in multi-step reasoning. We propose "Decompose, Look, and Reason" (DLR), a reinforced latent reasoning framework that dynamically decomposes queries into textual premises, extracts premise-conditioned continuous visual latents, and deduces answers through grounded rationales. We introduce a three-stage training pipeline and propose a novel Spherical Gaussian Latent Policy, to enable effective exploration in the latent space. Extensive experiments on vision-centric benchmarks show that DLR consistently outperforms strong baselines, including text-only, interleaved multimodal CoT, and latent reasoning methods, while providing superior stepwise interpretability.
Mengdan Zhu, Senhao Cheng, Liang Zhao
Emory University · University of Michigan, Ann Arbor
Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-language models (VLMs) for multimodal understanding. However, billion-parameter VLMs incur substantial memory and computational costs that hinder deployment in resource-constrained settings. Post-training quantization (PTQ) compresses models and accelerates inference without retraining, yet remains underexplored for VLMs. We identify two intrinsic VLM activation properties in PTQ: (1) visual over-representation, where vision tokens are excessive and often redundant, and (2) the modality gap separating text and vision tokens in the latent feature space. Prior methods largely overlook these properties, leading to quantization performance degradation. To address this mismatch, we propose SalQ-VLM, an importance-aware PTQ framework that prioritizes salient tokens and suppresses redundant vision tokens during calibration. We derive a gradient-driven importance factor that captures token-level importance variance and is theoretically grounded in the relationship among loss perturbation, activation errors, and output gradients. SalQ-VLM obtains this factor through a single lightweight block-wise gradient-caching pass and incorporates it into the layer-wise reconstruction objective. Because SalQ-VLM modifies only calibration, it adds no inference-time operations and remains compatible with existing high-performance kernels. Extensive evaluations across benchmarks and backbones show that SalQ-VLM consistently outperforms strong PTQ baselines, especially under ultra-low-bit quantization. Notably, it improves MME-RealWorld accuracy by 16.45% under INT2g128 quantization.
Yufei Xue, Yushi Huang, Lunjie Zhu +2
Institute of Artificial Intelligence (TeleAI), China Telecom · The Hong Kong University of Science and Technology