Efficient Latent Reasoning for Vision-Language Model via Dynamics-Aware Quantization and Distillation
Organizations: Beihang University
Abstract
Efficient chain-of-thought (CoT) reasoning in vision-language models (VLMs) requires reducing both intermediate reasoning steps and their execution cost. Latent CoT methods compress textual rationales into continuous thoughts, yet their full-precision (FP) models retain substantial storage and computation costs. Lowering numerical precision could reduce these remaining costs, yet direct quantization severely degrades reasoning accuracy, particularly with 4-bit weights and activations. Operator and perturbation analyses reveal key sensitivity and direction-dependent predictive effects of continuous-thought errors, motivating preservation of information retrieval and subsequent reasoning beyond reconstruction accuracy. We therefore introduce QL-VLM, a dynamics-aware Quantization framework for efficient Latent reasoning in Vision-Language Models (VLMs). First, Dynamics-Guided Quantization (DGQ) uses nonuniform codebooks that preserve rotary-key structure, mitigating quantization-induced distortion in vision-language information retrieval. Second, Dynamics-Preserving Distillation (DPD) aligns continuous thoughts, latent-query readouts, and predictions with a full-precision teacher, maintaining predictive consistency through subsequent low-precision reasoning. On ScienceQA-IMG with Qwen2-VL-2B-Instruct, 4-bit QL-VLM achieves 90.43% accuracy and improves baseline by 40.33%, with 16.64x fewer bit-operations and 6.40x lower measured latency than explicit CoT.
Figures & tables
| Method | #B | ScienceQA | A-OKVQA | M 3 CoT |
|---|---|---|---|---|
| Qwen2-VL-2B-Instruct | ||||
| Explicit CoT | 16 | 70.50 | 71.32 | 42.52 |
| Latent CoT | 16 | 79.97 | 79.91 | 44.16 |
| Q-VLM [ 25 ] | 4 | - | - | - |
| LLM-QAT [ 13 ] | 4 | - | - | - |
| Baseline | 4 | 50.10 | 34.10 | 20.40 |
| Method | #B | MMStar | MMMU-MC |
|---|---|---|---|
| Qwen2-VL-2B-Instruct | |||
| Explicit CoT | 16 | 40.50 | 31.20 |
| Latent CoT | 16 | 43.28 | 32.24 |
| Baseline | 4 | 30.11 | 24.19 |
| QL-VLM | 4 | 44.27 | 31.18 |
| Baseline | 8 | 37.26 | 28.79 |
| Configuration | Correspondence | Acc. (%) |
| Cartesian baseline | Eqs. ( 5 )-( 6 ) | 50.10 |
| Cartesian K + DPD | Eqs. ( 14 )-( 15 ) | 52.69 |
| + Polar encoding | Eq. ( 10 ), fixed grids | 60.20 |
| + Nonuniform codebooks | Learn | 64.28 |
| + Axis grouping | Axis index in | 70.11 |
| + Source grouping | Source index in | 72.69 |
| Variant | 2B | 7B | |||
|---|---|---|---|---|---|
| Cartesian baseline | 0 | 0 | 1 | 50.10 | 63.31 |
| DGQ only | 0 | 0 | 1 | 55.89 | 68.05 |
| 0.05 | 0 | 1 | 69.72 | 79.79 | |
| 0 | 0.05 | 1 | 72.74 | 82.13 | |
| ( QL-VLM ) | 0.05 | 0.05 | 1 | 90.43 | 93.78 |
| Latent CoT | Baseline | QL-VLM | Steps | CR | |
|---|---|---|---|---|---|
| 2 | 63.21 | 18.17 | 45.15 | 6.0 + 2 | 7.04 |
| 4 | 68.12 | 22.46 | 53.61 | 6.03 + 4 | 5.61 |
| 6 | 79.97 | 50.10 | 90.43 | 5.86 + 6 | 4.75 |
| 8 | 81.26 | 55.23 | 91.01 | 5.93 + 8 | 4.04 |
| 12 | 82.94 | 56.79 | 91.65 | 5.86 + 12 | 3.15 |