Balanced Twin Weight Factorization for Efficient MLLMs
Organizations: Amazon
Abstract
Weight Factorization (WF) methods aim to reduce a model's latency, memory, or both. These typically decompose a pretrained model into two or more factors that approximate the original model weights with fewer parameters and floating point operations (FLOPs). However, translating fewer FLOPs into reduced latency is challenging for WF methods in practice due to overhead introduced by replacing a single large weight matrix with multiple smaller ones. To address this, we propose Balanced Twin Weight Factorization (BTWF), which uses the unique structure of factorized models to simultaneously lower latency and increase learning capacity. Specifically, BTWF operates in the shared basis WF setting, where two or more layers are decomposed into a shared basis matrix and a layer-specific mixing matrix. If these shared layers have the same input (i.e., are twins), then we need only perform this shared forward pass once, reducing inference time. However, twin layers are often of varying sizes, and methods from prior work tend to favor the larger layer. In contrast, our approach boosts performance by combining Factor Component Balancing (FCB), which ensures every shared member is well represented in the selected factors, with Block Contribution Balancing (BCB), which considers downstream model effects. Further, we also show that an overparameterized BTWF, where some layer's factors result in higher FLOPs than the original during training, can boost task performance without additional resources at test time. Experiments on six models over nine benchmarks show BTWF obtains a 2-6% gain over the state-of-the-art without expensive finetuning (1-3.5% gain with it) at comparable computational costs.
Figures & tables
| FLOPs Budget | 30% | 50% | ||||
|---|---|---|---|---|---|---|
| Method | Seen | Unseen | All | Seen | Unseen | All |
| Wanda [ 91 ] | 0.8 0.3 | 1.2 0.5 | 1.1 0.4 | 1.2 0.2 | 2.0 0.2 | 1.7 0.2 |
| Putri [ 17 ] | 2.4 1.8 | 3.2 2.7 | 3.0 2.3 | 1.8 0.9 | 2.7 1.5 | 2.4 1.3 |
| Wanda [ 91 ] +Putri [ 17 ] | 25.2 1.0 | 24.0 0.3 | 24.4 0.5 | 3.0 1.9 | 3.2 2.3 | 3.1 2.2 |
| PruneLLM [ 113 ] | 1.2 0.4 | 2.3 0.5 | 1.9 0.5 | 0.2 0.1 | 0.7 0.6 | 0.5 0.5 |
| D 2 Prune [ 110 ] | 24.4 1.0 | 23.2 0.4 | 23.6 0.6 | 17.5 0.1 | 20.6 0.5 | 19.6 0.3 |
| FLOPs Budget | 10% | 30% | 50% | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Weight Selector | Budget Allocator | Seen | Unseen | All | Seen | Unseen | All | Seen | Unseen | All |
| LLM-Pruner [ 64 ] | LLM-Pruner [ 64 ] | 35.4 0.2 | 41.8 0.1 | 39.6 0.1 | 3.7 0.6 | 5.1 0.3 | 4.6 0.3 | 1.3 0.4 | 0.6 0.1 | 0.8 0.2 |
| SlimLLM [ 33 ] | SlimLLM [ 33 ] | 46.0 0.2 | 51.9 0.3 | 49.9 0.2 | 8.5 3.4 | 10.4 2.7 | 9.8 2.9 | 0.2 0.1 | 4.7 0.5 | 3.2 0.4 |
| PruneLLM [ 113 ] | PruneLLM [ 113 ] | 43.7 0.1 | 47.7 0.2 | 46.4 0.2 | 6.4 0.4 | 9.7 0.6 | 8.6 0.2 | 0.2 0.1 | 0.3 0.0 | 0.3 0.0 |
| Putri [ 17 ] | Putri [ 17 ] | 27.1 3.4 | 30.1 0.9 | 29.1 1.7 | 9.5 4.1 | 10.0 5.6 | 9.8 5.1 | – | – | – |
| PORTA [ 49 ] | PORTA [ 49 ] | 13.7 1.2 | 19.3 2.3 | 17.4 1.9 | 1.5 0.2 | 3.2 0.1 | 2.7 0.1 | 1.0 0.2 | 2.4 0.4 | 1.9 0.3 |
| FLOPs Budget | 30% | 50% | ||||
|---|---|---|---|---|---|---|
| Method | Seen | Unseen | All | Seen | Unseen | All |
| Wanda [ 91 ] | 61.4 0.7 | 32.1 0.2 | 41.9 0.3 | 22.6 3.1 | 8.5 2.8 | 13.2 2.9 |
| D 2 Prune [ 110 ] | 53.7 2.1 | 28.8 0.8 | 37.1 1.2 | 43.9 0.7 | 23.6 0.4 | 30.4 0.3 |
| PORTA [ 49 ] | 63.4 0.8 | 33.7 0.5 | 43.6 0.6 | 21.9 5.3 | 9.8 3.9 | 13.8 4.4 |
| RCPU [ 39 ] | 63.2 0.5 | 34.6 0.4 | 44.2 0.4 | 43.4 0.9 | 23.1 1.2 | 29.9 1.1 |
| TRSP [ 25 ] | 52.3 2.0 | 31.5 1.0 | 38.4 1.2 | 36.2 0.4 | 20.5 1.5 | 25.8 0.9 |
| Method | 10% All | 30% All | 50% All |
|---|---|---|---|
| InternVL3-1B [ 124 ] | |||
| SVD-LLM [ 105 ] | 31.9 0.5 | 18.5 0.7 | 3.6 1.9 |
| Basis Sharing [ 102 ] | 29.0 0.3 | 18.5 0.3 | 4.8 0.1 |
| Cat [ 63 ] | 37.2 1.6 | 21.1 0.7 | 18.2 0.3 |
| BTWF (ours) | 38.2 0.5 | 23.1 0.7 | 18.5 0.1 |
| Qwen2-VL-2B [ 103 ] | |||
| Method | 10% Recon. | 30% Recon. | 50% Recon. |
|---|---|---|---|
| InternVL3-2B [ 124 ] | |||
| Cat [ 63 ] | 2.77 0.11 | 4.65 0.15 | 7.45 0.16 |
| FCB ( Sec. 3.1 ) | 2.20 0.09 | 3.82 0.13 | 6.37 0.14 |
| Gemma-4-E2B [ 30 ] | |||
| Cat [ 63 ] | 2.41 0.06 | 4.19 0.08 | 6.59 0.10 |
| FCB ( Sec. 3.1 ) | 2.13 0.05 | 3.78 0.08 | 6.07 0.09 |
| Method | 10% All | 30% All | 50% All |
|---|---|---|---|
| BKV [ 67 ] | 37.5 0.7 | 23.5 0.2 | 17.4 0.3 |
| FCB ( Sec. 3.1 ) | 38.2 0.5 | 23.1 0.7 | 18.5 0.1 |
| Method | 10% All | 30% All | 50% All |
|---|---|---|---|
| Cat w/Skip [ 63 ] | 54.5 0.7 | 36.3 0.8 | 22.0 0.4 |
| BTWF (ours) w/ Skip | 55.1 0.4 | 39.3 0.5 | 23.3 0.3 |
| BTWF w/ OG & Skip | 55.4 0.2 | 39.7 0.6 | 23.8 0.1 |
| Method | 50% All | 70% All |
|---|---|---|
| GETA [ 78 ] | 31.4 0.5 | 22.0 3.2 |
| UniQL [ 16 ] | 33.2 1.6 | 22.4 1.5 |
| BTWF (ours) w/ GETA | 36.5 0.6 | 31.4 0.6 |
| Method | 50% All | 70% All |
|---|---|---|
| CoMP [ 107 ] | 18.0 3.5 | 8.5 2.6 |
| CoMP w/NLL | 23.1 2.1 | 10.8 2.8 |
| BTWF (ours) w/ CoMP | 25.1 0.4 | 19.6 0.1 |
| Memory | Latency | Speedup | ||
|---|---|---|---|---|
| Method | 50% All | (GB) | (ms) | vs. Dense |
| D 2 Prune [ 110 ] | 30.4 0.3 | 2.89 | 88.5 0.3 | 1.53 |
| RCPU [ 39 ] | 29.9 1.1 | 2.89 | 88.5 0.5 | 1.53 |
| AdaSVD [ 57 ] | 32.0 0.6 | 2.98 | 103.8 0.4 | 1.30 |
| Basis Sharing [ 102 ] | 24.7 0.7 | 2.84 | 103.8 0.1 | 1.30 |
| Cat [ 63 ] | 31.9 0.8 | 2.86 | 102.3 0.3 | 1.32 |
| FLOPs Budget | 10% | 30% | 50% | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Row weights | Seen | Unseen | All | Seen | Unseen | All | Seen | Unseen | All |
| BTWF w/o BCB | 48.3 0.2 | 46.3 0.9 | 47.0 0.6 | 33.6 0.4 | 29.6 0.9 | 30.9 0.6 | 23.4 0.6 | 20.7 0.5 | 21.6 0.4 |
| w/ FWSVD [ 41 ] | 48.2 0.2 | 46.3 0.3 | 46.9 0.2 | 33.1 0.8 | 28.7 0.6 | 30.1 0.5 | 23.2 0.1 | 21.6 0.3 | 22.1 0.2 |
| w/ MoDeGPT [ 60 ] | 48.3 0.4 | 46.4 1.0 | 47.0 0.6 | 33.8 0.8 | 30.3 1.4 | 31.4 1.1 | 23.5 0.6 | 20.7 1.0 | 21.7 0.9 |
| BTWF (ours) | 48.7 0.2 | 46.9 0.5 | 47.5 0.3 | 34.4 1.0 | 31.2 0.9 | 32.2 0.9 | 23.3 0.7 | 21.6 0.5 | 22.2 0.5 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| FLOPs Budget | 10% | 30% | 50% | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | Recovery Strategy | Seen | Unseen | All | Seen | Unseen | All | Seen | Unseen | All |
| BTWF (ours) | LoRA | 70.1 0.4 | 46.2 2.3 | 54.2 1.4 | 62.5 0.7 | 33.4 0.8 | 43.1 0.8 | 53.8 0.1 | 27.0 0.5 | 36.0 0.3 |
| Staged LoRA | 68.4 1.0 | 48.4 0.8 | 55.1 0.4 | 60.3 0.5 | 32.3 0.5 | 41.6 0.5 | 48.4 1.6 | 27.0 0.3 | 34.2 0.4 | |
| SFT | 70.5 0.6 | 48.7 0.5 | 56.0 0.5 | 61.9 1.9 | 35.2 0.9 | 44.1 0.2 | 50.1 0.6 | 27.1 0.3 | 34.8 0.2 | |
| Last 3 Blocks | 50.2 1.3 | 42.9 2.8 | 45.3 2.0 | 36.1 2.0 | 29.4 1.1 | 31.6 0.4 | 29.3 1.1 | 24.4 2.0 | 26.0 1.7 | |
| KD | 65.4 0.6 | 46.4 0.7 | 52.7 0.5 | 57.5 0.9 | 34.6 0.5 | 42.2 0.6 | 47.2 0.3 | 28.1 0.1 | 34.4 0.1 | |
| FLOPs Budget | 30% | 50% | ||||||
|---|---|---|---|---|---|---|---|---|
| Memory | Latency | Speedup | Memory | Latency | Speedup | |||
| Method | All | (GB) | (ms) | vs. Dense | All | (GB) | (ms) | vs. Dense |
| InternVL3-2B [ 124 ] (dense: 58.7 All, 4.34 GB, 135.2 ms) | ||||||||
| SVD-LLM [ 105 ] | 24.3 0.5 | 3.50 | 121.3 | 1.11 | 14.9 2.5 | 2.89 | 103.4 | 1.31 |
| Basis Sharing [ 102 ] | 22.9 0.4 | 3.42 | 123.2 | 1.10 | 13.8 5.7 | 2.84 | 103.8 | 1.30 |
| Cat [ 63 ] | 28.1 1.1 | 3.48 | 121.0 | 1.12 | 20.3 0.4 | 2.86 | 102.3 | 1.32 |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 30% | FLAP [ 4 ] | 3.2 0.2 | 0.8 0.7 | 15.5 2.7 | 27.8 3.0 | 0.1 0.0 | 0.0 0.0 | 1.3 0.9 | 22.1 0.8 | 19.8 0.1 | 10.1 0.6 |
| Wanda [ 91 ] | 0.4 0.1 | 0.1 0.0 | 2.1 1.1 | 2.2 1.0 | 0.2 0.1 | 0.0 0.1 | 0.1 0.0 | 1.8 0.9 | 2.9 1.0 | 1.1 0.4 | |
| Putri [ 17 ] | 1.6 1.6 | 1.2 2.0 | 4.4 4.7 | 3.2 4.0 | 0.3 0.3 | 0.0 0.0 | 2.2 3.7 | 6.7 7.8 | 7.1 5.6 | 3.0 2.3 | |
| Wanda [ 91 ] +Putri [ 17 ] | 27.7 1.3 | 21.8 0.4 | 26.2 1.3 | 41.3 0.6 | 7.3 0.5 | 7.5 0.3 | 32.8 0.7 | 27.3 0.4 | 27.5 0.4 | 24.4 0.5 | |
| AMP [ 71 ] | 0.6 0.2 | 0.0 0.0 | 1.9 0.5 | 3.4 0.2 | 0.2 0.1 | 0.0 0.0 | 0.1 0.0 | 0.8 0.3 | 1.9 0.8 | 1.0 0.1 | |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 10% | LLM-Pruner [ 64 ] | 20.7 0.5 | 40.5 0.3 | 44.9 0.6 | 41.5 0.1 | 36.1 0.3 | 43.6 1.0 | 58.8 0.3 | 36.0 0.5 | 34.7 0.3 | 39.6 0.1 |
| SlimLLM [ 33 ] | 26.5 0.6 | 48.5 0.2 | 63.0 0.3 | 44.0 0.2 | 61.1 0.3 | 50.8 1.5 | 69.4 0.1 | 41.3 0.3 | 44.9 0.5 | 49.9 0.2 | |
| PruneLLM [ 113 ] | 29.5 0.3 | 43.3 0.2 | 58.3 0.5 | 35.3 0.7 | 64.2 0.4 | 43.7 0.4 | 66.3 0.3 | 36.1 0.6 | 40.5 0.4 | 46.4 0.2 | |
| Putri [ 17 ] | 15.9 6.9 | 29.8 3.0 | 35.6 4.5 | 39.1 2.5 | 31.2 1.4 | 10.8 3.7 | 44.9 3.2 | 28.2 1.9 | 26.4 1.9 | 29.1 1.7 | |
| TRSP [ 25 ] | 6.7 1.9 | 32.8 2.2 | 37.6 4.6 | 42.5 1.0 | 45.0 4.7 | 30.7 2.3 | 43.5 2.9 | 34.2 1.0 | 26.1 2.1 | 33.2 2.2 | |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| (a) Source method end to end | |||||||||||
| 30% | AMP [ 71 ] | 69.9 0.2 | 52.5 6.2 | 62.7 0.1 | 39.8 5.0 | 9.4 0.3 | 11.6 1.5 | 51.9 5.3 | 31.5 0.7 | 34.2 0.9 | 40.4 1.6 |
| LLM-Pruner [ 64 ] | 50.0 1.7 | 55.3 1.1 | 42.9 4.4 | 32.3 3.2 | 20.0 2.3 | 17.9 0.3 | 58.5 1.8 | 29.8 0.9 | 29.7 1.5 | 37.4 1.1 | |
| SlimLLM [ 33 ] | 57.0 0.3 | 58.0 0.3 | 56.7 0.5 | 37.7 3.0 | 25.0 0.9 | 29.6 10.8 | 61.2 0.2 | 31.4 1.0 | 33.3 0.8 | 43.3 0.7 | |
| SVD-LLM [ 105 ] | 48.7 4.5 | 53.0 1.4 | 51.9 0.8 | 38.1 2.0 | 13.2 1.6 | 14.9 0.8 | 53.9 1.3 | 31.0 1.3 | 33.4 1.6 | 37.6 0.9 | |
| Seen | Unseen | ||||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 10% | InternVL3-1B | ||||||||||
| SVD-LLM [ 105 ] | 26.4 0.4 | 34.7 1.4 | 38.3 0.5 | 40.0 0.8 | 19.3 0.3 | 24.4 7.4 | 45.4 2.0 | 29.2 0.9 | 29.2 1.3 | 31.9 0.5 | |
| Basis Sharing [ 102 ] | 27.4 1.1 | 32.8 0.6 | 34.1 0.3 | 40.4 0.6 | 14.8 1.4 | 11.1 0.4 | 44.7 0.6 | 29.2 0.3 | 26.6 0.8 | 29.0 0.3 | |
| Cat [ 63 ] | 32.0 0.7 | 40.2 0.6 | 46.8 1.3 | 42.1 0.4 | 26.8 0.5 | 30.7 12.7 | 53.2 0.4 | 31.7 0.3 | 31.2 0.6 | 37.2 1.6 | |
| BTWF (ours) | 32.0 0.5 | 40.8 0.4 | 47.7 0.7 | 42.4 0.6 | 26.5 0.4 | 36.5 6.1 | 53.5 0.3 | 32.1 0.6 | 32.7 0.5 | 38.2 0.5 | |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 10% | BKV [ 67 ] | 32.2 0.2 | 40.7 0.2 | 47.5 1.4 | 43.4 0.5 | 25.8 0.3 | 31.5 9.1 | 53.3 0.5 | 31.6 0.2 | 31.6 1.2 | 37.5 0.7 |
| FCB (ours) | 32.0 0.5 | 40.8 0.4 | 47.7 0.7 | 42.4 0.6 | 26.5 0.4 | 36.5 6.1 | 53.5 0.3 | 32.1 0.6 | 32.7 0.5 | 38.2 0.5 | |
| 30% | BKV [ 67 ] | 28.6 0.2 | 24.4 0.9 | 26.8 0.9 | 37.7 0.6 | 5.4 0.2 | 4.2 0.4 | 34.0 0.7 | 26.4 0.3 | 24.4 0.8 | 23.5 0.2 |
| FCB (ours) | 27.3 0.6 | 22.4 2.6 | 26.4 0.1 | 36.7 0.1 | 5.8 0.3 | 4.7 0.8 | 33.1 1.7 | 25.9 0.7 | 25.5 1.2 | 23.1 0.7 | |
| 50% | BKV [ 67 ] | 11.7 0.6 | 11.6 1.2 | 20.1 0.2 | 33.4 0.7 | 4.5 0.3 | 4.0 0.6 | 24.2 1.7 | 25.6 0.4 | 21.4 0.1 | 17.4 0.3 |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 10% | Cat w/Skip [ 63 ] | 37.2 1.2 | 55.5 0.1 | 70.4 0.5 | 51.1 1.0 | 68.9 0.4 | 46.8 3.4 | 73.8 0.1 | 43.4 0.4 | 43.7 0.7 | 54.5 0.7 |
| BTWF (ours) | 36.1 0.4 | 48.6 0.4 | 61.5 0.3 | 47.1 0.5 | 47.7 1.7 | 43.4 2.3 | 64.1 0.1 | 37.7 0.0 | 41.6 1.2 | 47.5 0.3 | |
| BTWF w/Skip | 37.2 0.3 | 56.2 0.4 | 71.8 0.2 | 52.1 0.5 | 70.0 0.5 | 46.6 1.6 | 74.0 0.3 | 43.1 0.4 | 44.7 0.9 | 55.1 0.4 | |
| BTWF w/OG w/Skip | 37.2 0.5 | 56.1 0.1 | 71.3 0.4 | 50.6 0.4 | 70.2 0.5 | 51.3 1.1 | 73.9 0.2 | 43.0 0.7 | 44.9 0.7 | 55.4 0.2 | |
| 30% | Cat w/Skip [ 63 ] | 32.7 1.1 | 38.2 0.3 | 41.7 1.5 | 41.5 0.6 | 17.3 0.6 | 40.7 4.1 | 49.9 0.4 | 31.2 0.9 | 33.5 0.7 | 36.3 0.8 |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 50% | GETA [ 78 ] | 60.9 1.0 | 39.9 0.2 | 45.6 0.8 | 28.6 3.6 | 5.7 0.1 | 8.6 0.8 | 40.1 0.3 | 25.6 0.5 | 27.9 0.8 | 31.4 0.5 |
| UniQL [ 16 ] | 74.9 1.0 | 44.1 1.3 | 40.4 4.5 | 23.9 10.3 | 5.1 0.1 | 13.1 4.4 | 45.1 0.2 | 24.9 0.8 | 27.4 2.6 | 33.2 1.6 | |
| BTWF (ours) w/ GETA | 79.4 0.5 | 48.2 0.6 | 48.9 1.6 | 35.1 2.4 | 5.5 0.3 | 10.6 0.3 | 46.0 0.6 | 26.8 0.8 | 28.1 0.9 | 36.5 0.6 | |
| 70% | GETA [ 78 ] | 41.4 6.8 | 36.5 1.3 | 24.9 11.0 | 11.8 6.0 | 4.2 0.2 | 7.1 0.7 | 34.7 1.1 | 19.4 2.4 | 17.8 5.4 | 22.0 3.2 |
| UniQL [ 16 ] | 49.2 0.6 | 34.5 1.8 | 18.7 0.1 | 12.2 3.6 | 3.8 1.2 | 8.2 1.1 | 35.1 1.2 | 21.8 3.4 | 18.4 3.6 | 22.4 1.5 | |
| FLOPs | Seen | Unseen | |||||||||
| Budget | Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| 50% | CoMP [ 107 ] | 6.6 2.6 | 18.9 1.6 | 12.3 2.7 | 20.8 7.5 | 6.9 3.0 | 33.7 14.3 | 24.6 1.4 | 17.6 2.7 | 20.8 2.0 | 18.0 3.5 |
| CoMP w/ NLL | 8.7 4.0 | 20.8 2.8 | 20.6 7.0 | 32.3 7.9 | 11.6 3.4 | 36.6 5.2 | 27.6 5.2 | 23.7 2.6 | 26.2 3.3 | 23.1 2.1 | |
| BTWF (ours) w/ CoMP | 30.6 0.8 | 23.9 0.0 | 26.6 0.7 | 38.5 1.2 | 8.0 0.1 | 9.7 2.4 | 32.3 0.3 | 27.6 0.5 | 28.4 0.6 | 25.1 0.4 | |
| 70% | CoMP [ 107 ] | 3.7 2.3 | 0.2 0.2 | 17.3 3.7 | 19.4 12.5 | 0.2 0.2 | 0.0 0.0 | 0.2 0.1 | 17.1 6.0 | 18.1 1.7 | 8.5 2.6 |
| CoMP w/ NLL | 6.3 5.1 | 5.9 0.9 | 17.0 3.7 | 21.6 10.2 | 1.2 0.7 | 0.6 0.6 | 7.6 2.1 | 18.1 6.6 | 18.4 2.7 | 10.8 2.8 | |
| Dataset | Media | Train | Val | Test (source) | |
|---|---|---|---|---|---|
| Seen datasets | CLEVRER [ 115 ] | video | 61,064 | 14,235 | 10,000 (subset of official val) |
| GQA [ 44 ] | image | 65,093 | 25,803 | 12,578 (testdev-balanced) | |
| NExT-QA [ 109 ] | video | 7,864 | 896 | 4,221 (video-level re-split) | |
| Unseen datasets | Perception Test [ 74 ] | video | – | – | 19,140 (val) |
| VQAv2 [ 31 ] | image | – | – | 10,000 (subset of val) | |
| VizWiz [ 36 ] | image | – | – | 4,319 (val) |
| Seen | Unseen | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | CLEVRER | GQA | NExT-QA | Perc.Test | TextVQA | VizWiz | VQAv2 | VideoMME | LVB | All |
| InternVL3-1B [ 124 ] | 36.1 | 57.2 | 70.9 | 51.7 | 72.5 | 49.3 | 75.1 | 40.7 | 41.3 | 55.0 |
| InternVL3-2B [ 124 ] | 36.5 | 59.3 | 75.3 | 55.1 | 77.3 | 53.2 | 78.2 | 46.2 | 46.8 | 58.7 |
| Qwen2-VL-2B [ 103 ] | 38.8 | 60.0 | 83.9 | 47.7 | 70.2 | 64.3 | 78.7 | 45.0 | 43.4 | 59.1 |
| Qwen3-VL-2B [ 10 ] | 35.6 | 58.6 | 69.0 | 51.0 | 70.6 | 70.6 | 77.1 | 46.3 | 44.2 | 58.1 |
| Qwen3.5-2B [ 79 ] | 39.2 | 56.9 | 63.4 | 50.2 | 69.0 | 64.0 | 77.9 | 47.8 | 45.7 | 57.1 |
| Model | Seen | Unseen | All |
|---|---|---|---|
| InternVL3-1B [ 124 ] | 54.7 | 55.1 | 55.0 |
| InternVL3-2B [ 124 ] | 57.0 | 59.5 | 58.7 |
| Qwen2-VL-2B [ 103 ] | 60.9 | 58.2 | 59.1 |
| Qwen3-VL-2B [ 10 ] | 54.4 | 60.0 | 58.1 |
| Qwen3.5-2B [ 79 ] | 53.1 | 59.1 | 57.1 |
| Gemma4 E2B [ 30 ] | 37.5 | 48.1 | 44.6 |
LLM Inference Acceleration
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
See all 1,239 papers