Weight Factorization (WF) methods aim to reduce a model's latency, memory, or both. These typically decompose a pretrained model into two or more factors that approximate the original model weights with fewer parameters and floating point operations (FLOPs). However, translating fewer FLOPs into reduced latency is challenging for WF methods in practice due to overhead introduced by replacing a single large weight matrix with multiple smaller ones. To address this, we propose Balanced Twin Weight Factorization (BTWF), which uses the unique structure of factorized models to simultaneously lower latency and increase learning capacity. Specifically, BTWF operates in the shared basis WF setting, where two or more layers are decomposed into a shared basis matrix and a layer-specific mixing matrix. If these shared layers have the same input (i.e., are twins), then we need only perform this shared forward pass once, reducing inference time. However, twin layers are often of varying sizes, and methods from prior work tend to favor the larger layer. In contrast, our approach boosts performance by combining Factor Component Balancing (FCB), which ensures every shared member is well represented in the selected factors, with Block Contribution Balancing (BCB), which considers downstream model effects. Further, we also show that an overparameterized BTWF, where some layer's factors result in higher FLOPs than the original during training, can boost task performance without additional resources at test time. Experiments on six models over nine benchmarks show BTWF obtains a 2-6% gain over the state-of-the-art without expensive finetuning (1-3.5% gain with it) at comparable computational costs.
Figures & tables
Figure 1: Illustrative example selecting a shared rank-two subspace. Blue and orange denote whitened twin members W~k=WkL , whereas black dots mark selected directions. (a) Prior work directly decomposes twin member’s concatenation [ 102 , 3 , 63 ] , resulting in equal weighting of each channel. Thus, the larger member with more channels has a higher likelihood to have their factors selected (selecting {e1,e2} in this example). (b) our BTWF approach addresses the large-member bias via per-member norm balancing with FCB. Then BCB improves dense model fidelity by reweighting channels according to their sensitivity to the containing block’s output (resulting in {e1,e3} being chosen).
Figure 2: Example implementation of our BTWF shared factorization on an InternVL3-2B [ 124 ] backbone. (a) Shows the overall structure of the vision-language model, which remains unchanged. Instead, we factorize layers using Factor Component Balancing (FCB) described in Sec. 3.1 and Block Contribution Balancing (BCB) discussed in Sec. 3.2 . These can either use Overparameterized Groups (OG) from Sec. 3.3.1 that use large ranks to add model capacity (applied to the vision encoder in this example) or use a small rank to reduce memory and latency on twin groups within a transformer block. (b) shows attention twin groups where query, key, and value projections reuse za=Baxa , computed once, with separate mixing factors Aq , Ak , and Av . The attention operation is general and permits different query and key/value widths. (c) illustrates that gate and up projections in a SwiGLU MLP similarly can reuse zm=Bmxm .
FLOPs Budget
30%
50%
Method
Seen
Unseen
All
Seen
Unseen
All
Wanda [ 91 ]
0.8 ± 0.3
1.2 ± 0.5
1.1 ± 0.4
1.2 ± 0.2
2.0 ± 0.2
1.7 ± 0.2
Putri [ 17 ]
2.4 ± 1.8
3.2 ± 2.7
3.0 ± 2.3
1.8 ± 0.9
2.7 ± 1.5
2.4 ± 1.3
Wanda [ 91 ] +Putri [ 17 ]
25.2 ± 1.0
24.0 ± 0.3
24.4 ± 0.5
3.0 ± 1.9
3.2 ± 2.3
3.1 ± 2.2
PruneLLM [ 113 ]
1.2 ± 0.4
2.3 ± 0.5
1.9 ± 0.5
0.2 ± 0.1
0.7 ± 0.6
0.5 ± 0.5
D 2 Prune [ 110 ]
24.4 ± 1.0
23.2 ± 0.4
23.6 ± 0.6
17.5 ± 0.1
20.6 ± 0.5
19.6 ± 0.3
Table 1: Results on criteria-only structured weight pruning/WF results averaged over three runs on InternVL3-2B [ 124 ] using only 256 samples from CLEVRER [ 115 ] (one of the three Seen datasets) for any calibration. For every method, all layers in the network must reduce the same number of FLOPs ( i.e . , uniform pruning/WF). Our BTWF approach reports a 2-3% overall gain over prior work. See Tab. 2 for results that include pruning/WF strategy ( i.e . , unequal budgets across layers).
FLOPs Budget
10%
30%
50%
Weight Selector
Budget Allocator
Seen
Unseen
All
Seen
Unseen
All
Seen
Unseen
All
LLM-Pruner [ 64 ]
LLM-Pruner [ 64 ]
35.4 ± 0.2
41.8 ± 0.1
39.6 ± 0.1
3.7 ± 0.6
5.1 ± 0.3
4.6 ± 0.3
1.3 ± 0.4
0.6 ± 0.1
0.8 ± 0.2
SlimLLM [ 33 ]
SlimLLM [ 33 ]
46.0 ± 0.2
51.9 ± 0.3
49.9 ± 0.2
8.5 ± 3.4
10.4 ± 2.7
9.8 ± 2.9
0.2 ± 0.1
4.7 ± 0.5
3.2 ± 0.4
PruneLLM [ 113 ]
PruneLLM [ 113 ]
43.7 ± 0.1
47.7 ± 0.2
46.4 ± 0.2
6.4 ± 0.4
9.7 ± 0.6
8.6 ± 0.2
0.2 ± 0.1
0.3 ± 0.0
0.3 ± 0.0
Putri [ 17 ]
Putri [ 17 ]
27.1 ± 3.4
30.1 ± 0.9
29.1 ± 1.7
9.5 ± 4.1
10.0 ± 5.6
9.8 ± 5.1
–
–
–
PORTA [ 49 ]
PORTA [ 49 ]
13.7 ± 1.2
19.3 ± 2.3
17.4 ± 1.9
1.5 ± 0.2
3.2 ± 0.1
2.7 ± 0.1
1.0 ± 0.2
2.4 ± 0.4
1.9 ± 0.3
Table 2: Results on structured weight pruning/WF results on an InternVL3-2B [ 124 ] backbone averaged over three runs using only 256 samples from CLEVRER [ 115 ] (one of the three Seen datasets) for any calibration. Each method uses the budget allocation strategy from their paper to achieve the same FLOPs budget as used in Tab. 1 . Our approach includes Overparameterized Groups (OG) described in Sec. 3.3.1 for some layers, with any remaining layers using a higher rate of uniform WF to obtain the total FLOPs budget. Our BTWF approach reports a 3-6% overall gain over prior work.
FLOPs Budget
30%
50%
Method
Seen
Unseen
All
Seen
Unseen
All
Wanda [ 91 ]
61.4 ± 0.7
32.1 ± 0.2
41.9 ± 0.3
22.6 ± 3.1
8.5 ± 2.8
13.2 ± 2.9
D 2 Prune [ 110 ]
53.7 ± 2.1
28.8 ± 0.8
37.1 ± 1.2
43.9 ± 0.7
23.6 ± 0.4
30.4 ± 0.3
PORTA [ 49 ]
63.4 ± 0.8
33.7 ± 0.5
43.6 ± 0.6
21.9 ± 5.3
9.8 ± 3.9
13.8 ± 4.4
RCPU [ 39 ]
63.2 ± 0.5
34.6 ± 0.4
44.2 ± 0.4
43.4 ± 0.9
23.1 ± 1.2
29.9 ± 1.1
TRSP [ 25 ]
52.3 ± 2.0
31.5 ± 1.0
38.4 ± 1.2
36.2 ± 0.4
20.5 ± 1.5
25.8 ± 0.9
Table 3: Results on structured weight pruning/WF results averaged over three runs on an InternVL3-2B [ 124 ] backbone after performing recovery via finetuning on the Seen datasets. We find our BTWF approach offers 1-3.5% gains over prior work. See Appendix C for discussion on additional recovery strategies.
Method
10% All
30% All
50% All
InternVL3-1B [ 124 ]
SVD-LLM [ 105 ]
31.9 ± 0.5
18.5 ± 0.7
3.6 ± 1.9
Basis Sharing [ 102 ]
29.0 ± 0.3
18.5 ± 0.3
4.8 ± 0.1
Cat [ 63 ]
37.2 ± 1.6
21.1 ± 0.7
18.2 ± 0.3
BTWF (ours)
38.2 ± 0.5
23.1 ± 0.7
18.5 ± 0.1
Qwen2-VL-2B [ 103 ]
Table 4: Results on criteria-only structured weight pruning/WF averaged over three runs across five architectures using only 256 samples from CLEVRER [ 115 ] (one of the three Seen datasets) for any calibration. For every method, all layers in the network must reduce the same number of FLOPs ( i.e . , uniform pruning/WF). Our BTWF approach demonstrates a strong ability to generalize, reporting the best overall results in 12 of the 15 model and FLOPs budget settings and taking second best on the remaining 3.
Method
10% Recon. ↓
30% Recon. ↓
50% Recon. ↓
InternVL3-2B [ 124 ]
Cat [ 63 ]
2.77 ± 0.11
4.65 ± 0.15
7.45 ± 0.16
FCB ( Sec. 3.1 )
2.20 ± 0.09
3.82 ± 0.13
6.37 ± 0.14
Gemma-4-E2B [ 30 ]
Cat [ 63 ]
2.41 ± 0.06
4.19 ± 0.08
6.59 ± 0.10
FCB ( Sec. 3.1 )
2.13 ± 0.05
3.78 ± 0.08
6.07 ± 0.09
Table 5: Comparison of relative activation-space reconstruction error ∥(W−W^)X∥F/∥WX∥F (%) averaged over three runs using only 256 samples from CLEVRER [ 115 ] (one of the three Seen datasets) for any calibration. Our FCB balances the contribution of each member, resulting in lower overall error in affected layers.
Method
10% All
30% All
50% All
BKV [ 67 ]
37.5 ± 0.7
23.5 ± 0.2
17.4 ± 0.3
FCB ( Sec. 3.1 )
38.2 ± 0.5
23.1 ± 0.7
18.5 ± 0.1
Table 6: Comparison on InternVL3-1B [ 124 ] replacing FCB in our full method with BKV balancing used in prior work. We find FCB provides a performance boost in most settings.
Figure 3: Relative squared error against the dense InternVL3-2B [ 124 ] ( ×103 ) in language-model layers 15–27 without recovery, at 30% (left) and 50% (right) FLOPs reduction. Top: reconstruction error of the shared members (query/key/value and gate/up), each evaluated on the dense model’s input to it, i.e . , the quantity the factorization fits. Bottom: error of each decoder block’s output on image tokens, given the dense block input. With BCB ( Sec. 3.2 ) the members are reconstructed slightly less accurately, but the error that reaches the block output is consistently lower, especially in the final layer.
Method
10% All
30% All
50% All
Cat w/Skip [ 63 ]
54.5 ± 0.7
36.3 ± 0.8
22.0 ± 0.4
BTWF (ours) w/ Skip
55.1 ± 0.4
39.3 ± 0.5
23.3 ± 0.3
BTWF w/ OG & Skip
55.4 ± 0.2
39.7 ± 0.6
23.8 ± 0.1
Table 7: Evaluates joint WF and weight pruning on an InternVL3-2B [ 124 ] backbone averaged over three runs using only 256 samples from CLEVRER [ 115 ] (one of the three Seen datasets) for any calibration. We find that combining BTWF w/OG and Skip [ 63 ] weight pruning boosts performance by 1-3.5%.
Method
50% All
70% All
GETA [ 78 ]
31.4 ± 0.5
22.0 ± 3.2
UniQL [ 16 ]
33.2 ± 1.6
22.4 ± 1.5
BTWF (ours) w/ GETA
36.5 ± 0.6
31.4 ± 0.6
Table 8: Results on joint WF and 4 bit quantization on an InternVL3-2B [ 124 ] backbone averaged over three runs after performing recovery using the Seen datasets. Combining BTWF with prior work boosts performance by 3.5-9%.
Method
50% All
70% All
CoMP [ 107 ]
18.0 ± 3.5
8.5 ± 2.6
CoMP w/NLL
23.1 ± 2.1
10.8 ± 2.8
BTWF (ours) w/ CoMP
25.1 ± 0.4
19.6 ± 0.1
Table 9: Results on joint WF and token pruning on an InternVL3-2B [ 124 ] backbone averaged over three runs using only 256 samples from CLEVRER [ 115 ] (one of the three Seen datasets) for any calibration. Combining BTWF with prior work reports a 2-9%, even after improving prior methods (see Appendix G ).
Memory
Latency
Speedup
Method
50% All
(GB)
(ms)
vs. Dense
D 2 Prune [ 110 ]
30.4 ± 0.3
2.89
88.5 ± 0.3
1.53 ×
RCPU [ 39 ]
29.9 ± 1.1
2.89
88.5 ± 0.5
1.53 ×
AdaSVD [ 57 ]
32.0 ± 0.6
2.98
103.8 ± 0.4
1.30 ×
Basis Sharing [ 102 ]
24.7 ± 0.7
2.84
103.8 ± 0.1
1.30 ×
Cat [ 63 ]
31.9 ± 0.8
2.86
102.3 ± 0.3
1.32 ×
Table 10: Compares latency and memory costs with a NVIDIA L40S GPU on an InternVL3-2B [ 124 ] backbone averaged over three runs after performing recovery using the Seen datasets. We find BTWF obtains a 2% task performance gain at similar or better latencies and higher memory savings than prior work.
FLOPs Budget
10%
30%
50%
Row weights
Seen
Unseen
All
Seen
Unseen
All
Seen
Unseen
All
BTWF w/o BCB
48.3 ± 0.2
46.3 ± 0.9
47.0 ± 0.6
33.6 ± 0.4
29.6 ± 0.9
30.9 ± 0.6
23.4 ± 0.6
20.7 ± 0.5
21.6 ± 0.4
w/ FWSVD [ 41 ]
48.2 ± 0.2
46.3 ± 0.3
46.9 ± 0.2
33.1 ± 0.8
28.7 ± 0.6
30.1 ± 0.5
23.2 ± 0.1
21.6 ± 0.3
22.1 ± 0.2
w/ MoDeGPT [ 60 ]
48.3 ± 0.4
46.4 ± 1.0
47.0 ± 0.6
33.8 ± 0.8
30.3 ± 1.4
31.4 ± 1.1
23.5 ± 0.6
20.7 ± 1.0
21.7 ± 0.9
BTWF (ours)
48.7 ± 0.2
46.9 ± 0.5
47.5 ± 0.3
34.4 ± 1.0
31.2 ± 0.9
32.2 ± 0.9
23.3 ± 0.7
21.6 ± 0.5
22.2 ± 0.5
Table 11: Comparison of BCB with the row importance of FWSVD [ 41 ] and MoDeGPT [ 60 ] on InternVL3-2B [ 124 ] averaged over three runs using only 256 samples from CLEVRER [ 115 ] (one of the Seen datasets) for calibration. Each row replaces BCB’s row weights Dk in Eq. 5 on the same shared groups. We find our BCB approach outperforms prior work. See Sec. A.2 for discussion.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Recovery training loss on InternVL3-2B [ 124 ] without OG and with the vision MLP held at dense cost ( Eq. 8 ) at μ=μ∗=1.25 , the smallest rank that reproduces the dense layers exactly ( Eq. 9 ), and at μ=1.5 . Both OG variants have the same inference FLOPs and the same model before recovery, i.e . , they differ only in the extra rank directions available during training. Lines show the mean over 3 seeds and bands ± 1 standard deviation (5-point moving average within each epoch). The loss steps down at each epoch boundary because recovery repeats the training data, suggesting that some training samples have been memorized. See Appendix B for discussion.
FLOPs Budget
10%
30%
50%
Method
Recovery Strategy
Seen
Unseen
All
Seen
Unseen
All
Seen
Unseen
All
BTWF (ours)
LoRA
70.1 ± 0.4
46.2 ± 2.3
54.2 ± 1.4
62.5 ± 0.7
33.4 ± 0.8
43.1 ± 0.8
53.8 ± 0.1
27.0 ± 0.5
36.0 ± 0.3
Staged LoRA
68.4 ± 1.0
48.4 ± 0.8
55.1 ± 0.4
60.3 ± 0.5
32.3 ± 0.5
41.6 ± 0.5
48.4 ± 1.6
27.0 ± 0.3
34.2 ± 0.4
SFT
70.5 ± 0.6
48.7 ± 0.5
56.0 ± 0.5
61.9 ± 1.9
35.2 ± 0.9
44.1 ± 0.2
50.1 ± 0.6
27.1 ± 0.3
34.8 ± 0.2
Last 3 Blocks
50.2 ± 1.3
42.9 ± 2.8
45.3 ± 2.0
36.1 ± 2.0
29.4 ± 1.1
31.6 ± 0.4
29.3 ± 1.1
24.4 ± 2.0
26.0 ± 1.7
KD
65.4 ± 0.6
46.4 ± 0.7
52.7 ± 0.5
57.5 ± 0.9
34.6 ± 0.5
42.2 ± 0.6
47.2 ± 0.3
28.1 ± 0.1
34.4 ± 0.1
Appendix
Table 12: Results of recovery strategies averaged over three runs on an InternVL3-2B [ 124 ] backbone evaluated on the first 500 test samples of each dataset after recovery with the same budget using the Seen datasets. INTERLACE cannot reach 50% as it removes at most ⌊L/3⌋ layers. Best results are bolded, second best are underlined. Note that BTWF does not use OG in these experiments, refer to Tab. 3 for performance with OG. See Appendix C for further discussion.
Figure 5: Effect of the member-balance exponent γ (member weights αk=∥W~k∥F−γ ) on six architectures, factorizing every block linear of both towers without recovery. (a, b) All accuracy relative to γ=1 , averaged over the 10%, 30%, and 50% FLOPs budgets, without and with BCB, respectively. γ=0 is the unweighted joint fit. (c) Cost of using one fixed γ everywhere, where the gap to the best γ of the sweep is averaged (solid) and maximized (dashed) over the 18 [model, budget] pairs. Gray bands show the mean standard deviation.
Table 13: Latency and memory costs on an NVIDIA L40S GPU for the criteria-only setting of Tab. 4 ( i.e . , InternVL3-2B [ 124 ] backbone using only calibration data). We find BTWF reports higher performance with lower latencies and similar memory in most settings across architectures and FLOPs budgets. See Appendix E for additional discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
30%
FLAP [ 4 ]
3.2 ± 0.2
0.8 ± 0.7
15.5 ± 2.7
27.8 ± 3.0
0.1 ± 0.0
0.0 ± 0.0
1.3 ± 0.9
22.1 ± 0.8
19.8 ± 0.1
10.1 ± 0.6
Wanda [ 91 ]
0.4 ± 0.1
0.1 ± 0.0
2.1 ± 1.1
2.2 ± 1.0
0.2 ± 0.1
0.0 ± 0.1
0.1 ± 0.0
1.8 ± 0.9
2.9 ± 1.0
1.1 ± 0.4
Putri [ 17 ]
1.6 ± 1.6
1.2 ± 2.0
4.4 ± 4.7
3.2 ± 4.0
0.3 ± 0.3
0.0 ± 0.0
2.2 ± 3.7
6.7 ± 7.8
7.1 ± 5.6
3.0 ± 2.3
Wanda [ 91 ] +Putri [ 17 ]
27.7 ± 1.3
21.8 ± 0.4
26.2 ± 1.3
41.3 ± 0.6
7.3 ± 0.5
7.5 ± 0.3
32.8 ± 0.7
27.3 ± 0.4
27.5 ± 0.4
24.4 ± 0.5
AMP [ 71 ]
0.6 ± 0.2
0.0 ± 0.0
1.9 ± 0.5
3.4 ± 0.2
0.2 ± 0.1
0.0 ± 0.0
0.1 ± 0.0
0.8 ± 0.3
1.9 ± 0.8
1.0 ± 0.1
Appendix
Table 14: Per-dataset results of the criteria-only evaluation in Tab. 1 . See Appendix F for discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
10%
LLM-Pruner [ 64 ]
20.7 ± 0.5
40.5 ± 0.3
44.9 ± 0.6
41.5 ± 0.1
36.1 ± 0.3
43.6 ± 1.0
58.8 ± 0.3
36.0 ± 0.5
34.7 ± 0.3
39.6 ± 0.1
SlimLLM [ 33 ]
26.5 ± 0.6
48.5 ± 0.2
63.0 ± 0.3
44.0 ± 0.2
61.1 ± 0.3
50.8 ± 1.5
69.4 ± 0.1
41.3 ± 0.3
44.9 ± 0.5
49.9 ± 0.2
PruneLLM [ 113 ]
29.5 ± 0.3
43.3 ± 0.2
58.3 ± 0.5
35.3 ± 0.7
64.2 ± 0.4
43.7 ± 0.4
66.3 ± 0.3
36.1 ± 0.6
40.5 ± 0.4
46.4 ± 0.2
Putri [ 17 ]
15.9 ± 6.9
29.8 ± 3.0
35.6 ± 4.5
39.1 ± 2.5
31.2 ± 1.4
10.8 ± 3.7
44.9 ± 3.2
28.2 ± 1.9
26.4 ± 1.9
29.1 ± 1.7
TRSP [ 25 ]
6.7 ± 1.9
32.8 ± 2.2
37.6 ± 4.6
42.5 ± 1.0
45.0 ± 4.7
30.7 ± 2.3
43.5 ± 2.9
34.2 ± 1.0
26.1 ± 2.1
33.2 ± 2.2
Appendix
Table 15: Per-dataset results comparing method-specific budget allocators from Tab. 2 . See Appendix F for discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
(a) Source method end to end
30%
AMP [ 71 ]
69.9 ± 0.2
52.5 ± 6.2
62.7 ± 0.1
39.8 ± 5.0
9.4 ± 0.3
11.6 ± 1.5
51.9 ± 5.3
31.5 ± 0.7
34.2 ± 0.9
40.4 ± 1.6
LLM-Pruner [ 64 ]
50.0 ± 1.7
55.3 ± 1.1
42.9 ± 4.4
32.3 ± 3.2
20.0 ± 2.3
17.9 ± 0.3
58.5 ± 1.8
29.8 ± 0.9
29.7 ± 1.5
37.4 ± 1.1
SlimLLM [ 33 ]
57.0 ± 0.3
58.0 ± 0.3
56.7 ± 0.5
37.7 ± 3.0
25.0 ± 0.9
29.6 ± 10.8
61.2 ± 0.2
31.4 ± 1.0
33.3 ± 0.8
43.3 ± 0.7
SVD-LLM [ 105 ]
48.7 ± 4.5
53.0 ± 1.4
51.9 ± 0.8
38.1 ± 2.0
13.2 ± 1.6
14.9 ± 0.8
53.9 ± 1.3
31.0 ± 1.3
33.4 ± 1.6
37.6 ± 0.9
Appendix
Table 16: Per-dataset results after performing recovery using the Seen datasets. (a) runs each method’s own compression and recovery as described in its paper (not shown in Tab. 3 ), (b) reports the per-dataset SFT results for Tab. 3 . See Appendix F for discussion.
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
10%
InternVL3-1B
SVD-LLM [ 105 ]
26.4 ± 0.4
34.7 ± 1.4
38.3 ± 0.5
40.0 ± 0.8
19.3 ± 0.3
24.4 ± 7.4
45.4 ± 2.0
29.2 ± 0.9
29.2 ± 1.3
31.9 ± 0.5
Basis Sharing [ 102 ]
27.4 ± 1.1
32.8 ± 0.6
34.1 ± 0.3
40.4 ± 0.6
14.8 ± 1.4
11.1 ± 0.4
44.7 ± 0.6
29.2 ± 0.3
26.6 ± 0.8
29.0 ± 0.3
Cat [ 63 ]
32.0 ± 0.7
40.2 ± 0.6
46.8 ± 1.3
42.1 ± 0.4
26.8 ± 0.5
30.7 ± 12.7
53.2 ± 0.4
31.7 ± 0.3
31.2 ± 0.6
37.2 ± 1.6
BTWF (ours)
32.0 ± 0.5
40.8 ± 0.4
47.7 ± 0.7
42.4 ± 0.6
26.5 ± 0.4
36.5 ± 6.1
53.5 ± 0.3
32.1 ± 0.6
32.7 ± 0.5
38.2 ± 0.5
Appendix
Table 17: Per-dataset results for the criteria-only evaluation across architectures in Tab. 4 . See Appendix F for discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
10%
BKV [ 67 ]
32.2 ± 0.2
40.7 ± 0.2
47.5 ± 1.4
43.4 ± 0.5
25.8 ± 0.3
31.5 ± 9.1
53.3 ± 0.5
31.6 ± 0.2
31.6 ± 1.2
37.5 ± 0.7
FCB (ours)
32.0 ± 0.5
40.8 ± 0.4
47.7 ± 0.7
42.4 ± 0.6
26.5 ± 0.4
36.5 ± 6.1
53.5 ± 0.3
32.1 ± 0.6
32.7 ± 0.5
38.2 ± 0.5
30%
BKV [ 67 ]
28.6 ± 0.2
24.4 ± 0.9
26.8 ± 0.9
37.7 ± 0.6
5.4 ± 0.2
4.2 ± 0.4
34.0 ± 0.7
26.4 ± 0.3
24.4 ± 0.8
23.5 ± 0.2
FCB (ours)
27.3 ± 0.6
22.4 ± 2.6
26.4 ± 0.1
36.7 ± 0.1
5.8 ± 0.3
4.7 ± 0.8
33.1 ± 1.7
25.9 ± 0.7
25.5 ± 1.2
23.1 ± 0.7
50%
BKV [ 67 ]
11.7 ± 0.6
11.6 ± 1.2
20.1 ± 0.2
33.4 ± 0.7
4.5 ± 0.3
4.0 ± 0.6
24.2 ± 1.7
25.6 ± 0.4
21.4 ± 0.1
17.4 ± 0.3
Appendix
Table 18: Per-dataset results comparing FCB with BKV [ 67 ] from Tab. 6 . See Appendix F for discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
10%
Cat w/Skip [ 63 ]
37.2 ± 1.2
55.5 ± 0.1
70.4 ± 0.5
51.1 ± 1.0
68.9 ± 0.4
46.8 ± 3.4
73.8 ± 0.1
43.4 ± 0.4
43.7 ± 0.7
54.5 ± 0.7
BTWF (ours)
36.1 ± 0.4
48.6 ± 0.4
61.5 ± 0.3
47.1 ± 0.5
47.7 ± 1.7
43.4 ± 2.3
64.1 ± 0.1
37.7 ± 0.0
41.6 ± 1.2
47.5 ± 0.3
BTWF w/Skip
37.2 ± 0.3
56.2 ± 0.4
71.8 ± 0.2
52.1 ± 0.5
70.0 ± 0.5
46.6 ± 1.6
74.0 ± 0.3
43.1 ± 0.4
44.7 ± 0.9
55.1 ± 0.4
BTWF w/OG w/Skip
37.2 ± 0.5
56.1 ± 0.1
71.3 ± 0.4
50.6 ± 0.4
70.2 ± 0.5
51.3 ± 1.1
73.9 ± 0.2
43.0 ± 0.7
44.9 ± 0.7
55.4 ± 0.2
30%
Cat w/Skip [ 63 ]
32.7 ± 1.1
38.2 ± 0.3
41.7 ± 1.5
41.5 ± 0.6
17.3 ± 0.6
40.7 ± 4.1
49.9 ± 0.4
31.2 ± 0.9
33.5 ± 0.7
36.3 ± 0.8
Appendix
Table 19: Per-dataset results on joint WF and weight pruning from Tab. 7 . See Appendix F for discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
50%
GETA [ 78 ]
60.9 ± 1.0
39.9 ± 0.2
45.6 ± 0.8
28.6 ± 3.6
5.7 ± 0.1
8.6 ± 0.8
40.1 ± 0.3
25.6 ± 0.5
27.9 ± 0.8
31.4 ± 0.5
UniQL [ 16 ]
74.9 ± 1.0
44.1 ± 1.3
40.4 ± 4.5
23.9 ± 10.3
5.1 ± 0.1
13.1 ± 4.4
45.1 ± 0.2
24.9 ± 0.8
27.4 ± 2.6
33.2 ± 1.6
BTWF (ours) w/ GETA
79.4 ± 0.5
48.2 ± 0.6
48.9 ± 1.6
35.1 ± 2.4
5.5 ± 0.3
10.6 ± 0.3
46.0 ± 0.6
26.8 ± 0.8
28.1 ± 0.9
36.5 ± 0.6
70%
GETA [ 78 ]
41.4 ± 6.8
36.5 ± 1.3
24.9 ± 11.0
11.8 ± 6.0
4.2 ± 0.2
7.1 ± 0.7
34.7 ± 1.1
19.4 ± 2.4
17.8 ± 5.4
22.0 ± 3.2
UniQL [ 16 ]
49.2 ± 0.6
34.5 ± 1.8
18.7 ± 0.1
12.2 ± 3.6
3.8 ± 1.2
8.2 ± 1.1
35.1 ± 1.2
21.8 ± 3.4
18.4 ± 3.6
22.4 ± 1.5
Appendix
Table 20: Per-dataset results on joint WF and quantization from Tab. 8 . See Appendix F for discussion.
FLOPs
Seen
Unseen
Budget
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
50%
CoMP [ 107 ]
6.6 ± 2.6
18.9 ± 1.6
12.3 ± 2.7
20.8 ± 7.5
6.9 ± 3.0
33.7 ± 14.3
24.6 ± 1.4
17.6 ± 2.7
20.8 ± 2.0
18.0 ± 3.5
CoMP w/ NLL
8.7 ± 4.0
20.8 ± 2.8
20.6 ± 7.0
32.3 ± 7.9
11.6 ± 3.4
36.6 ± 5.2
27.6 ± 5.2
23.7 ± 2.6
26.2 ± 3.3
23.1 ± 2.1
BTWF (ours) w/ CoMP
30.6 ± 0.8
23.9 ± 0.0
26.6 ± 0.7
38.5 ± 1.2
8.0 ± 0.1
9.7 ± 2.4
32.3 ± 0.3
27.6 ± 0.5
28.4 ± 0.6
25.1 ± 0.4
70%
CoMP [ 107 ]
3.7 ± 2.3
0.2 ± 0.2
17.3 ± 3.7
19.4 ± 12.5
0.2 ± 0.2
0.0 ± 0.0
0.2 ± 0.1
17.1 ± 6.0
18.1 ± 1.7
8.5 ± 2.6
CoMP w/ NLL
6.3 ± 5.1
5.9 ± 0.9
17.0 ± 3.7
21.6 ± 10.2
1.2 ± 0.7
0.6 ± 0.6
7.6 ± 2.1
18.1 ± 6.6
18.4 ± 2.7
10.8 ± 2.8
Appendix
Table 21: Per-dataset results on joint WF and token pruning from Tab. 9 . See Appendix F for discussion.
Dataset
Media
Train
Val
Test (source)
Seen datasets
CLEVRER [ 115 ]
video
61,064
14,235
10,000 (subset of official val)
GQA [ 44 ]
image
65,093
25,803
12,578 (testdev-balanced)
NExT-QA [ 109 ]
video
7,864
896
4,221 (video-level re-split)
Unseen datasets
Perception Test [ 74 ]
video
–
–
19,140 (val)
VQAv2 [ 31 ]
image
–
–
10,000 (subset of val)
VizWiz [ 36 ]
image
–
–
4,319 (val)
Appendix
Table 22: Number of sample questions per split. The recovery set is the union of the three training splits (134,021 question–answer pairs).
Seen
Unseen
Method
CLEVRER
GQA
NExT-QA
Perc.Test
TextVQA
VizWiz
VQAv2
VideoMME
LVB
All
InternVL3-1B [ 124 ]
36.1
57.2
70.9
51.7
72.5
49.3
75.1
40.7
41.3
55.0
InternVL3-2B [ 124 ]
36.5
59.3
75.3
55.1
77.3
53.2
78.2
46.2
46.8
58.7
Qwen2-VL-2B [ 103 ]
38.8
60.0
83.9
47.7
70.2
64.3
78.7
45.0
43.4
59.1
Qwen3-VL-2B [ 10 ]
35.6
58.6
69.0
51.0
70.6
70.6
77.1
46.3
44.2
58.1
Qwen3.5-2B [ 79 ]
39.2
56.9
63.4
50.2
69.0
64.0
77.9
47.8
45.7
57.1
Appendix
Table 23: Per-dataset results for Tab. 24 .
Model
Seen
Unseen
All
InternVL3-1B [ 124 ]
54.7
55.1
55.0
InternVL3-2B [ 124 ]
57.0
59.5
58.7
Qwen2-VL-2B [ 103 ]
60.9
58.2
59.1
Qwen3-VL-2B [ 10 ]
54.4
60.0
58.1
Qwen3.5-2B [ 79 ]
53.1
59.1
57.1
Gemma4 E2B [ 30 ]
37.5
48.1
44.6
Appendix
Table 24: Base architecture performance to provide an oracle reference for each setting in our experiments. We report results for (top) off-the-shelf and (bottom) with Supervised FineTuning (SFT) over three random seeds.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Activation-aware low-rank factorization techniques yield strong compression results, but they are generally confined to linear layers, and existing whitening-based theory typically makes an implicit full-rank assumption on layer activations. We introduce a layer representation framework that allows us to understand and extend low-rank factorization beyond linear layers, including standard and grouped convolutions. Within this framework, we provide a broader characterization of whitening-based activation-aware factorization that naturally covers rank-deficient activations, and show that the resulting projection is optimal in terms of layer output distortion. We then use output distortion as a proxy to allocate per-layer ranks under explicit FLOP or parameter count budgets, with negligible overhead via a Lagrangian relaxation of the allocation problem. Together, these components form BALF, an end-to-end, fine-tuning-free pipeline for model compression. Across CNNs and vision transformers on CIFAR-10 and ImageNet-1K, BALF generally achieves higher accuracy than SVD-based factorization baselines at matched FLOP or parameter count targets, and remains competitive with other fine-tuning-free compression techniques.
David González-Martínez
Department of Computer Science and Artificial Intelligence, Universidad de Sevilla · Max Planck Institute for Intelligent Systems · University of Tübingen +2
Large Language Models (LLMs) offer strong capabilities but incur high inference costs due to dense computation and memory access. Training-free activation sparsity is a promising approach for efficient LLM inference, yet existing methods often rely solely on activation information and uniform sparsity ratios. This overlooks the critical interplay with weights and inter-block sensitivity variation, leading to suboptimal performance. We identify two key phenomena in modern LLMs: 1) less significant activations may align with highly important weights, and 2) sparsity sensitivity varies non-monotonically across model blocks. We propose Weight-aware Mixed-Granularity Training-free Activation Sparsity (WiSparse), which leverages both activation and weight information for adaptive sparsity allocation. Specifically, we introduce a weight-aware mechanism integrating activation magnitudes with precomputed weight norms to accurately identify salient channels. This is combined with a mixed-granularity allocation scheme: a global budget is distributed across blocks via evolutionary search to protect sensitive regions, then refined within blocks to minimize reconstruction error. We improve sparse kernels and demonstrate effectiveness on three representative models. Notably, at 50% sparsity, WiSparse preserves 97% of Llama3.1's dense performance, surpassing the strongest baseline by 2.23 percentage points while achieving a 21.4% acceleration in end-to-end inference speed. Our research advances the limits of training-free approaches for efficient LLM inference, pushing the boundaries of achievable speedup without training.
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity. However, existing methods struggle with heavy-tailed activation distributions and therefore keep activations in high precision, fundamentally limiting end-to-end inference acceleration. To overcome this limitation, we propose TWLA, a post-training quantization (PTQ) framework that achieves 1.58-bit weight compression and 4-bit activation quantization while maintaining high accuracy. TWLA comprises three components: (1) Euclidean-to-Manifold Asymmetric Ternary Quantizer (E2M-ATQ) minimizes layer-output error under weight ternarization via a two-stage optimization from Euclidean initialization to manifold relocation; (2) Kronecker Orthogonal Tri-Modal Shaping (KOTMS) applies a Kronecker-structured orthogonal rotation to reshape weights into ternary-friendly tri-modal distributions, while the shared rotation statistically suppresses activation outliers; and (3) Inter-Layer Aware Activation Mixed Precision (ILA-AMP) explicitly introduces adjacent-layer second-order interaction costs in bit allocation and jointly optimizes for the layer-wise disparity of activation quantization gains induced by the shared orthogonal transform, preventing cascades triggered by a few weak layers. Extensive experiments demonstrate that TWLA maintains high accuracy under W1.58A4, while delivering significant inference acceleration. The code is available at https://github.com/Kishon-zzx/TWLA.
Zhixiong Zhao, Zukang Xu, Zhixuan Chen +3
Houmo AI, China · 2Southeast University, Nanjing, China