Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SVD) is a principled tool for parameter reduction, evidence from large language models (LLMs) suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks. Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective. We apply SVDtrunc to FLUX.dev across compression levels ranging from 40-90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models. Project page: https://vislearn.github.io/SVDtrunc/
Figures & tables
Figure 1: Quantitative results of our SVDtrunc approach compared to the state of the art. We plot quality relative to the original FLUX.dev model [ 19 ] (in %, ↑ ) against parameter reduction (%, ↑ ). The quality is averaged across the relative performances on the GenEval [ 10 ] , DPG [ 13 ] , and HPSv2 [ 40 ] benchmarks. We outperform prior models across all compression-levels.
Figure 2: Qualitative results of applying our SVDtrunc compression schema to FLUX.dev [ 19 ] , ranging from 68% to 39% of the original parameter count. A minor degradation in quality is visible for 39% of parameters.
Figure 3: SVDtrunc Overview. (left) We first generate N probed models, by individually compressing each of the N blocks. Then, each probed model generates a set of images. By comparing these sets with the generations of the original model, we obtain a quality-based ranking of block importance. We use this ranking (right) to assign importance-aware compression factors to each block of the final compressed model.
Figure 4: Ranking metric justification. Depending on which block is compressed, the generated image of the compressed model (right) can deviate from the image generated by the uncompressed one (left) . However, the pixel-wise change is not necessarily correlated with quality, making pixel-wise similarity scores less suitable for block ranking.
Figure 5: Block sensitivity comparison . Generated images after probing double-stream blocks of FLUX.dev [ 19 ] with pruning (top) vs . truncated SVD (bottom) .
Method
P [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.dev [ 19 ]
100
0.647
31.70
83.9
0.00
EcoDiff [ 45 ]
90
0.452
29.67
78.0
14.52
Pruning
90
0.646
31.02
83.3
1.00
SVDtrunc (ours)
90
0.638
29.03
78.9
5.24
EcoDiff [ 45 ]
80
0.187
23.63
49.1
45.99
Pruning
80
0.381
24.88
65.2
28.31
Table 1: Training-free compression. While importance-aware Pruning and EcoDiff collapse even under moderate compression, our SVDtrunc maintains stable generation quality even at 70% of the original parameter count.
Method
Venue
Params. [%] ↓
VRAM [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
Rank ↓
FLUX.dev [ 19 ]
–
100
100.0
0.647
31.70
83.9
0.00
–
Chroma-HD [ 29 ]
HF
75
82.5
0.593
–
84.0
4.09
3
FluxLite [ 36 ]
HF’24
68
78.8
0.523
31.31
79.3
8.61
5
TinyFusion [ 8 ]
CVPR’25
68
74.4
0.511
–
77.2
14.48
6
Dense2MoE [ 46 ]
ICCV’25
75
–
0.403
–
73.6
24.97
9
FastFlux [ 2 ]
AAAI’26
80
–
0.530
27.26
–
16.04
8
Table 2: Quantitative results for compression of FLUX.dev [ 19 ] . The final Rank is based on the average quality reduction R . Our medium-sized model SVDtrunc-m achieves first rank. Notably, our small-sized model SVDtrunc-s, which has considerably fewer parameters than all alternative approaches, is runner-up.
Method
Params. [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
PixArt- Σ [ 3 ]
100 (0.6B)
0.535
30.50
78.8
0.00
SVDtrunc
90
0.543
29.34
78.5
0.22
SVDtrunc
80
0.528
29.79
78.0
1.62
SVDtrunc
70
0.532
29.64
78.2
1.38
SVDtrunc
60
0.528
29.34
78.0
2.04
SVDtrunc
50
0.496
28.28
76.6
5.79
Table 3: Results for PixArt- Σ model. The degradation in quality of our SVDtrunc results are gradual, without abrupt failure.
Figure 6: Example of domain drift (top) and memorisation loss (bottom) under different parameter budgets.
Method
P [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.schnell [ 19 ]
100
0.667
30.85
85.0
0.00
SVDtrunc (ours)
90
0.666
29.59
85.2
1.34
SVDtrunc (ours)
80
0.668
29.48
84.9
1.48
SVDtrunc (ours)
70
0.648
29.30
84.8
2.72
SVDtrunc (ours)
60
0.621
29.05
84.3
4.53
Table 4: Results for temporally-distilled FLUX.schnell [ 19 ] model. The degradation in quality of our SVDtrunc results are gradual, without abrupt failure.
Method
P [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.dev
100
0.647
31.70
83.9
0.00
Random
68
0.633
31.19
82.9
1.63
Uniform
68
0.633
31.38
82.8
1.46
Ours
68
0.645
31.27
83.4
0.75
Random
57
0.588
30.22
81.1
5.68
Uniform
57
0.613
30.92
82.3
3.19
Table 5: Compression Schedule Ablation. We evaluate uniform, random and importance-aware (our) SVD compression schedules after training.
Figure S1 : Conceptual SVD Compression. High-level illustration of where truncated SVD–based low-rank approximations are applied within multi-modal diffusion transformer (MMDiT) blocks [ 7 ] . Green modules denote components compressed with block-wise low-rank approximations, while grey components contain negligible or no parameters and are left unchanged to preserve the original architecture.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Params. [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.dev [ 19 ]
100 (12B)
0.647
31.70
83.9
0.00
PPCL [ 24 ]
68
0.605
–
80.0
5.55
SVDtrunc
90
0.660
31.36
83.6
−0.19
SVDtrunc
80
0.651
31.39
83.6
0.21
SVDtrunc-m
68
0.645
31.27
83.4
0.75
SVDtrunc-s
57
0.616
31.22
82.6
2.60
Appendix
Table S1 : Quantitative results for compression of FLUX.dev [ 19 ] and PixArt- Σ [ 3 ] . For clarity, we report the average quality reduction across GenEval [ 10 ] , HPSv2 [ 40 ] and DPG [ 13 ] benchmarks in relation to the respective uncompressed model as R . SVDtrunc preserves most of the original model performance even at parameter budgets of ≤ 60 %, exhibiting graceful degradation as compression increases.
Method
Params. [%] ↓
Two Object ↑
Single Object ↑
Colour Attribute ↑
Colours ↑
Counting ↑
Position ↑
Overall ↑
FLUX.dev [ 19 ]
100
0.768
0.981
0.455
0.758
0.709
0.210
0.647
PPCL [ 24 ]
68
0.726
0.978
0.380
0.785
0.593
0.170
0.605
SVDtrunc
90
0.806
0.984
0.473
0.769
0.703
0.223
0.656
SVDtrunc
80
0.801
0.988
0.468
0.755
0.663
0.235
0.651
SVDtrunc-m
68
0.818
0.991
0.451
0.739
0.641
0.230
0.645
SVDtrunc-s
57
0.790
0.991
0.418
0.729
0.569
0.203
0.616
Appendix
Table S2 : Detailed quantitative results on the object focused benchmark GenEval [ 10 ] . For FLUX.dev [ 19 ] , compression down to moderate parameter budgets (57 %) mainly degrades performance on object counting, while other tasks remain relatively stable. In contrast, PixArt-based compression primarily affects positional generation, while performance on other tasks remains largely preserved.
Method
Params. [%] ↓
Concept Art ↑
Photo ↑
Anime ↑
Painting ↑
Overall ↑
FLUX.dev [ 19 ]
100
31.56
30.26
33.04
31.93
31.70
SVDtrunc
90
31.01
30.09
32.73
31.56
31.35
SVDtrunc
80
31.00
30.18
32.74
31.64
31.39
SVDtrunc-m
68
30.94
29.97
32.62
31.53
31.27
SVDtrunc-s
57
31.00
29.90
32.53
31.45
31.22
SVDtrunc
49
30.82
29.86
32.21
31.14
31.01
Appendix
Table S3 : Detailed quantitative results on HPSv2 [ 40 ] . SVDtrunc shows consistent and graceful degradation across all categories, without any single category degrading disproportionately.
Method
Params. [%] ↓
Global ↑
Entity ↑
Attribute ↑
Relation ↑
Other ↑
Overall ↑
FLUX.dev [ 19 ]
100
83.3
90.1
86.8
93.5
82.8
83.9
PPCL [ 24 ]
68
85.1
87.9
85.6
89.9
87.8
80.0
SVDtrunc
90
83.6
89.9
87.0
93.4
83.2
83.6
SVDtrunc
80
82.4
89.9
86.8
93.2
82.8
83.6
SVDtrunc-m
68
81.8
89.2
86.7
93.0
79.6
83.4
SVDtrunc-s
57
82.1
89.1
86.5
92.5
80.0
82.6
Appendix
Table S4 : Detailed quantitative results on DPG [ 13 ] . SVDtrunc maintains competitive performance across all DPG categories with graceful degradation under stronger compression.
start P
end P
Steps
K
cm
α
FLUX.dev [ 19 ]
100
90
20000
30
0.25
8.30
90
80
20000
30
0.25
5.94
100
68
40000
48
0.60
12.70
68
57
20000
30
0.35
4.58
57
49
20000
30
0.40
9.38
Appendix
Table S5 : Training details. For every training of SVDtrunc from a start parameter budget to an end parameter budget Pstart⟶Pend we provide the number of compressed blocks K , with the maximum compression cm , the compression slope α , and the number of training steps. For readability, we provide α scaled by 1000.
Figure S2 : Quality–parameter trade-off for different numbers of compressed blocks.
Figure S3 : Domain drift examples. With aggressive compression, the model can diverge from manga or cartoonish styles towards more realistic images.
Figure S4 : Memorisation loss examples. With aggressive compression, the model can not reproduce iconic instances like the tower of Pisa or Bugs Bunny.