Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SVD) is a principled tool for parameter reduction, evidence from large language models (LLMs) suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks. Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective. We apply SVDtrunc to FLUX.dev across compression levels ranging from 40-90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models. Project page: https://vislearn.github.io/SVDtrunc/
Figures & tables
Figure 1: Quantitative results of our SVDtrunc approach compared to the state of the art. We plot quality relative to the original FLUX.dev model [ 19 ] (in %, ↑ ) against parameter reduction (%, ↑ ). The quality is averaged across the relative performances on the GenEval [ 10 ] , DPG [ 13 ] , and HPSv2 [ 40 ] benchmarks. We outperform prior models across all compression-levels.
Figure 2: Qualitative results of applying our SVDtrunc compression schema to FLUX.dev [ 19 ] , ranging from 68% to 39% of the original parameter count. A minor degradation in quality is visible for 39% of parameters.
Figure 3: SVDtrunc Overview. (left) We first generate N probed models, by individually compressing each of the N blocks. Then, each probed model generates a set of images. By comparing these sets with the generations of the original model, we obtain a quality-based ranking of block importance. We use this ranking (right) to assign importance-aware compression factors to each block of the final compressed model.
Figure 4: Ranking metric justification. Depending on which block is compressed, the generated image of the compressed model (right) can deviate from the image generated by the uncompressed one (left) . However, the pixel-wise change is not necessarily correlated with quality, making pixel-wise similarity scores less suitable for block ranking.
Figure 5: Block sensitivity comparison . Generated images after probing double-stream blocks of FLUX.dev [ 19 ] with pruning (top) vs . truncated SVD (bottom) .
Method
P [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.dev [ 19 ]
100
0.647
31.70
83.9
0.00
EcoDiff [ 45 ]
90
0.452
29.67
78.0
14.52
Pruning
90
0.646
31.02
83.3
1.00
SVDtrunc (ours)
90
0.638
29.03
78.9
5.24
EcoDiff [ 45 ]
80
0.187
23.63
49.1
45.99
Pruning
80
0.381
24.88
65.2
28.31
Table 1: Training-free compression. While importance-aware Pruning and EcoDiff collapse even under moderate compression, our SVDtrunc maintains stable generation quality even at 70% of the original parameter count.
Method
Venue
Params. [%] ↓
VRAM [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
Rank ↓
FLUX.dev [ 19 ]
–
100
100.0
0.647
31.70
83.9
0.00
–
Chroma-HD [ 29 ]
HF
75
82.5
0.593
–
84.0
4.09
3
FluxLite [ 36 ]
HF’24
68
78.8
0.523
31.31
79.3
8.61
5
TinyFusion [ 8 ]
CVPR’25
68
74.4
0.511
–
77.2
14.48
6
Dense2MoE [ 46 ]
ICCV’25
75
–
0.403
–
73.6
24.97
9
FastFlux [ 2 ]
AAAI’26
80
–
0.530
27.26
–
16.04
8
Table 2: Quantitative results for compression of FLUX.dev [ 19 ] . The final Rank is based on the average quality reduction R . Our medium-sized model SVDtrunc-m achieves first rank. Notably, our small-sized model SVDtrunc-s, which has considerably fewer parameters than all alternative approaches, is runner-up.
Method
Params. [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
PixArt- Σ [ 3 ]
100 (0.6B)
0.535
30.50
78.8
0.00
SVDtrunc
90
0.543
29.34
78.5
0.22
SVDtrunc
80
0.528
29.79
78.0
1.62
SVDtrunc
70
0.532
29.64
78.2
1.38
SVDtrunc
60
0.528
29.34
78.0
2.04
SVDtrunc
50
0.496
28.28
76.6
5.79
Table 3: Results for PixArt- Σ model. The degradation in quality of our SVDtrunc results are gradual, without abrupt failure.
Figure 6: Example of domain drift (top) and memorisation loss (bottom) under different parameter budgets.
Method
P [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.schnell [ 19 ]
100
0.667
30.85
85.0
0.00
SVDtrunc (ours)
90
0.666
29.59
85.2
1.34
SVDtrunc (ours)
80
0.668
29.48
84.9
1.48
SVDtrunc (ours)
70
0.648
29.30
84.8
2.72
SVDtrunc (ours)
60
0.621
29.05
84.3
4.53
Table 4: Results for temporally-distilled FLUX.schnell [ 19 ] model. The degradation in quality of our SVDtrunc results are gradual, without abrupt failure.
Method
P [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.dev
100
0.647
31.70
83.9
0.00
Random
68
0.633
31.19
82.9
1.63
Uniform
68
0.633
31.38
82.8
1.46
Ours
68
0.645
31.27
83.4
0.75
Random
57
0.588
30.22
81.1
5.68
Uniform
57
0.613
30.92
82.3
3.19
Table 5: Compression Schedule Ablation. We evaluate uniform, random and importance-aware (our) SVD compression schedules after training.
Figure S1 : Conceptual SVD Compression. High-level illustration of where truncated SVD–based low-rank approximations are applied within multi-modal diffusion transformer (MMDiT) blocks [ 7 ] . Green modules denote components compressed with block-wise low-rank approximations, while grey components contain negligible or no parameters and are left unchanged to preserve the original architecture.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Params. [%] ↓
GenEval ↑
HPSv2 ↑
DPG ↑
R [%] ↓
FLUX.dev [ 19 ]
100 (12B)
0.647
31.70
83.9
0.00
PPCL [ 24 ]
68
0.605
–
80.0
5.55
SVDtrunc
90
0.660
31.36
83.6
−0.19
SVDtrunc
80
0.651
31.39
83.6
0.21
SVDtrunc-m
68
0.645
31.27
83.4
0.75
SVDtrunc-s
57
0.616
31.22
82.6
2.60
Appendix
Table S1 : Quantitative results for compression of FLUX.dev [ 19 ] and PixArt- Σ [ 3 ] . For clarity, we report the average quality reduction across GenEval [ 10 ] , HPSv2 [ 40 ] and DPG [ 13 ] benchmarks in relation to the respective uncompressed model as R . SVDtrunc preserves most of the original model performance even at parameter budgets of ≤ 60 %, exhibiting graceful degradation as compression increases.
Method
Params. [%] ↓
Two Object ↑
Single Object ↑
Colour Attribute ↑
Colours ↑
Counting ↑
Position ↑
Overall ↑
FLUX.dev [ 19 ]
100
0.768
0.981
0.455
0.758
0.709
0.210
0.647
PPCL [ 24 ]
68
0.726
0.978
0.380
0.785
0.593
0.170
0.605
SVDtrunc
90
0.806
0.984
0.473
0.769
0.703
0.223
0.656
SVDtrunc
80
0.801
0.988
0.468
0.755
0.663
0.235
0.651
SVDtrunc-m
68
0.818
0.991
0.451
0.739
0.641
0.230
0.645
SVDtrunc-s
57
0.790
0.991
0.418
0.729
0.569
0.203
0.616
Appendix
Table S2 : Detailed quantitative results on the object focused benchmark GenEval [ 10 ] . For FLUX.dev [ 19 ] , compression down to moderate parameter budgets (57 %) mainly degrades performance on object counting, while other tasks remain relatively stable. In contrast, PixArt-based compression primarily affects positional generation, while performance on other tasks remains largely preserved.
Method
Params. [%] ↓
Concept Art ↑
Photo ↑
Anime ↑
Painting ↑
Overall ↑
FLUX.dev [ 19 ]
100
31.56
30.26
33.04
31.93
31.70
SVDtrunc
90
31.01
30.09
32.73
31.56
31.35
SVDtrunc
80
31.00
30.18
32.74
31.64
31.39
SVDtrunc-m
68
30.94
29.97
32.62
31.53
31.27
SVDtrunc-s
57
31.00
29.90
32.53
31.45
31.22
SVDtrunc
49
30.82
29.86
32.21
31.14
31.01
Appendix
Table S3 : Detailed quantitative results on HPSv2 [ 40 ] . SVDtrunc shows consistent and graceful degradation across all categories, without any single category degrading disproportionately.
Method
Params. [%] ↓
Global ↑
Entity ↑
Attribute ↑
Relation ↑
Other ↑
Overall ↑
FLUX.dev [ 19 ]
100
83.3
90.1
86.8
93.5
82.8
83.9
PPCL [ 24 ]
68
85.1
87.9
85.6
89.9
87.8
80.0
SVDtrunc
90
83.6
89.9
87.0
93.4
83.2
83.6
SVDtrunc
80
82.4
89.9
86.8
93.2
82.8
83.6
SVDtrunc-m
68
81.8
89.2
86.7
93.0
79.6
83.4
SVDtrunc-s
57
82.1
89.1
86.5
92.5
80.0
82.6
Appendix
Table S4 : Detailed quantitative results on DPG [ 13 ] . SVDtrunc maintains competitive performance across all DPG categories with graceful degradation under stronger compression.
start P
end P
Steps
K
cm
α
FLUX.dev [ 19 ]
100
90
20000
30
0.25
8.30
90
80
20000
30
0.25
5.94
100
68
40000
48
0.60
12.70
68
57
20000
30
0.35
4.58
57
49
20000
30
0.40
9.38
Appendix
Table S5 : Training details. For every training of SVDtrunc from a start parameter budget to an end parameter budget Pstart⟶Pend we provide the number of compressed blocks K , with the maximum compression cm , the compression slope α , and the number of training steps. For readability, we provide α scaled by 1000.
Figure S2 : Quality–parameter trade-off for different numbers of compressed blocks.
Figure S3 : Domain drift examples. With aggressive compression, the model can diverge from manga or cartoonish styles towards more realistic images.
Figure S4 : Memorisation loss examples. With aggressive compression, the model can not reproduce iconic instances like the tower of Pisa or Bugs Bunny.
Diffusion Transformers (DiTs) deliver remarkable image and video generation quality but incur high computational cost, limiting scalability and on-device deployment. We introduce CoReDiT, a structured token pruning framework for DiTs across vision tasks. CoReDiT uses a linear-time spatial coherence score to estimate local redundancy in the latent token lattice and skips high coherence (redundant) tokens in self-attention. To maintain a dense representation and avoid visual discontinuities, we reconstruct skipped attention outputs via coherence-guided aggregation of spatially neighboring retained tokens. We further introduce a progressive, block-adaptive pruning schedule that increases pruning gradually and allocates larger budgets to blocks and denoising steps with higher redundancy. Across state-of-the-art diffusion backbones including PixArt-α and MagicDrive-V2, CoReDiT achieves up to 55% self-attention FLOPs reduction and inference speedups of 1.33x on cloud GPUs and 1.72x on mobile NPUs, while maintaining high visual quality. Notably, CoReDiT also increases on-device memory head-room, enabling higher-resolution generation.
Diffusion Transformer (DiT) architectures have significantly advanced Text-to-Image (T2I) generation but suffer from prohibitive computational costs and deployment barriers. To address these challenges, we propose an efficient compression framework that transforms the 60-layer dual-stream MMDiT-based Qwen-Image into lightweight models without training from scratch. Leveraging this framework, we introduce Amber-Image, a series of streamlined T2I models. We first derive Amber-Image-10B using a timestep-sensitive depth pruning strategy, where retained layers are reinitialized via local weight averaging and optimized through layer-wise distillation and full-parameter fine-tuning. Building on this, we develop Amber-Image-6B by introducing a hybrid-stream architecture that converts deep-layer dual streams into a single stream initialized from the image branch, further refined via progressive distillation and lightweight fine-tuning. Our approach reduces parameters by 70% and eliminates the need for large-scale data engineering. Notably, the entire compression and training pipeline-from the 10B to the 6B variant-requires fewer than 2,000 GPU hours, demonstrating exceptional cost-efficiency compared to training from scratch. Extensive evaluations on benchmarks like DPG-Bench and LongText-Bench show that Amber-Image achieves high-fidelity synthesis and superior text rendering, matching much larger models.
Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art (SoTA) results. We further demonstrate scalability by applying our method to models with up to 70B parameters.
Shengkun Tang, Cong Zeng, Zongfang Liu +2
MBZUAI · Technology Innovation Institute UAE · Zhejiang Univeristy +2