B-DENSE: Branching For Dense Ensemble Network Supervision Efficiency

Cherish Puniani; Tushar Kumar; Arnav Bendre; Gaurav Kumar; Shree Singhi

cs.LGFeb 17, 2026

B-DENSE: Branching For Dense Ensemble Network Supervision Efficiency

Authors: Cherish Puniani, Tushar Kumar, Arnav Bendre, Gaurav Kumar, Shree Singhi

Organizations: Indian Institute of Technology, Roorkee Roorkee, Uttarakhand, India

Abstract

Inspired by non-equilibrium thermodynamics, diffusion models have achieved state-of-the-art performance in generative modeling. However, their iterative sampling nature results in high inference latency. While recent distillation techniques accelerate sampling, they discard intermediate trajectory steps. This sparse supervision leads to a loss of structural information and introduces significant discretization errors. To mitigate this, we propose B-DENSE, a novel framework that leverages multi-branch trajectory alignment. We modify the student architecture to output $K$ -fold expanded channels, where each subset corresponds to a specific branch representing a discrete intermediate step in the teacher's trajectory. By training these branches to simultaneously map to the entire sequence of the teacher's target timesteps, we enforce dense intermediate trajectory alignment. Consequently, the student model learns to navigate the solution space from the earliest stages of training, demonstrating superior image generation quality compared to baseline distillation frameworks.

Figures & tables

Figure 1: This visualization illustrates a single iteration of the proposed B-DENSE method. Here, $x_{t}$ represents a noisy image at a random time step $t$ . At the same time, $x_{t-3}$ is output directly mapped by the standard distillation method, and $x_{t-1}$ and $x_{t-2}$ are consecutive intermediate denoised outputs that will be skipped by the standard distillation method. The blue, yellow, and red channels indicate branched channels for $K=3$ . Blue, yellow, and red arrows illustrate the mapping of each three-channel configuration to the final output and to an intermediate step used to calculate the reconstruction loss for each corresponding branch.

Methodology	512	256	128
ProgressiveDistillation (Baseline)	11.96	21.52	39.66
B-DENSE (Ours)	8.92	12.04	20.81

Table 1: Comparison of the FID of both frameworks on CIFAR-10 with various NFE

Figure 2: Results on calculating FID for 50k images

Table 4

Panel of Figure 3 — Figure 3: Comparison of results of B-DENSE (right) and SFD (left)

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Fitted $(a,b)$ parameter pairs sorted by ascending FID score. The color scale indicates the FID score (lower is better). The clustering of high-performing configurations (dark purple) demonstrates a strong preference for a specific relationship between the growth rate $a$ and intercept $b$ .

		FID (NFE 2)
a	b	ImageNet 64	CIFAR-10
1.386	-4.274	9.57	4.40
1.337	-4.195	9.72	4.40
1.230	-4.085	9.84	4.40
1.278	-4.137	9.72	4.41
1.277	-4.207	9.64	4.41

Appendix

Table 4: Top-5 weight configurations found via Optuna parameter search. The weights follow the form

\lambda_{i}=\exp(a\cdot i+b)

. We report FID scores at NFE 2 for both ImageNet

64\times 64

and CIFAR-10

32\times 32

. All configurations outperform the SFD baseline.

Diffusion Model Distillation

Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.

See all 189 papers

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models
May 30, 2026Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain +1Model Compression Classifier-Free Guidance
Khulna University of Engineering & Technology, Bangladesh · University Clermont Auvergne, France
One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration
May 20, 2026Chaoyang Wang, Yunhai TongDiffusion Model Distillation Few-Step Diffusion Sampling
Peking University
TamedFlow: Trajectory-Aware Distillation for Teacher-Assisted Diffusion Sampling
Oct 19, 2026Shuaiqi Cheng, Jiahao Zhang, Boyuan Gu +5Diffusion Sampling Diffusion Model Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · University of Electronic Science and Technology of China · The Hong Kong University of Science and Technology +2

May 30, 2026J/K move · Enter open · S save

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo

Khulna University of Engineering & Technology, Bangladesh · University Clermont Auvergne, France

Model Compression Classifier-Free Guidance Diffusion Model Distillation Diffusion Model Compression Teacher-Student Learning

Open paper AskPDF arXiv

Parameter compression of class-conditional diffusion models reveals an underexplored limitation in output-level distillation: the unconditional score branch remains unsupervised, leaving the classifier-free guidance gap underdetermined in the student. This gap, amplified at every denoising step, admits degenerate solutions where both branches collapse toward identical predictions, rendering guidance ineffective despite low output-level training loss. This paper introduces DASH, a dual-branch distillation framework that independently supervises both score branches, uniquely specifying target branch outputs for each training sample through independent branch constraints, with an anchor term regularising conditional predictions toward ground-truth noise. The framework further introduces TIRT Transfer, which copies the teacher's converged per-timestep importance curriculum into the student as a frozen prior, eliminating the need to relearn it within limited distillation budgets. Experiments on CIFAR-10 and CIFAR-100 demonstrate that 5.9x compression maintains quality within 4 FID points of the teacher at 50-step DDIM sampling, considerably outperforming training from scratch with guidance fidelity well preserved. Ablation studies confirm that unconditional supervision is the dominant contribution, accounting for over 60% of total distillation gain. Curriculum transfer and anchor regularisation provide complementary benefit, together validating dual-branch constraints as empirically essential for guidance-preserving compression.

Abstract

Diffusion Model Distillation

Explore similar work

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration

TamedFlow: Trajectory-Aware Distillation for Teacher-Assisted Diffusion Sampling