When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities

Wenshuai Yao; Wenyong Zhou

cs.CVAug 20, 2026

When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities

Authors: Wenshuai Yao, Wenyong Zhou

Organizations: School of Integrated Circuits, Peking University, Beijing, China · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China

Abstract

Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers dominated by linear operations. Analog compute-in-memory (CIM) can alleviate these costs by executing linear operations within weight-storing memory arrays. However, CIM nonidealities perturb effective weights, with errors accumulating along the state-dependent denoising trajectory; their interaction with classifier-free guidance (CFG) remains underexplored. In this paper, we characterize the impact of analog CIM nonidealities on DiT sampling. Although conditional and unconditional predictions can each remain close to their clean counterparts, their difference (the CFG residual) is disproportionately attenuated and rotated. Identifying this residual as a controllable failure channel, we propose a retraining-free, sampler-side recalibration that adjusts only the CFG scale for a given CIM condition. Trajectory-level analysis shows that moderate recalibration strengthens the target-oriented component preserved in the distorted residual, enabling earlier commitment to a prompt-consistent semantic region. In contrast, excessive guidance amplifies the full noisy residual and degrades quality, resulting in a finite, noise-dependent optimum. Extensive experiments on PixArt-Sigma, PixArt-alpha, and DiT-XL/2 show that the optimal guidance scale increases with CIM noise. Using 30,000 samples per condition, guidance recalibration consistently restores generation quality across simulated CIM mappings, closing at least 87% of the CIM-induced FID gap at a CIM noise level of 0.20. It reduces FID from 59.22 to 20.49 on PixArt-Sigma, 72.37 to 21.12 on PixArt-alpha, and 20.89 to 6.62 on DiT-XL/2.

Figures & tables

Figure 1: Qualitative comparison across four prompts. Each triplet shows clean generation, CIM with clean-selected guidance, and CIM with recalibrated guidance.

Figure 2: Hybrid analog CIM deployment. (a) CIM-based matrix multiplication. (b) DiT mapping: linear layers are executed on CIM arrays, while other operations remain digital.

Figure 3: PixArt- $\Sigma$ generations at $1024\times 1024$ resolution under increasing CIM nonidealities at the clean-selected scale $w_{0}=2.0$ .

Figure 4: CFG and residual diagnostics under CIM nonidealities. (a) Branch perturbations. (b) Residual rotation. (c) Clean-aligned decomposition. (d–e) Timestep statistics at

\sigma_{\mathrm{CIM}}=0.20

; shading indicates 95% confidence intervals.

Figure 5: Conceptual sampling trajectories under (a) clean inference, (b) CIM inference with the clean-selected scale

w_{0}

, and (c) CIM inference with the recalibrated scale

w^{\star}

. Shaded contours indicate semantic basins; arrows denote base and guidance updates. Each trajectory segment is the vector sum of the two component arrows.

Model and benchmark	$\sigma_{\mathrm{CIM}}$	CFG $w$	FID $\downarrow$	KID $\times 10^{3}\downarrow$	Alignment $\uparrow$	Precision $\uparrow$	Density $\uparrow$	Coverage $\uparrow$
PixArt- $\Sigma$ ( $512^{2}$ , COCO)	0 (clean)	1.5	20.51	7.75	30.34	0.550	0.654	0.595
	0.05	1.5 ( $=w_{0}$ )	20.31	7.68	30.31	0.551	0.647	0.582
	0.10	1.5 $\!\rightarrow\!$ 2.0	21.12 $\!\rightarrow\!$ 19.98	8.27 $\!\rightarrow\!$ 7.05	30.09 $\!\rightarrow\!$ 30.69	0.513 $\!\rightarrow\!$ 0.552	0.575 $\!\rightarrow\!$ 0.654	0.556 $\!\rightarrow\!$ 0.580
	0.15	1.5 $\!\rightarrow\!$ 2.5	28.62 $\!\rightarrow\!$ 20.18	12.85 $\!\rightarrow\!$ 6.87	29.37 $\!\rightarrow\!$ 30.79	0.432 $\!\rightarrow\!$ 0.543	0.423 $\!\rightarrow\!$ 0.628	0.477 $\!\rightarrow\!$ 0.572
	0.20	1.5 $\!\rightarrow\!$ 4.5	59.22 $\!\rightarrow\!$ 20.49	30.41 $\!\rightarrow\!$ 6.49	27.32 $\!\rightarrow\!$ 31.17	0.248 $\!\rightarrow\!$ 0.527	0.203 $\!\rightarrow\!$ 0.602	0.292 $\!\rightarrow\!$ 0.560
PixArt- $\alpha$ ( $512^{2}$ , COCO)	0 (clean)	1.5	21.63	8.21	29.98	0.545	0.624	0.567

Table 1: Large-scale population-level evaluation across three diffusion transformers and multiple simulated CIM mappings, with 30,000 samples per setting. Noisy entries report

w_{0}\!\rightarrow w_{\sigma}^{\star}

; a single value is shown when the selected scales coincide. Underlining marks the recalibrated endpoint, and bold marks improvement over the clean-guidance baseline.

Figure 6: Guidance response on a 3,000-prompt PixArt-

\Sigma

sweep. (a) FID and (b) CLIPScore versus

w

. (c) FID at

w_{0}

and

w_{\sigma}^{\star}

Figure 7: Qualitative comparisons under CIM noise. Left: PixArt- $\Sigma$ text-to-image generation at $\sigma_{\mathrm{CIM}}=0.15$ , using $w_{0}=1.5$ and $w_{\sigma}^{\star}=2.5$ . Right: SD3.5 Medium text rendering at $\sigma_{\mathrm{CIM}}=0.10$ , using $w_{0}=1.5$ and $w_{\sigma}^{\star}=3.0$ .

Figure 8: Timestep semantic-entry analysis. (a) Ten evenly spaced checkpoints from 20-step predicted-clean trajectories for two prompts; rows show clean inference, CIM with $w_{0}$ , and CIM with $w_{\sigma}^{\star}$ . (b–d) Prompt retrieval top-1 accuracy across all steps. (e) Persistent semantic-entry indices at $\theta=0.30$ .

Method	$\sigma_{\mathrm{CIM}}=0.15$	$\sigma_{\mathrm{CIM}}=0.20$
Clean-best CFG	44.93	75.94
CFG Rescale	40.40	40.72
APG	39.21	41.08
C 2 FG	38.11	39.52
Limited-Interval Guidance	37.28	38.76
Fixed recalibration (ours)	36.60	37.61

Table 2: FID in the shared 3,000-prompt PixArt-

\Sigma

comparison of sampler-side guidance controls.

(a) Endpoint counterfactuals
Metric	Clean	CIM $w_{0}$	CIM $w^{\star}$	Restore $u$	Restore $g$	Remove $\eta$	Restore $\alpha$
FID $\downarrow$	36.19	75.41	36.64	42.83	68.34	127.75	44.49
CLIP $\uparrow$	30.45	27.27	31.27	28.52	28.87	23.23	30.35
DINO-clean $\uparrow$	1.000	0.463	0.673	0.621	0.631	0.261	0.699

Table 3: Residual-surgery analysis at

\sigma_{\mathrm{CIM}}=0.20

over 3,000 prompts. (a) Endpoint counterfactuals. (b) FID along intervention paths from untreated CIM (

0

) to the full intervention (

1

Explore similar work

ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware
Sep 1, 2026Yuannuo Feng, Yizhe Chen, Wenshuai Yao +4Compute-in-Memory Diffusion Sampling
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Zhicun Research Lab, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China +1
Guidance Breaks the Fitted Operator: A Terminal-Fitted Repair for Classifier-Free Guidance
Jul 8, 2026Shiheng ZhangClassifier-Free Guidance Diffusion Sampling
University of Washington
Trajectory-Consistent Calibration for Cache-Accelerated Diffusion Models
May 24, 2026Mingyu Liang, Dingkun Xu, Jingwei XuDiffusion Transformer Diffusion-Based Image Generation
Laboratory for Novel Software Technology, Nanjing University, China

Sep 1, 2026J/K move · Enter open · S save

ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware

Yuannuo Feng, Yizhe Chen, Wenshuai Yao, Yuxin Xie+3

School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Zhicun Research Lab, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China+1

Compute-in-Memory Diffusion Sampling Diffusion Models

Open paper AskPDF arXiv

Diffusion models achieve strong image generation quality but incur high iterative denoising costs. Analog compute-in-memory (CIM) can accelerate matrix-vector multiplications, yet spatial memory variations perturb weights and accumulate during sampling. Unlike conventional neural networks, diffusion models' temporal sensitivity to hardware noise remains underexplored. We investigate diffusion inference using a noise model calibrated and validated against measurements collected from multiple physical CIM chips. Our results show that the early, high-noise denoising stage is substantially more vulnerable than the final refinement stage. A first-order trajectory analysis attributes this behavior to the repeated propagation of correlated prediction errors induced by a fixed hardware mapping. Based on this observation, we propose ASSERT, a training-free sampler that uses higher stochasticity early and smoothly transitions to deterministic denoising. The injected stochasticity changes subsequent activation trajectories and thereby reduces their alignment with persistent spatial errors. Across the evaluated settings, ASSERT achieves up to 2.58

\times

lower FID than deterministic DDIM on high-resolution datasets and 7.68

\times

lower FID in the CIFAR-10 step-count study, without changing model parameters or the number of network evaluations.

Abstract

Explore similar work

ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware

Guidance Breaks the Fitted Operator: A Terminal-Fitted Repair for Classifier-Free Guidance

Trajectory-Consistent Calibration for Cache-Accelerated Diffusion Models