D-Scope: Decomposing and Steering Diffusion Transformers with Sparse Autoencoders

Xinyue Xu; Jiahao Zhang; Lijie Hu; Peter Hase; Hao Wang

cs.CVSep 30, 2026

D-Scope: Decomposing and Steering Diffusion Transformers with Sparse Autoencoders

Authors: Xinyue Xu, Jiahao Zhang, Lijie Hu, Peter Hase, Hao Wang

Organizations: Pivotal Research · Mohamed bin Zayed University of Artificial Intelligence · Schmidt Sciences · Stanford University · University of Illinois at Urbana-Champaign

Abstract

Sparse autoencoders (SAEs) reveal visual structure in diffusion transformers (DiTs), but interpreting a feature does not establish whether it can be used to control generation. We introduce D-Scope (Diffusion Scope), a framework that connects feature interpretation to generation control through shared visual evidence. D-Scope aggregates SigLIP2 embeddings of highly activating image patches into visual centroids. Matching target text descriptions against these visual centroids in the shared image-text embedding space then enables retrieval of individual features without per-feature text annotations. The underlying patches provide evidence for inspecting each selection, while spatially masked interventions test the corresponding decoder direction at varying strengths under fixed generation conditions. We characterize 150 SAEs across two model families and five layers, and introduce a benchmark of 100 target concepts with ten contexts each spanning under-specified and explicit-conflict conditions. Our empirical results show that high reconstruction fidelity can coexist with low dictionary utilization and limited visual-evidence coverage. Under per-case best-of-sweep strength selection, contrastive retrieval yields larger mean regional SigLIP2 gains than direct retrieval across the tested steering configurations, without consistently improving outside-region preservation. D-Scope provides an inspectable framework for evaluating sparse DiT features through their visual evidence and the effects of their decoder directions on generation. The demo is available at https://jiahaozhang-public.github.io/d-scope/.

Figures & tables

Panel of Figure 1 — (a) Varying strength for a fixed target query

Panel of Figure 2 — (b) Same source image with a different target query

Figure 2: D-Scope links the visual evidence of each SAE feature to its decoder direction. Columns follow Sec. 3.1 – 3.3 ; bottom panels show examples. (a) SAE training and dictionary metrics. (b) Activating patches; #1360 spans too few distinct source images to be indexed. (c) Offline: patches are pooled into a visual centroid $\mathbf{v}_{i}$ . Online: a target query, used directly or contrasted with a reference query, retrieves $i^{\star}$ . (d) Contrasting “red dress” with “dress” removes the object-dominated feature #14976. (e) The decoder direction of $i^{\star}$ is added within the target region, and the resulting image is compared with the unsteered image $I^{(0)}$ . (f) With #5824, raising $\rho$ from 0.15 to 0.30 does not increase $\Delta_{\mathrm{target}}$ but nearly doubles LPIPS-out.

Figure 3: Reconstruction fidelity alone does not characterize the feature pool available for visual-evidence retrieval. (a) Raw-space explained variance (EV) versus dictionary utilization. (b) Dictionary utilization versus visual-evidence coverage. (c) Percentage-point change in utilization from mean or geometric-median centering relative to the matched uncentered configuration; horizontal bars denote medians. (d) Median visual-evidence coverage across models and layers for each SAE family and input setting; outlined cells indicate the two highest median coverage values.

Panel of Figure 4 — Figure 4: Single-feature steering with D-Scope across generators and SAE dictionaries. (a) “Wooden table” interventions using Oracle Prompt, Oracle Dense, direct retrieval, and contrastive retrieval; dashed outlines mark the target region, and top activating patches are shown for the retrieved SAE features. (b) Regional target gain and outside-region change for middle-layer dictionaries, normalized by Oracle Dense. Faded and solid markers denote direct and contrastive retrieval, respectively, with matched dictionaries connected by lines. (c) Change in target gain from mean or geometric-median centering relative to the matched uncentered configuration; horizontal bars denote medians.

		SANA Student				Nitro Student				SANA Teacher
		Direct		Contrastive		Direct		Contrastive		Direct		Contrastive
Method	Input	$\Delta\uparrow$	LPIPS $\downarrow$	$\Delta\uparrow$	LPIPS $\downarrow$	$\Delta\uparrow$	LPIPS $\downarrow$	$\Delta\uparrow$	LPIPS $\downarrow$	$\Delta\uparrow$	LPIPS $\downarrow$	$\Delta\uparrow$	LPIPS $\downarrow$
No intervention	–	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$	$0.0{\scriptstyle\,\pm\,0.0}$
Oracle Prompt	–	$37.9{\scriptstyle\,\pm\,1.6}$	$392.8{\scriptstyle\,\pm\,8.6}$	$37.9{\scriptstyle\,\pm\,1.6}$	$392.8{\scriptstyle\,\pm\,8.6}$	$36.4{\scriptstyle\,\pm\,0.8}$	$289.6{\scriptstyle\,\pm\,6.0}$	$36.4{\scriptstyle\,\pm\,0.8}$	$289.6{\scriptstyle\,\pm\,6.0}$	$39.3{\scriptstyle\,\pm\,0.5}$	$421.1{\scriptstyle\,\pm\,10.7}$	$39.3{\scriptstyle\,\pm\,0.5}$	$421.1{\scriptstyle\,\pm\,10.7}$
Oracle Dense	–	$38.8{\scriptstyle\,\pm\,0.4}$	$147.4{\scriptstyle\,\pm\,2.5}$	$38.8{\scriptstyle\,\pm\,0.4}$	$147.4{\scriptstyle\,\pm\,2.5}$	$29.0{\scriptstyle\,\pm\,0.4}$	$59.5{\scriptstyle\,\pm\,1.1}$	$29.0{\scriptstyle\,\pm\,0.4}$	$59.5{\scriptstyle\,\pm\,1.1}$	$34.8{\scriptstyle\,\pm\,0.2}$	$219.2{\scriptstyle\,\pm\,7.6}$	$34.8{\scriptstyle\,\pm\,0.2}$	$219.2{\scriptstyle\,\pm\,7.6}$
L1	None	$20.9{\scriptstyle\,\pm\,0.5}$	$\mathbf{169.3}{\scriptstyle\,\pm\,2.2}$	$\mathbf{31.1}{\scriptstyle\,\pm\,0.5}$	$169.9{\scriptstyle\,\pm\,4.8}$	$20.0{\scriptstyle\,\pm\,0.1}$	$58.4{\scriptstyle\,\pm\,4.4}$	$\mathbf{22.6}{\scriptstyle\,\pm\,0.2}$	$\mathbf{54.3}{\scriptstyle\,\pm\,2.7}$	$18.8{\scriptstyle\,\pm\,0.5}$	$\mathbf{230.8}{\scriptstyle\,\pm\,12.0}$	$\mathbf{29.2}{\scriptstyle\,\pm\,0.4}$	$248.1{\scriptstyle\,\pm\,9.1}$

Table 1: Complete fine-grained steering benchmark. We evaluate all 15 middle-layer SAE configurations with direct and contrastive retrieval in three model environments; the SANA teacher reuses the student-trained SANA dictionaries. Region SigLIP improvement (

\Delta\uparrow

) and LPIPS-out (

\downarrow

) are reported as mean

\pm

sample SD across three generation-seed means, with both metrics and SDs multiplied by

10^{3}

. For each swept method and case-seed pair, steering strength maximizes Region SigLIP

\Delta

; LPIPS-out is measured on the same output. Oracle Prompt uses the target prompt, while Oracle Dense uses the source-target activation difference. Bold denotes the better retrieval strategy within each dictionary, model, and metric based on unrounded means.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Decision	Empirical finding	Recommendation
Dictionary selection	High reconstruction EV can coexist with low utilization and evidence coverage (Fig. 3 a,b). The three weakest steering configurations in each generation–retrieval setting also have the smallest visual indices (Tables 1 and 7 ).	Assess utilization, coherence, and coverage alongside reconstruction fidelity. Validate steering performance separately, since a larger visual index does not guarantee stronger control.
Bias initialization	Centering increases utilization in every matched comparison. For L1 and JumpReLU, it expands visual indices by $1.8$ – $4.4\times$ and improves mean target gain in 22 of 24 matched comparisons (Fig. 3 c; Tables 1 and 7 ).	Start with mean or geometric-median centering, then verify target alignment and preservation for the chosen generator and retrieval strategy.
SAE family	Centered TopK has the highest median evidence coverage. Mean-centered L1 achieves the highest mean target gain in five of six generator–retrieval combinations (Fig. 3 d; Table 1 ).	Start with centered TopK for visual inspection and mean-centered L1 for steering, checking preservation in both cases.
Layer	Layer 26 has lower input-space EV and equal or higher coverage than layer 2 in matched comparisons. Intermediate trends are not uniformly monotonic (Fig. 6 ).	Compare candidate layers jointly on reconstruction, utilization, coherence, and coverage.
Retrieval	Contrastive retrieval yields higher mean target gains in all 45 matched comparisons, without consistently improving preservation (Section D.2 ).	Use contrastive retrieval as a starting point for target alignment, and assess LPIPS-out separately.
Dictionary reuse	SANA student dictionaries retain positive target gains in the teacher, with higher LPIPS-out than in the student (Table 1 ).	Re-evaluate target alignment and preservation when transferring dictionaries to another generation setting.

Appendix

Table 2: Practical takeaways for building and using DiT SAEs. Steering findings use middle-layer dictionaries and per-case best-of-sweep strength selection.

Setting	Configuration
Experimental grid
Models	SANA-Sprint 0.6B; Nitro-1-PixArt 0.6B
Layers	2, 8, 14, 20, 26 (0-indexed)
Activation	Post-block residual, one-step generation
Width	1,152 input; 18,432 dictionary (16 $\times$ )
Input settings

Appendix

Table 3: SAE training configurations. All formal runs use a fresh initialization and one training seed.

Figure 5: Realized sparsity and visual evidence. (a,b) Sparsity distributions across layers and input settings. White dots mark medians, thick bars indicate interquartile ranges, and thin bars span observed ranges. Shading marks the tuning target

[64,100]

. TopK is shown as a single point at

\ell_{0}=64

. (c,d) DINOv3 coherence versus evidence coverage, with each point representing a dictionary with eligible visual evidence.

Figure 6: Dictionary quality across transformer layers. Columns correspond to generators. Rows show SAE input-space EV, dictionary utilization, and evidence coverage. Each box summarizes five input settings within a layer and SAE family. Boxes indicate interquartile ranges, center lines mark medians, whiskers span observed ranges, and dots show individual settings.

Layer	SAE	Input	EV (input)	EV (raw)	Mean $\ell_{0}$	Util. (%)	Cov. (%)	DINO	SigLIP
2	L1	None	0.9824	0.9824	94.0	3.4	3.3	0.195	0.726
		Mean center	0.9726	0.9726	115.0	24.5	25.0	0.195	0.732
		Geo. center	0.9738	0.9738	121.4	25.1	20.1	0.189	0.731
		LayerNorm	0.9823	N/A	95.3	3.4	4.9	0.188	0.724
		RMS	0.9803	0.9803	81.7	2.9	1.1	0.190	0.720
2	TopK	None	0.9956	0.9956	64.0	17.7	12.0	0.193	0.730

Appendix

Table 4: Complete SAE results for SANA-Sprint.

Layer	SAE	Input	EV (input)	EV (raw)	Mean $\ell_{0}$	Util. (%)	Cov. (%)	DINO	SigLIP
2	L1	None	0.9922	0.9922	92.7	3.4	3.3	0.193	0.782
		Mean center	0.9798	0.9798	67.3	9.9	7.6	0.195	0.781
		Geo. center	0.9801	0.9801	67.0	10.0	12.5	0.192	0.784
		LayerNorm	0.9909	N/A	83.3	2.9	3.3	0.192	0.780
		RMS	0.9899	0.9899	84.4	3.4	3.3	0.186	0.777
2	TopK	None	0.9997	0.9997	64.0	1.5	0.5	0.155	0.767

Appendix

Table 5: Complete SAE results for Nitro-1-PixArt.

Component	Setting
Prompts	First 10,000 prompts of the evaluation split (Table 3 ), disjoint from training and validation prompts; the first 1,000, with the same seeds, form the dictionary-evaluation set
Generation	$1024\times 1024$ , single step, guidance 0 (timestep 400 for Nitro-1-PixArt);
Activations	Layer-14 post-block residual of all visual tokens ( $32\times 32$ for SANA-Sprint, $64\times 64$ for Nitro-1-PixArt), mapped by $\mathcal{T}$ and encoded by the SAE
Evidence	$J=64$ images with the largest positive per-image maximum; one token per image; ties broken by image order
Patch	Square of four token cells centered on the selected token, clipped at the image boundary
Card	Patches in an $8\times 8$ grid by descending activation, for every feature with a positive image

Appendix

Table 6: Feature-catalog construction. Settings shared by all 30 layer-14 dictionaries.

SAE	Input	SANA-Sprint	Nitro-1-PixArt
L1	None	3,661 (19.9)	1,583 (8.6)
	Mean center	15,754 (85.5)	6,983 (37.9)
	Geo. center	15,525 (84.2)	6,976 (37.8)
	LayerNorm	3,188 (17.3)	1,422 (7.7)
	RMS	3,432 (18.6)	1,567 (8.5)
TopK	None	14,206 (77.1)	12,411 (67.3)

Appendix

Table 7: Visual index size. Indexed features out of 18,432 in each layer-14 dictionary, with percentages in parentheses. Retrieval considers only indexed features; the SANA teacher uses the SANA-Sprint indices.

Category	Targets	Cases	Example change
Color	20	200	Green dress $\rightarrow$ red dress
Material	20	200	Wooden chair $\rightarrow$ metal chair
Texture or pattern	20	200	Plain shirt $\rightarrow$ striped shirt
Local appearance	20	200	Straight hair $\rightarrow$ curly hair
Local state or geometry	20	200	Closed umbrella $\rightarrow$ open umbrella
Total	100	1,000

Appendix

Table 8: Fine-grained steering benchmark. Each target concept has five under-specified and five explicit-conflict contexts.

Category: Local state or geometry Target region: rose
Steering instruction: make the rose bloom
Target query $q$ : rose in full bloom
	Under-specified	Explicit-conflict
Source prompt $c$	a single rose in a garden with morning dew	a single closed rose bud in a garden with morning dew
Source state	Blooming state is unspecified.	A closed bud is explicitly specified.
Reference query $q_{\mathrm{base}}$	rose	rose bud

Appendix

Table 9: Matched rose-blooming cases. The scene and target query are shared, while the source state and reference query differ.

Figure 7: Target alignment by concept category and source context. Columns correspond to the three generation environments: SANA-Sprint (student), SANA (teacher), and Nitro-1-PixArt; the top and bottom rows show under-specified and explicit-conflict source contexts, respectively. Colors denote SAE families (L1, TopK, and JumpReLU), while open squares and filled circles denote Direct and Contrastive direction retrieval. Lines connect the two direction sources within the same SAE family and concept category. For each point, we first average over the five input configurations within each generation seed, then show the mean with sample SD across three seeds. Target-alignment improvement

\Delta_{\mathrm{target}}

is reported in units of

10^{-3}

Panel of Figure 8 — Figure 8: Learned SAE features as intervention and readout interfaces. (a) Suppressing a style-associated feature reduces the targeted style while retaining non-target styles and object content; mismatched interventions have little effect. (b) Image-level activation of a nudity-associated feature on held-out annotated examples, yielding an AUROC of $0.980$ . (c) Qualitative examples of targeted and mismatched style suppression for SANA and Nitro. (d) Spatial activation maps of the selected feature for examples with and without nudity.

Image domain	Top-1	Top-5	Released	Other	Matched	Top-5
	Validation		External Top-1			External
SANA-Sprint	60.2	83.3	9.9	51.1	77.5	95.1
Nitro-1-PixArt	51.7	74.8	9.5	52.6	66.7	88.4

Appendix

Table 10: Style-evaluator accuracy (%). Validation contains 2,040 images per generator and the external screen contains 1,224. External top-1 results compare the released classifier, the evaluator trained on the other generator, and the matched evaluator.

	SANA-Sprint			Nitro-1-PixArt
Style	Val.	Ext.	Base.	Val.	Ext.	Base.
Blossom Season	87.5	91.7	90.0	85.0	87.5	90.0
Comic Etch	85.0	100.0	90.0	92.5	100.0	100.0
Mosaic	90.0	91.7	92.5	82.5	100.0	100.0
Neon Lines	95.0	100.0	100.0	95.0	100.0	100.0
Pencil Drawing	90.0	100.0	100.0	97.5	100.0	100.0

Appendix

Table 11: Recognition accuracy (%) for the ten retained styles. Style inclusion requires at least 80% accuracy on both validation and external images for both generators. Baseline denotes recognition on the unmodified generations used in the suppression experiment.

Component	Setting
SAE site	Layer-14 post-block residual, 18,432 features
SAE families	L1, TopK, JumpReLU
Input settings	None, mean centering, geometric-median centering, LayerNorm, RMS scaling
Target styles	10 shared styles
Object contexts	Architectures, Birds, Cats, Dogs, Flowers, Horses, Human, Trees
Generation seeds	Five

Appendix

Table 12: Style-suppression evaluation setup.

Figure 9: Style-suppression trade-offs across SAE configurations. Point estimates show target-style suppression (UA), retained-style accuracy (IRA), and plain-prompt object accuracy (CRA) for all layer-14 dictionaries in SANA-Sprint and Nitro-1-PixArt; error bars denote 95% bootstrap intervals. Shaded rows indicate the configurations selected by maximizing UA subject to mean IRA and CRA of at least 90%; dashed lines mark the 90% retention reference.

SANA-Sprint
SAE family	Input convention	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	Control UA
L1	None	89.0	52.6	57.8	49.5
L1	Mean	38.0	96.6	92.5	3.5
L1	Geo. median	37.8	96.5	92.5	3.5
L1	LayerNorm	70.0	71.1	80.7	26.7
L1	RMS	29.0	85.2	93.0	15.7

Appendix

Table 13: Complete style-suppression results (%). Each row averages ten target styles, five generation seeds, and eight objects. UA, IRA, and CRA are measured after targeted intervention; control UA uses the cyclic mismatched feature. Bold rows maximize UA subject to mean IRA and CRA of at least 90% within each model. This choice uses the reported evaluation data. Intervals for all targeted metrics are shown in Figure 9 . Mean and Geo. median denote the two bias-initialization conventions.

	SANA-Sprint				Nitro-1-PixArt
Target style	UA	IRA	CRA	Control	UA	IRA	CRA	Control
Blossom Season	70.0	96.9	92.5	10.0	10.0	99.7	97.5	10.0
Comic Etch	12.5	97.2	92.5	10.0	22.5	98.6	97.5	0.0
Mosaic	100.0	96.9	92.5	7.5	32.5	95.8	97.5	0.0
Neon Lines	2.5	95.0	92.5	0.0	90.0	96.1	97.5	0.0
Pencil Drawing	100.0	96.4	92.5	0.0	87.5	77.8	100.0	0.0

Appendix

Table 14: Per-style results for the representative dictionaries (%). SANA uses TopK without input normalization; Nitro uses mean-centered JumpReLU. Each target evaluates 40 target-style, 360 retained-style, and 40 plain-object images per intervention condition. Control columns report mismatched-feature UA. Mean retention above 90% does not imply that every target style meets this constraint.

Panel of Figure 10 — Figure 10: Qualitative success cases (1/2). The targeted condition modifies one SAE feature; dictionary settings are shown at left. Success denotes visible attenuation of the target style, without requiring identical subject count or composition.

Panel of Figure 11 — Figure 11: Qualitative success cases (2/2). Additional selected successes across SAE input conventions and families. Conditions, image scale and seed match Figure 10 . These selected examples do not estimate the frequency of successful suppression.

Panel of Figure 12 — Figure 12: Qualitative failure cases: incomplete suppression. Residual brushwork, mosaic structure, saturated colour or etched appearance remains visible after intervention. These visual judgments are separate from classifier-defined UA: the classifier predicts a non-target style in these cases, but this does not establish complete visual removal.

Split	Generated	Nude	Non-nude	Uncertain	Used
Discovery (paired)	80	35	42	3	77
Test (paired)	80	35	43	2	78
Test (hard negatives)	40	0	40	0	40
All test	120	35	83	2	118
Total	200	70	125	5	195

Appendix

Table 15: Dataset composition and human annotation counts. Nude and non-nude denote human image labels; uncertain examples are excluded from binary analysis.

Component	Setting
Generator	SANA-Sprint 0.6B
Generation	$1024\times 1024$ , one inference step, guidance 0
SAE	Layer-14 mean-centered L1
Dictionary size	18,432 features
Spatial resolution	$32\times 32=1024$ tokens
Reporting pool size	$K=16$

Appendix

Table 16: Fixed configuration for the single-feature readout experiment. Generator and SAE parameters remain frozen throughout.

Panel of Figure 13 — Figure 13: Feature semantics and aggregation robustness. (a) High-activation patches for feature 4867. (b) Sensitivity to spatial aggregation. Feature 4867 is consistently recovered across pooling sizes, with held-out AUROC between 0.979 and 0.980.

Panel of Figure 14 — Figure 14: Matched nude-clothed examples. Each adjacent nude-clothed pair shares the same scene and seed. Each example shows the generated image (left) and its feature-4867 activation overlay (right).

Panel of Figure 15 — Figure 15: Hard-negative examples. Representative non-nude examples with potentially confounding content. Each example shows the generated image (left) and its feature-4867 activation overlay (right).

Explore similar work

Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models
Jun 30, 2026Enrico Cassano, Riccardo Renzulli, Rayyan Ahmed +2Diffusion Models Concept Erasure
University of Turin, Computer Science Department, Italy · LTCI, Télécom Paris, Institut Polytechnique de Paris, France
Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformers
May 28, 2026Zihao Xue, Yan Wang, Zhen Bi +7Diffusion Transformers Transformer Architectures
1Huzhou Normal University · 2Alibaba Group · University of Science and Technology of China +2
Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation
Jul 3, 2026Chaofan Gan, Zicheng Zhao, Yuanpeng Tu +6Diffusion Transformers Representational Capacity
Shanghai Jiao Tong University, Shanghai, China · The University of Hong Kong, Hong Kong, China · Monash University, Melbourne, Australia +2

Jun 30, 2026, cs.CVJ/K move · Enter open · S save

Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models

Enrico Cassano, Riccardo Renzulli, Rayyan Ahmed, Marco Grangetto+1

University of Turin, Computer Science Department, Italy · LTCI, Télécom Paris, Institut Polytechnique de Paris, France

Diffusion Models Concept Erasure

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) have recently been proposed as interpretable tools for concept-level manipulation, under the assumption that isolated features can serve as controllable intervention points. In this work, we systematically evaluate this assumption in the context of object erasure and steering in diffusion models. We show that while SAEs reliably detect and localize semantic concepts within diffusion model activations, direct intervention in their latent space frequently induces out-of-distribution activations, resulting in severe visual artifacts. To disentangle detection from intervention, we use SAE activations purely as semantic detectors to identify image regions containing the target object, and replace those patch embeddings with the ones that do not contain it. This detection-based replacement preserves the diffusion model's activation statistics and produces significantly cleaner erasure results than latent steering. Our findings reveal a fundamental gap between concept detection and concept intervention in diffusion models: monosemantic or sparse features are not inherently suitable as control knobs for steering. These results position SAEs as powerful interpretability tools for analyzing generative models, but highlight important limitations when used for direct manipulation, such as unlearning.

Abstract

Explore similar work

Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models

Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformers

Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation