LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization

Lingyi Long; Congyue Zhao; Xiaoou Li; Yanmin Qian; Tao Huang

cs.CVOct 13, 2026

LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization

Authors: Lingyi Long, Congyue Zhao, Xiaoou Li, Yanmin Qian, Tao Huang

Organizations: Shanghai Jiao Tong University · VUI Labs Research

Abstract

Post-hoc linearization is an attractive way to accelerate pretrained Video Diffusion Transformers (Video DiTs), yet existing methods largely treat the quality loss induced by attention replacement as a recovery problem. We show that a more fundamental bottleneck lies upstream: initialization. Random initialization discards pretrained knowledge, whereas manual weight reuse transfers it through rigid and incomplete source--target correspondences. We introduce LinMorph, which recasts Video DiT linearization as a learnable softmax-to-linear weight transformation problem. Instead of copying predefined counterparts, LinMorph learns a cross-parameter morphism that lets each target Gated DeltaNet (GDN) parameter draw jointly from pretrained query, key, value, and output projections, including for parameters with no direct softmax counterpart. A single morphism is shared across layers and learned through local teacher alignment, while producing layer-specific initializations from their respective pretrained weights. The resulting weights are then materialized as independent GDN modules for progressive distillation and joint recovery.Since the morphism operates in weight space rather than token space, it can be learned on short, low-resolution clips and directly reused at larger video scales. Across pretrained Video DiTs, LinMorph enables markedly faster recovery and better final generation quality than random initialization and heuristic weight reuse. Results show that efficient linearization depends not only on how a converted model is recovered, but critically on how pretrained knowledge is transferred before recovery. Project page: https://lonelyling.github.io/LinMorph.

Figures & tables

Figure 1: Comparisons of different weight initialization methods. (a) Random initialization discards pretrained knowledge, while (b) manual copying relies on partial one-to-one correspondences. (c) Our LinMorph learns a cross-parameter transformation that initializes both matched and target-specific linear-attention parameters. (d) The resulting initialization yields substantially faster recovery and lower KD loss.

Figure 2: Overview of LinMorph. A shared weight morphism is first learned from the frozen softmax teacher and used to generate GDN initialization weights. After materialization, the mapper is discarded and the GDN modules are progressively introduced and recovered by prediction distillation. The resulting hybrid Video DiT is finally fine-tuned end-to-end without the teacher.

Figure 3: Shared weight mapper. Shown for a target parameter

j

. Alignment and construction are target-dependent; residual correction applies only to

q,k,v,o,g,\beta

Figure 4: Learned cross-parameter structure. Left: source-mixing coefficients. Right: block-NMSE degradation after restricting or permuting the learned source assignments.

Initialization	Block NMSE $\downarrow$	Block Cos. $\uparrow$
Random	0.09483	0.96211
Manual copying	0.09446	0.96209
Base	0.01360	0.99321
Base + residual	0.01239	0.99381

Table 1: Pre-recovery initialization fidelity. Evaluated on 64 HQ videos, 30 blocks, and five diffusion timesteps.

Init.	Upd.	Block NMSE $\downarrow$	Block Cos. $\uparrow$	Output NMSE $\downarrow$	Output Cos. $\uparrow$
Random	1,500	0.1522	0.9315	0.0250	0.9873
Manual	1,500	0.1359	0.9379	0.0230	0.9884
Random	3,000	0.1168	0.9449	0.0168	0.9915
Manual	3,000	0.0905	0.9565	0.0137	0.9931
LinMorph	1,500	0.0525	0.9738	0.0101	0.9949

Table 2: Controlled initialization ablation on two Video DiT backbones. Within each backbone, all methods use the same target architecture, recovery recipe, and evaluation protocol.

Model	VB1 $\uparrow$	VB2 $\uparrow$	Model	VB1 $\uparrow$	VB2 $\uparrow$
General video-generation models
Wan2.1-1.3B a	82.49	56.02	CogVideoX-1.5-5B c	82.01	53.40
MoAlign-2B b	81.30	55.90	CogVideoX-2B b	80.60	54.90
VideoREPA-2B b	80.50	55.00	VideoCrafter2 d	80.44	–
Efficient video-generation models
Wan-DMD (2 steps) a	83.28	56.67	PyramidalWan-DMD-PT ⋆,a	82.72	51.75

Table 3: Overall generation quality on VBench and VBench-2.0. Scores are on a 0–100 scale; higher is better. LinMorph uses 50-step sampling. † denotes an approximate score read from the

480\times 832

panel of Figure 5 in Ghafoorian et al. [9] . Reference provenance and evaluation details are provided in Appendix C.6 .

Dimension	LinVideo	LinMorph (21 GDN)	$\Delta$
Imaging quality	66.07	69.04	$+2.97$
Aesthetic quality	59.41	63.90	$+4.49$
Motion smoothness	98.19	98.58	$+0.39$
Dynamic degree	59.67	60.56	$+0.89$
Background consistency	96.72	97.16	$+0.44$
Subject consistency	95.12	95.48	$+0.36$

Table 4: Dimension-wise VBench comparison with LinVideo. All eight dimensions in the Wan2.1-1.3B panel of Huang et al. [10] are included. Scores are multiplied by 100; higher is better. Bold marks the higher score in each row. LinVideo converts 16/30 blocks; LinMorph converts 21/30.

Schedule	Total updates	Block NMSE $\downarrow$	Block Cos. $\uparrow$
One-shot $0\rightarrow 21$	4,500	0.18406	0.90358
Progressive $0\rightarrow 7\rightarrow 14\rightarrow 21$	4,500	0.17931	0.90728

Table 5: Block alignment after 21-GDN distillation. Both schedules use the same mapped initialization bank and 4,500 total optimizer updates. Metrics use student depth rollout, before final teacher-free full-model fine-tuning. Complete prediction diagnostics appear in Appendix C.3 .

Figure 5: 21-GDN inference speedup. Attention/backbone: median and IQR over three paired rounds; end-to-end: one pair. Protocol: Appendix C.7 .

Transformation	Block NMSE $\downarrow$	Block Cos. $\uparrow$	Output NMSE $\downarrow$	Output Cos. $\uparrow$
Manual	0.1002 / –	0.9614 / –	4.1482 / –	0.2395 / –
FullMix	0.0177 / 0.0161	0.9912 / 0.9920	0.7810 / 0.6390	0.6456 / 0.7051
PreHeadMix	0.0221 / 0.0207	0.9890 / 0.9897	1.5084 / 1.2746	0.5176 / 0.5745
PostHeadMix	0.0175 / 0.0162	0.9913 / 0.9919	0.7376 / 0.6166	0.6625 / 0.7135
PreChannelMix	0.0183 / 0.0166	0.9908 / 0.9917	0.9758 / 0.6926	0.6071 / 0.6893
PostChannelMix	0.0207 / 0.0181	0.9897 / 0.9910	1.4928 / 0.8154	0.4759 / 0.6536

Table 6: Comparison of weight-transformation designs. Entries report results without / with residual refinement. Block metrics use teacher-forced inputs; output metrics evaluate the cascade of all 30 mapped modules. Evaluation uses 64 videos and five diffusion timesteps. Bold marks the best result in each metric.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\ell$	Transformer block index.
$s,\mathcal{S},S$	Source index, source set $\{Q,K,V,O\}$ , and its size $S=4$ .
$j,b,i$	Target-parameter index, mixing-group index, and channel index.
$t,\sigma_{t},c$	Diffusion scheduler timestep, corresponding noise level, and text condition.
$D,H_{s},d_{s}$	Teacher width, head count, and head width, with $D=H_{s}d_{s}$ .
$D_{g},H,d_{k},d_{v}$	GDN hidden width, recurrent head count, and per-head key/value widths.

Appendix

Table 7: Notation used throughout the manuscript.

Target	Native shape	Source reduction	Fixed base
Decay projection $a$	$H\times D_{g}$	$U_{s}$	Zero
Update projection $\beta$	$H\times D_{g}$	$U_{s}$	Zero
$A_{\log}$ , $\mathrm{dt}$ bias	$H$	$r_{s}^{\mathrm{head}}$	Native initialization
Short-conv weights	$C_{j}\times 1\times K$	$r_{s}^{\mathrm{channel}}$	Identity kernel
Output-norm scale	$d_{v}$	$r_{s}^{\mathrm{norm}}$	Ones

Appendix

Table 8: Auxiliary mappings in FullMix. Shapes use stored-weight conventions. The convolution channel count

C_{j}

C_{k}

for

q,k

and

C_{v}

for

v

;

K

is the short-convolution kernel length.

Parameter set	Parameters (M)
Shared base mapper	84.93
Shared residual refinement (base frozen)	151.07
Single native GDN (reference)	151.24
Single GDN with bridges (reference)	160.68
7 GDN modules with bridges	1,124.75
14 GDN modules with bridges	2,249.50

Appendix

Table 9: Parameter-count comparison for the specified architecture. Mapping rows count the parameters trainable in each separate fitting step. Progressive recovery trains all selected GDN modules and their bridges. Full-DiT fine-tuning additionally updates the remaining model components. Counts are not measurements of wall-clock time.

Setting	Wan2.1	Wan2.2
Source backbone	T2V-1.3B	TI2V-5B, T2V mode
Replacement blocks	$\{0,4,8,12,16,20,24\}$	$\{0,5,10,15,20,25\}$
Recovery data	HQ videos	OpenVid
Training clip shape	$81\times 480\times 832$	$81\times 736\times 1280$
Training devices	8 GPUs	8 GPUs
Per-device batch	1	1

Appendix

Table 10: Recovery settings for the initialization ablations. These are not the mapper-fitting or final teacher-free fine-tuning settings.

Figure 6: Wan2.2 recovery with six GDN blocks. Total training recovery loss with EMA-100 smoothing. All methods use the same backbone-specific recovery objective.

Schedule	Block NMSE $\downarrow$	Block Cos. $\uparrow$	Output NMSE $\downarrow$	Output Cos. $\uparrow$
One-shot	0.18406	0.90358	0.03415	0.98260
Progressive	0.17931	0.90728	0.03868	0.98026

Appendix

Table 11: Complete diagnostics before final joint fine-tuning. Both 21-GDN models use 4,500 distillation updates. Block metrics measure depth-rollout outputs at replaced layers; output metrics measure the final DiT velocity prediction on the same fixed evaluation inputs.

Backbone	Initialization	Updates	Attn. NMSE $\downarrow$	Attn. Cos. $\uparrow$	Norm ratio
Wan2.1	LinMorph	1500	0.43849	0.75766	0.86764
Wan2.1	Manual	1500	1.02142	0.26047	0.36950
Wan2.1	Random	1500	1.10699	0.08895	0.26422
Wan2.1	Random	3000	1.09900	0.15997	0.39579
Wan2.2	LinMorph	1250	0.96955	0.48265	0.92147
Wan2.2	LinMorph	1500	1.00650	0.47709	0.94721

Appendix

Table 12: Teacher-forced attention diagnostics for initialization. Rows shared with Table 2 use the same checkpoints. An additional 1,500-update LinMorph checkpoint is included for Wan2.2. A norm ratio close to zero indicates an attenuated student attention output; low NMSE alone need not imply good directional agreement.

Evaluation size	Mapper	Attention NMSE $\downarrow$	Attention Cos. $\uparrow$	Block NMSE $\downarrow$	Block Cos. $\uparrow$
$17\times 256\times 448$	Base	0.29951	0.83543	0.01242	0.99379
$17\times 256\times 448$	Base + residual	0.27237	0.85161	0.01121	0.99439
$81\times 480\times 832$	Base	0.34784	0.80592	0.01549	0.99231
$81\times 480\times 832$	Base + residual	0.32078	0.82269	0.01418	0.99296

Appendix

Table 13: Reuse across video shapes without mapper refitting. Both variants are fitted at

17\times 256\times 448

; each is then evaluated at both sizes. Each row averages 9,600 teacher-forced local comparisons (64 videos, five timesteps, 30 blocks). Residual denotes the separately fitted native-shape correction, not post-transfer optimization.

Figure 7: Layerwise local alignment across video sizes. The same low-resolution-trained base and residual-refined mappers are evaluated at two video sizes. Block indices are zero-based; all 30 source layers participated in mapper fitting. Lines connect measured layers and are not uncertainty estimates.

Layers sharing a mapper	Block NMSE $\downarrow$	Block Cos. $\uparrow$
1 layer (independent mappers)	0.20388	0.94327
6 layers (group-shared mappers)	0.20734	0.94558
30 layers (one shared mapper)	0.01560	0.99221

Appendix

Table 14: Cross-layer sharing and generalization. Block NMSE and cosine are averaged over five evenly spaced evaluation blocks (0, 6, 12, 18, and 24). For the one-layer setting, five separate mappers are each trained on one corresponding block; for the six-layer setting, five separate mappers are trained on consecutive groups

[0,5]

[6,11]

[12,17]

[18,23]

, and

[24,29]

. Each mapper is evaluated at all five evaluation blocks. The table then reports the equal-weight mean across the five mappers. The 30-layer setting trains one mapper on all layers and evaluates it at the same five blocks.

Frames	Wan (s)	21 GDN (s)	Speedup	Wan (GiB)	21 GDN (GiB)
81	166.9	213.2	0.78 $\times$	15.50	28.40
241	959.0	753.7	1.27 $\times$	18.42	44.61
321	1604.7	1094.0	1.47 $\times$	19.87	52.72

Appendix

Table 15: Full-generation latency and peak allocated memory. All runs use

480\times 832

, batch one, BF16, 50 denoising steps, and CFG six. Each latency is one timed generation after one excluded first generation. Memory is PyTorch peak allocated memory in GiB, not total board memory.

Diffusion Model Inference Acceleration

Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.

See all 509 papers

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Sep 28, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9Neural Network Initialization Vision Transformer
Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore +3
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Rotary Positional Embeddings Low-Rank Attention
Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1
Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Sep 17, 2026Haocheng Xi, Yiming Xie, Hexu Zhao +8Video Diffusion Models Streaming Video Generation
University of California, Berkeley · Impossible, Inc. · University of Texas at Austin

Sep 28, 2026J/K move · Enter open · S save

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Shiqi Huang+8

Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · Nanyang Technological University · ST Engineering Geo-Insights, Singapore+3

Neural Network Initialization Vision Transformer Efficient ViTs Transfer Learning Knowledge Distillation

Open paper AskPDF arXiv

Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the attention's token routing behavior can be recovered through distillation with a proper loss design, letting linear ViTs reduce the gap and even match Softmax ones. In contrast, the MLP weights, which carry the learned representation, are operator-agnostic: they can be transferred by simple direct copying, which already carries most of the benefit of the pre-trained weights. Thus, copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones. These findings hold consistently across various linear ViT variants, different model sizes, and diverse datasets. We hope this study deepens the understanding of reusing pre-trained weights across attention operators: copy what stays the same and distill what differs, to recover the benefit across the Softmax-to-linear boundary.

Abstract

Diffusion Model Inference Acceleration

Explore similar work

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation