Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels

Alexandra Zelenin; Alexandra Zhuravlyova

cs.LGMar 23, 2026

Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels

Authors: Alexandra Zelenin, Alexandra Zhuravlyova

Abstract

Weight-Decomposed Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction, but its forward pass requires the row-wise norm of W + sBA, a computation that every major framework we surveyed implements by materializing the dense [d_out, d_in] product BA. At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16, making high-rank DoRA costly and often infeasible on common single-GPU setups once hundreds of adapted modules and checkpointing are involved. We present two systems contributions. A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r^2) intermediates, eliminating the dense product. Fused Triton kernels collapse the four-kernel DoRA composition into a single pass, reducing memory traffic by about 4x and using a numerically stable form that avoids catastrophic cancellation in the near-unity rescaling regime where magnitude scales concentrate in practice. Across six 8-32B vision-language models (VLMs) on three NVIDIA GPUs (RTX 6000 PRO, H200, B200) at r = 384 in bf16, the fused implementation is 1.5-2.0x faster than Hugging Face PEFT's DoRA implementation for inference and 1.5-1.9x faster for gradient computation (optimizer step excluded), with up to 7 GB lower peak VRAM. Microbenchmarks on six GPUs spanning four architecture generations (L40S, A100, RTX 6000 PRO, H200, B200, B300) confirm 1.5-2.7x compose-kernel speedup. Final-logit cosine similarity exceeds 0.9999 across all model/GPU pairs, and multi-seed training curves match within 7.1 x 10^-4 mean per-step loss delta over 2000 steps.

Figures & tables

Concrete : $d_{\text{out}}\!=\!d_{\text{in}}\!=\!8192$ , $r\!=\!512$ , fp32
	PEFT	Factored (ours)
Rank-dependent persistent	$\mathcal{O}(d_{\text{in}}^{2}+d_{\text{out}}d_{\text{in}})$	$\mathcal{O}(d_{\text{out}}r+r^{2})$
Chunk buffer (transient)	Included above	$[d_{\text{out}},\text{cs}]$ per chunk †
Per-module temps	3–4 dense $[d_{\text{out}},d_{\text{in}}]$	$\mathbf{U}\![d_{\text{out}},r]$ + $\mathbf{G}\![r,r]$
Identity matrix	Every forward call	Never created
Theory: dense (B@A)	256 MB	N/A

Table 1: The factored norm reduces rank-dependent persistent memory by

15

× at

d\!=\!8192

r\!=\!512

in fp32. Measured reductions are smaller (

3.2

×) because allocator deltas include the rank-independent base-norm transient (§ 2.3 ).

Figure 1: The stable compose form achieves

3.0

× lower peak error near

g\approx 1

(bf16,

d_{\text{out}}=8192

d_{\text{in}}=2048

). The naive form

g\odot(s\cdot\text{lora}+\text{base})-\text{base}

exhibits catastrophic cancellation; the stable form and fused kernel both remain near the bf16 quantization floor. Reference: fp64.

Figure 2: Three-tier dispatch: fused backward for training (Tier 1), fused forward for inference (Tier 2), eager fallback for CPU, no-Triton, or sub-crossover shapes (Tier 3).

Tier	Path	When	Tradeoff
1	Fused Backward	Training + CUDA + Triton + auto-gate/force-on	Fastest above crossover; saves inner for bwd
2	Fused Forward	Inference + CUDA + Triton	Speed without bwd memory
3	Eager Fallback	CPU / no Triton / force-off / sub-crossover	Universal compatibility

Table 2: Dispatch tiers and their selection criteria.

GPU	Arch	Memory	BW (TB/s)	Scope
L40S	SM89 Ada	48 GB GDDR6	0.86	Micro
A100-SXM4	SM80 Ampere	80 GB HBM2e	2.04	Micro
RTX 6000 PRO	SM120 Blackwell	96 GB GDDR7	1.60	Micro+Model
H200	SM90 Hopper	141 GB HBM3e	4.80	Micro+Model
B200	SM100 Blackwell	192 GB HBM3e	7.70	Micro+Model
B300	SM103 Blackwell	268 GB HBM3e	7.70	Micro

Table 3: Benchmark hardware. “Micro”: microbenchmark coverage. “Model”: full model-level gradient-computation and inference benchmarks.

	Speedup vs. PEFT DoRA			Speedup vs. Eager
Model	RTX	H200	B200	RTX	H200	B200
Qwen2.5-VL-32B	OOM	1.73×	1.74×	OOM	1.20×	1.22×
Qwen3-VL-32B	OOM	1.66×	1.67×	OOM	1.18×	1.21×
Qwen3.5-27B	1.51×	1.57×	1.57×	1.22×	1.21×	1.23×
Gemma3-27B	1.53×	1.61×	1.56×	1.20×	1.19×	1.21×
Mistral-Sm-24B	1.66×	1.87×	1.87×	1.22×	1.21×	1.23×

Table 4: Gradient-computation speedup on 8–32B VLMs (

r\!=\!384

, bf16, seq=4096, bs=1, ga=8, loss_tokens=1024 , 20 repeats). The HF PEFT DoRA baseline takes 46–87% longer per iteration than fused. 32B models OOM on RTX 6000 PRO (96 GB) under all configurations. See Table 5 for absolute times.

	Fused (s)			Eager (s)			PEFT (s)
Model	RTX	H200	B200	RTX	H200	B200	RTX	H200	B200
Qwen2.5-VL-32B	OOM	30.9	23.3	OOM	37.1	28.5	OOM	53.6	40.4
Qwen3-VL-32B	OOM	33.4	25.4	OOM	39.5	30.6	OOM	55.4	42.3
Qwen3.5-27B	39.5	25.4	19.4	48.1	30.6	23.9	59.5	39.8	30.5
Gemma3-27B	41.6	27.5	21.4	50.0	32.7	25.8	63.6	44.1	33.4
Mistral-Sm-24B	32.5	20.9	15.9	39.7	25.2	19.6	53.9	39.1	29.8

Table 5: Absolute gradient-computation time (seconds). Each iteration covers 8 gradient-accumulation micro-steps; 32 768 tokens total. Standard deviations

\leq 0.13

s (CV

<

1.7%).

Figure 3: Gradient-computation speedup across six VLMs on three GPUs (bf16,

r\!=\!384

, seq=4096). (a) Fused vs. the HF PEFT DoRA baseline:

1.46

–

1.87

×. (b) Fused vs. eager:

1.18

–

1.24

×. 32B models OOM on RTX 6000 PRO under all configurations.

Figure 4: Inference speedup:

1.5

–

2.0

× over the HF PEFT DoRA baseline. All six models run on all three GPUs, including 32B on RTX 6000 PRO (96 GB) that OOM during gradient computation.

		vs. PEFT DoRA		vs. Eager
Model	Rank	Grad.	Infer.	Grad.	Infer.
Qwen3.5-27B	384	1.57×	1.65×	1.21×	1.16×
	512	1.61×	1.68×	1.18×	1.14×
	768	1.53×	1.59×	1.15×	1.11×
Qwen3-VL-32B	384	1.66×	1.78×	1.18×	1.14×
	512	1.70×	1.82×	1.16×	1.12×

Table 6: Speedup vs. the HF PEFT DoRA baseline grows with rank; speedup vs. eager decreases modestly (H200, bf16, seq=4096, 20 repeats).

Figure 5: Dense (B@A) position in the eager-to-fused gap (0% = eager, 100% = fused). Negative values: dense (B@A) is slower than eager. The benefit is GPU-bandwidth-sensitive; the factored approach is robust.

Figure 6: Compose kernel speedup vs. eager (bf16) across six GPUs. (a) Forward:

1.5

–

4.5

×. (b) Autograd: gains compound with activation size.

Figure 7: Bandwidth utilization (fp32, six GPUs). Fused approaches

{\sim}50\%

of peak on all architectures; eager values are approximate lower bounds.

Figure 8: Backward speedup (bf16). Below

{\sim}4096\times 4096

, launch overhead dominates; above

{\sim}8192\times 8192

, fused wins on all GPUs.

Figure 9: Norm memory reduction. (a) Theoretical persistent working set. (b) Measured allocator delta. The MoE shape

8192\times 28672

achieves

11

× measured reduction.

Shape	Rank	PEFT	Factored	Meas. ×	Theory ×
$4096\times 4096$	64	192 MB	65 MB	3.0×	63×
$4096\times 4096$	384	192 MB	71 MB	2.7×	9.8×
$4096\times 4096$	512	192 MB	73 MB	2.6×	7.1×
$8192\times 8192$	384	768 MB	245 MB	3.1×	20.4×
$8192\times 8192$	512	768 MB	241 MB	3.2×	15.1×
$8192\times 8192$	768	768 MB	236 MB	3.2×	9.8×

Table 7: Norm memory: measured allocation delta and theoretical reduction (fp32, H200). Measured reductions are smaller than theoretical because they include the rank-independent base-norm transient (§ 2.3 ).

Figure 10: Norm latency vs. rank (RTX 6000 PRO, fp32). The PEFT time is constant in

r

; factored scales linearly. At

r\leq 128

, factored matches the reference due to reduced memory traffic.

Figure 11: Memory profile (H200, bf16,

d\!=\!4096

, bs=4, seq=2048). (a) Fused reduces forward peak by 64 MB. (b) Savings grow with batch×seq; backward peak is unchanged.

Model	Method	RTX	H200	B200
Qwen2.5-VL-32B	Eager	OOM	99.2	99.2
	Fused	OOM	98.4	98.4
	Dense (B@A)	OOM	100.6	100.6
	PEFT	OOM	103.5	103.5
Mistral-Sm-24B	Eager	71.6	71.7	71.6
	Fused	70.6	70.7	70.6

Table 8: Model-level peak VRAM (GB). Fused uses less than all baselines on every model. 32B models OOM on RTX 6000 PRO.

	Compose fwd	Backward	E2E	Norm mem
L40S bf16	1.47×	1.06×	1.05×	0.8×
A100 bf16	1.73×	1.16×	1.07×	0.8×
RTX 6000 PRO bf16	1.92×	1.22×	1.08×	0.8×
H200 bf16	2.00×	1.08×	1.06×	0.8×
B200 bf16	2.70×	1.23×	1.08×	0.8×
B300 bf16	2.62×	1.22×	1.08×	0.8×

Table 9: Geometric mean microbenchmark speedups (all shapes, 200 repeats). Norm memory

0.8

× in bf16 means factored uses more memory for the isolated norm due to fp32 accumulation transients (§ 2.3 ).

Seed	Steps	Mean $\|\Delta\|$	Max $\|\Delta\|$	Eval $\|\Delta\|$	Wall (fused/eager)
1	2000	$7.2\!\times\!10^{-4}$	$1.1\!\times\!10^{-2}$	$9.2\!\times\!10^{-5}$	330/362 min
2	2000	$6.9\!\times\!10^{-4}$	$3.3\!\times\!10^{-3}$	$1.4\!\times\!10^{-4}$	330/359 min
3	2000	$7.1\!\times\!10^{-4}$	$4.1\!\times\!10^{-3}$	$3.3\!\times\!10^{-5}$	330/359 min
Grand mean		$7.1\!\times\!10^{-4}$	—	$8.9\!\times\!10^{-5}$	330/360 min

Table 10: Multi-seed convergence: eager vs. fused training loss (Qwen3.5-9B-Base,

r\!=\!384

, 2000 steps). Grand mean per-step delta

7.1\times 10^{-4}

; final eval losses agree to

<1.5\times 10^{-4}

Figure 12: Convergence: eager vs. fused are visually indistinguishable (Qwen3.5-9B-Base,

r\!=\!384

, seed 3 of 3). (a) Training loss (25-step smoothing). (b) Eval loss (200-step intervals). (c) Gradient norms.

Scenario	Config	Rationale
Training, CUDA, above crossover	Tier 1 (fused bwd)	Full speedup; 0.1–1.0 GB less VRAM than eager
Training, memory-constrained	Tier 1 + frozen mag	inner skipped; lower peak VRAM
Inference, CUDA	Tier 2 (fused fwd)	Compose speedup, no backward memory
CPU / no Triton	Tier 3 (eager)	Automatic fallback
$r\geq 384$ , any single GPU	Factored norm	PEFT path 46–87% slower
$r\leq 64$ , $d\leq 4096$	Either norm	Factored overhead minimal

Table 11: Recommended configuration by scenario.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Feature	Status	Scope	Notes
Mixed-precision bf16	✓	Bench+Conv	All model benchmarks and convergence runs
Mixed-precision fp16	✓	CI	Operator tests only; no model-level fp16 runs
Gradient checkpointing	✓	Bench+Conv	All model benchmarks use gradient checkpointing
ZeRO-2	✓	Conv	All convergence runs use ZeRO-2
ZeRO-3	✓	CI	Parameter gathering tested; no model-level runs
FSDP1	✓	CI	summon_full_params path tested

Appendix

Table 12: Compatibility matrix. Scope: Bench = model-level benchmarks, Conv = convergence runs, CI = operator-level test suite.

Model	Method	RTX 6000 PRO	H200	B200
Qwen2.5-VL-32B	Eager	OOM	99.2	99.2
	Fused	OOM	98.4	98.4
	Dense (B@A)	OOM	100.6	100.6
	PEFT	OOM	103.5	103.5
Qwen3-VL-32B	Eager	OOM	99.2	99.1
	Fused	OOM	98.4	98.4

Appendix

Table 13: Model-level gradient-computation peak VRAM (GB) across three GPUs, all six models. Same setup as Table 8 . Values from peak_vram_mb .

Figure 13: Single-layer E2E overhead decomposition (B200, bf16,

d=4096

, bs=4, seq=2048). Single-layer E2E does not predict model-level speedup: compose gains compound across

\sim

500 DoRA modules while per-layer backward overhead is amortized.

Figure 14: Single-layer E2E speedup (eager/fused) across six GPUs and ranks (bf16,

d=4096

, bs=4, seq=2048). All GPUs show consistent improvement.

Figure 15: Single-layer E2E speedup vs. hidden dimension (bf16,

r=384

, six GPUs). The benefit peaks at

h=3072

–

4096

, corresponding to common LLM sizes.

	Compose fwd	Backward	E2E	Norm mem
L40S fp32	1.53×	1.21×	1.04×	3.2×
A100 fp32	1.64×	1.20×	1.02×	3.2×
RTX 6000 PRO fp32	1.97×	1.27×	1.06×	3.2×
H200 fp32	1.84×	1.13×	1.03×	3.2×
B200 fp32	2.35×	1.26×	1.03×	3.2×
B300 fp32	2.23×	1.21×	1.03×	3.2×

Appendix

Table 14: Geometric mean microbenchmark speedups, fp32 (all shapes, 200 repeats). Complement to Table 9 .

Framework	Version	File Path	Pattern
HF PEFT	20a9829	peft/tuners/lora/dora.py	torch.eye
torchtune	v0.5.0	modules/peft/dora.py	Same algorithm
Unsloth	2026.3.7	Disables custom kernels	Falls back to PEFT
SWIFT (ms-swift)	a807cb9	Defers to PEFT/Unsloth	No custom code
LLaMA-Factory	v0.9.3	Delegates to PEFT	No custom code
Axolotl	v0.6.0	Delegates to PEFT	No custom code

Appendix

Table 15: DoRA norm implementation in major fine-tuning frameworks (February 2026).

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
Sep 8, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamLLM Fine-Tuning Low-Rank Adaptation
Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA
D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation
Feb 16, 2026Nozomu Fujisawa, Masaaki KondoLLM Fine-Tuning Low-Rank Adaptation
Keio University
CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference
Feb 26, 2026Hung-Hsuan ChenNonlinear LoRA Adapter Tuning
Computer Science and Information Engineering National Central University Taoyuan, Taiwan

Sep 8, 2026J/K move · Enter open · S save

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

Ashfak Yeafi, Mehedi Hasan, Md Khairul Islam

Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, Bangladesh · Department of Computer Science and Engineering, Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA

LLM Fine-Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) adapts large language models (LLMs) to downstream tasks while updating only a small fraction of their pretrained parameters. Low-Rank Adaptation (LoRA) uses two trainable low-rank matrices, while Weight-Decomposed Low-Rank Adaptation (DoRA) further separates weight magnitude and direction but retains the dense LoRA-style factorization in its directional branch. We propose ChainDoRA, a weight-decomposed adaptation framework that constructs the directional low-rank factors from a connected Tensor-Train (TT) chain, where the adapter rank forms the boundary rank between input- and output-side TT contractions and an independent TT rank controls representation capacity and parameter cost. Under a controlled 15,119-example response-only adaptation setting with LLaMA-7B, ChainDoRA is evaluated against matched LoRA and DoRA baselines on seven commonsense reasoning benchmarks. ChainDoRA with TT rank 16 achieves a seven-task average accuracy of 72.30%, compared with 69.88% for LoRA and 69.39% for DoRA, while requiring only 5.35M trainable parameters versus 56.10M for LoRA and 56.98M for DoRA, corresponding to a 90.62% reduction relative to DoRA. Ablations over TT rank and adapter placement show controllable parameter-accuracy trade-offs, indicating that connected TT parameterization can substantially reduce the parameter cost of magnitude-direction adaptation while preserving, and in this setting improving, downstream reasoning performance.

Abstract

Low-Rank Adaptation

Explore similar work

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation

CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference