Second-order optimizers improve deep neural network training using curvature information, but their practical efficiency is limited by GPU execution overhead. AdaFisher reduces Fisher Information Matrix (FIM) approximation costs through diagonal Kronecker structures and exponential moving averages, yet its PyTorch implementation requires sequential CUDA kernel launches, intermediate tensor allocations, and repeated memory accesses. We introduce CAFIN, a GPU-native CUDA kernel fusion framework that improves AdaFisher execution efficiency without changing its optimization formulation. The proposed approach progressively integrates FIM preconditioner construction, exponential moving average updates, gradient preconditioning, momentum accumulation, weight decay, and parameter updates into fused CUDA kernels. Shared-memory reuse, register-resident computation, and in-place operations further reduce memory traffic and intermediate allocations. Experiments using ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet across NVIDIA GTX 1660 Ti and RTX 3060 Laptop GPUs demonstrate up to 78.4% lower isolated FIM construction latency and 2.1-26.9% lower optimizer-step latency. The implementation also eliminates approximately 18.91 MB of intermediate GPU memory allocations per optimizer step. Numerical validation confirms consistency within the predefined FP32 tolerance, while convergence accuracy remains comparable across evaluated datasets and random seeds. These findings demonstrate that CUDA kernel fusion can substantially accelerate second-order optimization while preserving AdaFisher's underlying mathematical formulation and learning behavior.
Figures & tables
Fig. 1: Progressive kernel fusion in AdaFisher showing reduction of per-layer CUDA kernel launches. From left to right: original (6 kernels), fused preconditioner (5 kernels), fused EMA + preconditioner (4 kernels), and fully fused optimizer (1 kernel). The fusion reduces kernel dispatch overhead, eliminates intermediate tensor allocations, and improves GPU execution efficiency.
Fig. 2: Comparison of execution flow between baseline and fully fused AdaFisher optimizers. The baseline uses multiple CUDA kernels with intermediate global memory transfers, while the fully fused implementation combines all operations into a single kernel, leveraging registers and shared memory to minimize DRAM traffic and kernel-launch overhead.
Fig. 3: Layer-wise Frobenius norm error between baseline and fully fused AdaFisher implementation on CIFAR-10. The dashed line represents the unified numerical tolerance threshold ( 2×10−6 ). All layers remain within the predefined numerical tolerance, confirming numerical consistency between the baseline and fully fused FP32 implementations.
Dataset
Max Frobenius Error
Threshold
Status
CIFAR-10
1.663×10−6
2×10−6
PASS
CIFAR-100
1.858×10−6
2×10−6
PASS
TABLE II: Numerical correctness across datasets under the predefined FP32 numerical tolerance ( 2×10−6 ).
GPU
Arch.
Cores
VRAM
BW
GTX 1660 Ti
Turing (sm_75)
1536
6.4 GB
288 GB/s
RTX 3060 L
Ampere (sm_86)
3840
6.4 GB
336 GB/s
TABLE III: Hardware Configuration
Hyperparameter
Value
Learning rate η
0.001
Momentum β
0.9
EMA decay γ
0.8
Damping λ
0.001
Weight decay ω
5×10−4
Batch size
256
TABLE IV: Hyperparameter Configuration
Mode
Description
Eager Baseline
Baseline PyTorch implementation using kron() and separate tensor operations
Fused Preconditioner
Single fused outer-product CUDA kernel for FIM construction
Fused EMA + Prec.
EMA update fused with outer-product computation
Fully Fused Optimizer
Entire optimizer step fused into a single CUDA kernel
Fig. 4: TOST equivalence analysis across CIFAR-10 and CIFAR-100. The shaded region represents the equivalence margin ( ±0.1% ). Overlapping confidence intervals indicate statistical equivalence of baseline and fused optimization behavior across multiple seeds.
Fig. 5: Roofline analysis showing that the fully fused AdaFisher reduces DRAM transfers, increases arithmetic intensity, and achieves higher GPU performance compared to the baseline.
Fig. 6: Per-layer FIM construction latency (GTX 1660 Ti, log scale). (a) Latency across all ten layers; the Fused Preconditioner reduces average latency from 170.4 µs to 37.1 µs. (b) Mean latency showing progressive speedup at each fusion stage.
Fig. 7: GPU memory usage per FIM call for baseline and fused AdaFisher. The fused preconditioner eliminates intermediate tensor allocations, reducing total memory per optimizer step by 18.91 MB.
Fig. 8: 10-epoch validation on CIFAR-10 (batch=128). Left : per-epoch optimizer time. Right : summary speedup vs. accuracy trade-off showing 21.8% speedup while preserving convergence.
Fig. 9: Optimizer step latency for baseline, fused preconditioner, and fully fused AdaFisher on GTX 1660 Ti. Fusion reduces per-step latency, with the fully fused optimizer achieving an additional 1.5–4.3 percent speedup.
Fig. 10: Optimizer-step speedup across datasets for the fully fused AdaFisher kernel. The results show consistent latency reduction on CIFAR-10 and CIFAR-100, demonstrating robustness of kernel fusion across dataset scales.
Fig. 11: Distribution of optimizer-step latency under baseline and fused execution. The fused implementation exhibits reduced variance and lower mean execution time, a difference that is statistically significant under Welch’s t -test.
Component
Fraction of epoch
Speedup
Epoch gain
FIM construction
8%
78.4%
6.3%
Optimizer step total
20%
26.9%
5.4%
Backward pass
72%
—
—
Total training time saved (18-hour run):
1.13 h
TABLE X: Amdahl’s Law Analysis: Predicted Speedup from Fusion
Fig. 12: CIFAR-10 convergence over 200 epochs (ResNet-18, GTX 1660 Ti, seeds 42/123/456). Left: Test accuracy vs. training epoch for all seeds — curves overlap across all seeds, confirming that kernel fusion does not alter learning dynamics. Right: Final test accuracy at epoch 200 across seeds. Mean accuracy difference between Eager Baseline and Fully Fused Optimizer = −0.23% (CIFAR-10 only), within baseline run-to-run variance ( ±0.04% ).
Fig. 13: Convergence validation: ResNet-18 on CIFAR-100 (GTX 1660 Ti, 200 epochs, seeds 42/123/456). Left panels : accuracy trajectories for both modes—curves overlap across all seeds. Right panel : final accuracy per seed. Mean difference =−0.31% , within baseline run-to-run variance ( ±0.23% ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Layer
Eager
Alg.
Fused
Fused/Eager
Fused/Alg.
conv1 ( 3→64 )
157.6
87.5
27.3
+ 82.7%
+ 68.8%
layer1.0 ( 64→64 )
161.7
95.2
27.9
+ 82.7%
+ 70.7%
layer1.1 ( 64→64 )
163.1
94.6
30.5
+ 81.3%
+ 67.8%
layer2.0 ( 64→128 )
162.5
105.3
33.0
+ 79.7%
+ 68.7%
layer2.1 ( 128→128 )
163.3
91.3
29.5
+ 82.0%
+ 67.7%
layer3.0 ( 128→256 )
153.4
91.1
34.2
+ 77.7%
+ 62.4%
Appendix
TABLE XII: Per-Layer FIM Construction Latency (µs, GTX 1660 Ti)
Epoch
Eager Opt (s)
Fused Opt (s)
Test Acc (%)
Eager / Fused
1
9.61
—
49.3 / 47.5
2
5.38
4.47
65.8 / 65.1
3
5.32
4.49
72.3 / 71.9
4
5.27
4.45
76.1 / 75.9
5
5.32
4.45
76.8 / 77.3
Appendix
TABLE XIII: 10-Epoch Validation: Per-Epoch Optimizer Time and Accuracy (CIFAR-10, GTX 1660 Ti, batch=128)
We present a comparative study of CUDA optimization strategies applied to forward and backward propagation in a shallow neural network. Three stacked optimizations are evaluated: (1) tiled shared memory with bank-conflict elimination via +1-column padding, (2) pre-transposed weight matrices for coalesced global memory access, and (3) a fused MatMul+ReLU kernel that eliminates intermediate global-memory round-trips. Experiments on an NVIDIA Tesla T4 (CUDA 13.0) across three dataset sizes show that the fully optimized implementation achieves a 1.41x speedup over the baseline CUDA version on the large dataset (25,600 samples), reducing execution time from 21.0s to 14.8s. Results are compared against a sequential CPU baseline and an OpenMP parallel implementation, demonstrating the effectiveness of memory-access optimization in GPU-accelerated deep learning primitives.
Rania Zitouni, Nadine Bousdjira, Sarah Hasnaoui +2
E´ cole Nationale Supe´rieure d’Informatique (ESI) · High Performance Computing (HPC)
AdamW is a default optimizer for deep learning, but its moment states add two parameter-sized buffers to training memory, increasing the cost of large-scale pretraining. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook. Gefen reduces AdamW's optimizer memory footprint by up to 8x while maintaining performance, saving 6.5 GiB per billion parameters. Prior work shares second moments across parameters grouped along the Hessian's block-diagonal structure, but relies on hand-specified architectural rules and leaves unexplained why such grouping works. We prove that large mixed Hessian entries constrain the ratio of squared gradients toward one, explaining why shared second moments are accurate when the squared gradients they pool are similar. The Hessian need not be computed: its block structure is inherited by squared gradients, allowing blocks to be found directly. Gefen therefore infers block structure from initial squared gradients, requiring no architecture-specific metadata or user-tuned hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the blocks for first-moment scaling. Across diverse pretraining experiments, Gefen achieves the lowest peak optimizer memory among compared methods that maintain AdamW-level performance. In single-machine and distributed training, the reduced footprint enables larger microbatches and substantially improves throughput over AdamW, making Gefen a drop-in replacement that can train larger models or use larger global batch sizes. We provide the complete Python implementation, including fused CUDA kernels at https://github.com/ndvbd/Gefen
Nadav Benedek, Tomer Koren, Ohad Fried
Reichman University · Tel Aviv University, Google Research
The exponential growth in demand for GPU computing resources has created an urgent need for automated CUDA optimization strategies. While recent advances in LLMs show promise for code generation, current SOTA models achieve low success rates in improving CUDA speed. In this paper, we introduce CUDA-L1, an automated reinforcement learning framework for CUDA optimization that employs a novel contrastive RL algorithm. CUDA-L1 achieves significant performance improvements on the CUDA optimization task: trained on A100, it delivers an average speedup of x3.12 with a median speedup of x1.42 against default baselines over across all 250 CUDA kernels of KernelBench, with peak speedups reaching x120. In addition to the default baseline provided by KernelBench, CUDA-L1 demonstrates x2.77 over Torch Compile, x2.88 over Torch Compile with reduce overhead, x2.81 over CUDA Graph implementations, and x7.72 over cuDNN libraries. Furthermore, the model also demonstrates portability across different GPU architectures. Beyond these benchmark results, CUDA-L1 demonstrates several properties: it 1) discovers a variety of CUDA optimization techniques and learns to combine them strategically to achieve optimal performance; 2) uncovers fundamental principles of CUDA optimization, such as the multiplicative nature of optimizations; 3) identifies non-obvious performance bottlenecks and rejects seemingly beneficial optimizations that actually harm performance. The capabilities demonstrate that, RL can transform an initially poor-performing LLM into an effective CUDA optimizer through speedup-based reward signals alone, without human expertise or domain knowledge. This paradigm opens possibilities for automated optimization of CUDA operations, and holds promise to substantially promote GPU efficiency and alleviate the rising pressure on GPU computing resources.