Second-order optimizers improve deep neural network training using curvature information, but their practical efficiency is limited by GPU execution overhead. AdaFisher reduces Fisher Information Matrix (FIM) approximation costs through diagonal Kronecker structures and exponential moving averages, yet its PyTorch implementation requires sequential CUDA kernel launches, intermediate tensor allocations, and repeated memory accesses. We introduce CAFIN, a GPU-native CUDA kernel fusion framework that improves AdaFisher execution efficiency without changing its optimization formulation. The proposed approach progressively integrates FIM preconditioner construction, exponential moving average updates, gradient preconditioning, momentum accumulation, weight decay, and parameter updates into fused CUDA kernels. Shared-memory reuse, register-resident computation, and in-place operations further reduce memory traffic and intermediate allocations. Experiments using ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet across NVIDIA GTX 1660 Ti and RTX 3060 Laptop GPUs demonstrate up to 78.4% lower isolated FIM construction latency and 2.1-26.9% lower optimizer-step latency. The implementation also eliminates approximately 18.91 MB of intermediate GPU memory allocations per optimizer step. Numerical validation confirms consistency within the predefined FP32 tolerance, while convergence accuracy remains comparable across evaluated datasets and random seeds. These findings demonstrate that CUDA kernel fusion can substantially accelerate second-order optimization while preserving AdaFisher's underlying mathematical formulation and learning behavior.
Figures & tables
Fig. 1: Progressive kernel fusion in AdaFisher showing reduction of per-layer CUDA kernel launches. From left to right: original (6 kernels), fused preconditioner (5 kernels), fused EMA + preconditioner (4 kernels), and fully fused optimizer (1 kernel). The fusion reduces kernel dispatch overhead, eliminates intermediate tensor allocations, and improves GPU execution efficiency.
Fig. 2: Comparison of execution flow between baseline and fully fused AdaFisher optimizers. The baseline uses multiple CUDA kernels with intermediate global memory transfers, while the fully fused implementation combines all operations into a single kernel, leveraging registers and shared memory to minimize DRAM traffic and kernel-launch overhead.
Fig. 3: Layer-wise Frobenius norm error between baseline and fully fused AdaFisher implementation on CIFAR-10. The dashed line represents the unified numerical tolerance threshold ( 2×10−6 ). All layers remain within the predefined numerical tolerance, confirming numerical consistency between the baseline and fully fused FP32 implementations.
Dataset
Max Frobenius Error
Threshold
Status
CIFAR-10
1.663×10−6
2×10−6
PASS
CIFAR-100
1.858×10−6
2×10−6
PASS
TABLE II: Numerical correctness across datasets under the predefined FP32 numerical tolerance ( 2×10−6 ).
GPU
Arch.
Cores
VRAM
BW
GTX 1660 Ti
Turing (sm_75)
1536
6.4 GB
288 GB/s
RTX 3060 L
Ampere (sm_86)
3840
6.4 GB
336 GB/s
TABLE III: Hardware Configuration
Hyperparameter
Value
Learning rate η
0.001
Momentum β
0.9
EMA decay γ
0.8
Damping λ
0.001
Weight decay ω
5×10−4
Batch size
256
TABLE IV: Hyperparameter Configuration
Mode
Description
Eager Baseline
Baseline PyTorch implementation using kron() and separate tensor operations
Fused Preconditioner
Single fused outer-product CUDA kernel for FIM construction
Fused EMA + Prec.
EMA update fused with outer-product computation
Fully Fused Optimizer
Entire optimizer step fused into a single CUDA kernel
Fig. 4: TOST equivalence analysis across CIFAR-10 and CIFAR-100. The shaded region represents the equivalence margin ( ±0.1% ). Overlapping confidence intervals indicate statistical equivalence of baseline and fused optimization behavior across multiple seeds.
Fig. 5: Roofline analysis showing that the fully fused AdaFisher reduces DRAM transfers, increases arithmetic intensity, and achieves higher GPU performance compared to the baseline.
Fig. 6: Per-layer FIM construction latency (GTX 1660 Ti, log scale). (a) Latency across all ten layers; the Fused Preconditioner reduces average latency from 170.4 µs to 37.1 µs. (b) Mean latency showing progressive speedup at each fusion stage.
Fig. 7: GPU memory usage per FIM call for baseline and fused AdaFisher. The fused preconditioner eliminates intermediate tensor allocations, reducing total memory per optimizer step by 18.91 MB.
Fig. 8: 10-epoch validation on CIFAR-10 (batch=128). Left : per-epoch optimizer time. Right : summary speedup vs. accuracy trade-off showing 21.8% speedup while preserving convergence.
Fig. 9: Optimizer step latency for baseline, fused preconditioner, and fully fused AdaFisher on GTX 1660 Ti. Fusion reduces per-step latency, with the fully fused optimizer achieving an additional 1.5–4.3 percent speedup.
Fig. 10: Optimizer-step speedup across datasets for the fully fused AdaFisher kernel. The results show consistent latency reduction on CIFAR-10 and CIFAR-100, demonstrating robustness of kernel fusion across dataset scales.
Fig. 11: Distribution of optimizer-step latency under baseline and fused execution. The fused implementation exhibits reduced variance and lower mean execution time, a difference that is statistically significant under Welch’s t -test.
Component
Fraction of epoch
Speedup
Epoch gain
FIM construction
8%
78.4%
6.3%
Optimizer step total
20%
26.9%
5.4%
Backward pass
72%
—
—
Total training time saved (18-hour run):
1.13 h
TABLE X: Amdahl’s Law Analysis: Predicted Speedup from Fusion
Fig. 12: CIFAR-10 convergence over 200 epochs (ResNet-18, GTX 1660 Ti, seeds 42/123/456). Left: Test accuracy vs. training epoch for all seeds — curves overlap across all seeds, confirming that kernel fusion does not alter learning dynamics. Right: Final test accuracy at epoch 200 across seeds. Mean accuracy difference between Eager Baseline and Fully Fused Optimizer = −0.23% (CIFAR-10 only), within baseline run-to-run variance ( ±0.04% ).
Fig. 13: Convergence validation: ResNet-18 on CIFAR-100 (GTX 1660 Ti, 200 epochs, seeds 42/123/456). Left panels : accuracy trajectories for both modes—curves overlap across all seeds. Right panel : final accuracy per seed. Mean difference =−0.31% , within baseline run-to-run variance ( ±0.23% ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Layer
Eager
Alg.
Fused
Fused/Eager
Fused/Alg.
conv1 ( 3→64 )
157.6
87.5
27.3
+ 82.7%
+ 68.8%
layer1.0 ( 64→64 )
161.7
95.2
27.9
+ 82.7%
+ 70.7%
layer1.1 ( 64→64 )
163.1
94.6
30.5
+ 81.3%
+ 67.8%
layer2.0 ( 64→128 )
162.5
105.3
33.0
+ 79.7%
+ 68.7%
layer2.1 ( 128→128 )
163.3
91.3
29.5
+ 82.0%
+ 67.7%
layer3.0 ( 128→256 )
153.4
91.1
34.2
+ 77.7%
+ 62.4%
Appendix
TABLE XII: Per-Layer FIM Construction Latency (µs, GTX 1660 Ti)
Epoch
Eager Opt (s)
Fused Opt (s)
Test Acc (%)
Eager / Fused
1
9.61
—
49.3 / 47.5
2
5.38
4.47
65.8 / 65.1
3
5.32
4.49
72.3 / 71.9
4
5.27
4.45
76.1 / 75.9
5
5.32
4.45
76.8 / 77.3
Appendix
TABLE XIII: 10-Epoch Validation: Per-Epoch Optimizer Time and Accuracy (CIFAR-10, GTX 1660 Ti, batch=128)