DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

Nobutaka Ono

cs.LGMay 7, 2026

DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

Authors: Nobutaka Ono

Organizations: Tokyo Metropolitan University, 6–6 Asahigaoka, Hino-shi, Tokyo 191–0065, Japan.

Abstract

In this paper, we propose DiBA (Diagonal and Binary Matrix Approximation), a compact matrix factorization for neural network weight compression. Many components of modern networks, including linear layers, $1\times1$ convolutions, attention projections, and embedding layers, have dense matrix weights. DiBA approximates $A\in\mathbb{R}^{m\times n}$ by $\widehat A=D_1B_1D_2B_2D_3$ , where $D_1,D_2,D_3$ are diagonal matrices and $B_1,B_2$ are $0/1$ binary matrices. The intermediate dimension $k$ controls the trade-off between theoretical storage and approximation accuracy. For matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from $mn$ to $m+k+n$ . For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-matrix layers by DiBA factors, freezes the binary matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-token accuracy from 0.4447 to 0.5210 and Speech Commands test accuracy for an Audio Spectrogram Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors.

Explore similar work

May 23, 2026cs.LG

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component

R

has small propagated logit effect

L_s \| R ψ_1(s) \|_\infty

, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive

L_2

-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge

σ_+

gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16

2{:}4{+}

ToMe reaches

83.41\%

top-1 (

-1.70

pp from dense) at

59.81\%

sparse-execution MAC reduction, with

1.388\times

best-observed A40 native-

2{:}4

backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives

2.705\times

. At structured sparsity, ViT-B/16

6{:}12

reaches

83.74\%

, ViT-L/16

8{:}16

dense+permutation reaches

85.33\%

(

-0.51

pp), and ConvNeXtV2-Base

12{:}16

reaches

86.35\%

(

-0.37

pp). For CNNs, ResNet50

8{:}16

dense+permutation reaches

75.87\%

(

-0.26

pp), and ResNet152d CAST-conv+permutation reaches

81.33\%

(

-1.53

pp) at

{\sim}50\%

MAC accounting with a

1.62\times

A40 im2col

+2{:}4

sparse-GEMM audit.

Leonid Berlyand, Theo Bourdais, Houman Owhadi +1

Deep Neural Network Resnet-18

Ask PDF

May 23, 2025cs.LG

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks. However, leveraging the full observed Fisher information is too expensive for large models, so most methods rely on simple diagonal approximations. While efficient, this approach ignores parameter correlations, often resulting in reduced performance on downstream tasks. In this work, we mitigate these limitations and propose Generalized Fisher-Weighted SVD (GFWSVD), a post-training LLM compression technique that accounts for both diagonal and off-diagonal elements of the Fisher information matrix, providing a more accurate reflection of parameter importance. To make the method tractable, we introduce a scalable adaptation of the Kronecker-factored approximation algorithm for the observed Fisher information. We demonstrate the effectiveness of our method on LLM compression, showing improvements over existing compression baselines. For example, at a 20 compression rate on the MMLU benchmark, our method outperforms FWSVD, which is based on a diagonal approximation of the Fisher information, by 5 percent, SVD-LLM by 3 percent, and ASVD by 6 percent compression rate.

Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva +2

Large Language Model Compression Singular Value Decomposition Fisher Information Model Weights

Ask PDF

Aug 6, 2026cs.LG

BaKron: Efficient Quantization with Kronecker-Factored Hessians

We accelerate a family of algorithms for neural network quantization whose geometry is informed by any Kronecker-factored approximation of the Hessian. GPTQ-style adaptive rounding typically uses one-sided information derived from input activations. Two-sided Kronecker-factored Hessian approximations can additionally capture correlations across output coordinates, but applying GPTQ directly in the vectorized weight domain is computationally expensive. Building on the two-sided adaptive-rounding formulation used by BoA and YAQA, we introduce BaKron, an efficient solver that combines anti-diagonal parallelism with a recursive divide-and-conquer construction. For an

m\times n

weight matrix, BaKron uses

O(m+n)

sequential steps while reducing the total work from

O(m^2n^2)

O(mn(m+n))

. Thus, it matches the cubic scaling of GPTQ while exploiting richer curvature information. Moreover, BaKron is modular with respect to both the base quantizer and the Hessian estimator. We also provide practical benchmarks, consider a range of Hessians that BaKron can be called with, find an efficient technique to compute these Hessians, and evaluate the algorithm experimentally.

Johann Birnick, Rayan Saab

Mixed-Precision Quantization Low-Bit Quantization Hessian Matrix Spectral Preconditioning Low-Rank Factorization Lightweight Convolutional Neural Network Hardware Accelerator+2

Ask PDF