Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

Akira Sakai; Yuma Ichikawa

cs.LGFeb 19, 2026

Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

Authors: Akira Sakai, Yuma Ichikawa

Organizations: Fujitsu Limited · Tokai University · Riken Center for AIP

Abstract

Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression -- the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.

Figures & tables

Figure 1: One-bit wall. Shannon’s rate-distortion lower bound for binary sign patterns under Hamming distortion evaluated using an entropy-rate proxy estimated from pretrained weights. Across all models, this proxy is close to one, and the bound is nearly indistinguishable from that of an i.i.d. Rademacher baseline, indicating that sign patterns contain little redundancy.

Panel of Figure 2 — Figure 2: Empirical validation of one-bit wall. (a) SVD compressibility (best rank- $r$ approximation error) of the raw weight matrix $W$ , sign matrix $S=\sign(W)$ , and magnitude matrix $A=|W|$ as a function of rank ratio $r/d$ ( $d=\min(m,n)$ ). (b) Spectral fit of sign matrices to an i.i.d. Rademacher baseline using a two-sample KS test on normalized singular values. (c) Initialization-to-trained sign drift in a Transformer trained on next-token prediction: flip ratio vs. initialization across layers (input $\rightarrow$ output) and pooled.

Panel of Figure 3 — Figure 3: Sign lock-in validation. Left: Histogram of the effective outer-to-outer flip count $K^{\mathrm{eff}}_{T}(\rho)$ across scalar weights; see Appendix C.4 for $T$ , $\rho$ , and $\epsilon$ . Right: Tail probability ${\mathbb{P}}[K^{\mathrm{eff}}_{T}(\rho)\geq k]$ on a log scale for multiple learning rates, with dashed geometric fits of the form $\hat{h}\,\hat{g}^{\,k-1}$ .

Panel of Figure 4 — Figure 4: Billion-scale sweep of the lock-in parameters $(\hat{h},\hat{g})$ .

Figure 5: Re-generable sign templates drive the effective sign cost to zero. A random sign matrix such as

S_{\mathrm{rand}}

has no compact description and costs about one bit per weight. In contrast,

S_{\mathrm{temp}}

is the elementwise sign of a rank-

1

outer product and is therefore re-generable from a small specification (a random seed and the rank). Storing only this specification amortizes the per-weight sign cost to

\approx 0

bits as the model grows, freeing the entire bit budget for the magnitudes.

Figure 6: Flip–quality trade-off with a compressible sign template (

r{=}2

). Validation perplexity mean

\pm

std over three seeds vs. the mean per-step sign flip rate ( flip_mean ). Each curve fixes the gap threshold

a_{\mathrm{init}}

and sweeps the log-barrier weight

\lambda

from left to right (

0.5,0.3,0.1,0.05,0.01,0.001,0.0001

). Stronger stabilization suppresses flips but can worsen perplexity, while intermediate

a_{\text{init}}

and

\lambda

achieve large flip reduction with little loss in validation quality.

Panel of Figure 7 — Figure 7: Sign vs magnitude low-rank compressibility example. Relative Frobenius error ${E}_{r}(M)=\lVert M-M_{r}\rVert_{F}/\lVert M\rVert_{F}$ as a function of rank ratio $r/d$ (log scale), for the sign matrix $S=\sign(W)$ and magnitude matrix $A=|W|$ , evaluated on final trained weights under baseline, gap initialization only, and gap initialization with regularization.

Panel of Figure 8 — Figure 8: Performance vs. effective bits per weight ( $\mathrm{bpw}_{\mathrm{eff}}$ ) on benchmark tasks. Markers indicate the mean over three seeds and error bars show one standard deviation. Lower is better for perplexity (CharLM, Text8-Char), while higher is better for accuracy (DBPedia14).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Estimated lock-in parameters over gap-init and regularization. Left:

\hat{h}

(initial-hit factor) as a function of the gap threshold

a_{\text{init}}

for different log-barrier weights

\lambda

. Right:

\hat{g}

(re-entry ratio) as a function of

\lambda

(log scale) for different

a_{\text{init}}

. Points show mean

\pm

std over three seeds; see Appendix C.4 for the full hyperparameters (including

T

\rho

, and

\epsilon

Figure 10: Representative one-dimensional weight trajectories (down sampled for printing). We plot a representative 1D weight trajectory (down-sampled)

w_{t}

under baseline (

a_{\text{init}}{=}0,\lambda{=}0

), a gap-dominant setting, and a strongly regularized setting, together with the outer/boundary bands used to define

(\sigma_{k},\tau_{k})

and

K^{\mathrm{eff}}_{T}(\rho)

. Regularization keeps trajectories away from the boundary neighborhood, suppressing re-entry.

Model	Params (B)	$d_{\mathrm{model}}$	Layers	Heads
31M	0.032	512	10	8
64M	0.064	768	9	12
126M	0.126	1024	10	16
260M	0.260	1344	12	21
399M	0.399	1664	12	26
604M	0.604	2048	12	32

Appendix

Table 1: Parameter-scale sweep. We report the configurations of Transformers

Figure 11: Sign lock-in trajectory and stopping times. We illustrate the one-dimensional weight trajectory (down-sampled)

w_{t}

together with the outer region and the sign-boundary neighborhood used in the sign lock-in theory. The outer region is defined as

\{|w_{t}|\geq\rho\}

and the boundary neighborhood as

\{|w_{t}|\leq\epsilon\}

. Starting from an outer-entry time

\sigma_{0}

(outer region, positive sign), the trajectory hits the boundary neighborhood at

\tau_{1}

and exits to the opposite outer side at

\sigma_{1}

(negative sign), then returns to the boundary at

\tau_{2}

and re-enters the original outer side at

\sigma_{2}

(positive sign). This outer-to-outer sign evolution is counted by the effective flip count

K^{\mathrm{eff}}_{T}(\rho)

Panel of Figure 16 — Figure 16: Empirical induction of sign floating mode via local $l_{1}$ attraction near the boundary (Appendix D.6 ).

#Params	Vanilla	Gap+Reg	Degradation PPL
0.3M	5.22	5.61	+0.39
1.9M	4.55	4.69	+0.14
4.8M	4.67	4.64	-0.03
10.8M	4.67	4.63	-0.04
25.4M	4.70	4.69	-0.01
59.3M	4.82	4.78	-0.04

Appendix

Table 2: Best validation PPL on Transformer next-token prediction.

#Params	Vanilla Best	Vanilla Final	Vanilla Overfit	Gap+Reg Best	Gap+Reg Final	Gap+Reg Overfit
0.3M	5.22	5.22	+0.00	5.61	5.56	-0.05
1.9M	4.55	4.69	+0.14	4.69	4.70	+0.01
4.8M	4.67	5.96	+1.29	4.64	4.67	+0.03
10.8M	4.67	9.42	+4.75	4.63	4.94	+0.31
25.4M	4.70	12.58	+7.88	4.69	5.76	+1.07
59.3M	4.82	15.36	+10.54	4.78	5.79	+1.01

Appendix

Table 3: Overfitting: best vs. final PPL on Transformer next-token prediction.

$K_{\mathrm{eff}}$	Count	Empirical prob.	Fitted prob.	Abs. error
0	168032	0.840160	0.840160	0.000000
1	25405	0.127025	0.126960	0.000065
2	5170	0.025850	0.026116	0.000266
3	1120	0.005600	0.005372	0.000228
4	230	0.001150	0.001105	0.000045
5	37	0.000185	0.000227	0.000042

Appendix

Table 4: Pythia-1B trained on 300B tokens: effective flip-count histogram and geometric-tail fit.

bpw	Sign Template (ours)	Vanilla SVD (baseline)	$\Delta$
0.50	5.3	0.5	+4.8
0.75	23.1	0.5	+22.6
1.00	42.0	0.5	+41.5
1.50	65.6	12.1	+53.5
2.00	74.7	65.1	+9.6
2.50	78.1	76.3	+1.8

Appendix

Table 5: ImageNet Top-5 accuracy (%) of ResNet under compression via SVD with NormalFloat factor quantization.

Phase	Statistic	Value
Early	median update	$9.999\times 10^{-5}$
Early	90th-percentile update	$1.191\times 10^{-4}$
Early	99th-percentile update	$1.540\times 10^{-4}$
Early	99.9th-percentile update	$1.865\times 10^{-4}$
Early	maximum update	$2.346\times 10^{-4}$
Early	# steps with max update $>2\rho$	0

Appendix

Table 6: Phase-wise empirical AdamW update magnitudes.

Model	$E_{r}(\sign(W))$	$E_{r}(\|W\|)$	$E_{r}(W)$
MLP	0.927	0.573	0.895
ResNet	0.943	0.593	0.921
TinyLlama	0.934	0.571	0.908

Appendix

Table 7: Low-rank approximation error by component at

r/d=0.03125

Flip count	Observed count	Fitted count
0	39161	39161
1	2059	1929
2	703	961
3	574	479
4	263	239
5	142	119

Appendix

Table 8: Effective sign-flip histogram in the CNN experiment.

model	brotli(q11)	brotli(q5)	bz2(l9)	gzip(l9)	lzma(p6)	raw_deflate(l9)	snappy	zlib(l9)	zstd(l10)	zstd(l3)
Rademacher baseline (2048x2048)	1.000	1.000	1.005	1.000	1.000	1.000	1.000	1.000	1.000	1.000
2D Ising baseline (beta=0.6)	0.001	0.002	0.004	0.025	0.004	0.025	0.067	0.025	0.002	0.003
Low-rank sign baseline (rank=2)	0.047	0.054	0.062	0.184	0.043	0.184	0.417	0.184	0.053	0.083
MLP-Mixer-B16	0.978	0.983	0.998	0.985	0.984	0.985	1.000	0.985	0.983	0.983
ResNet18	0.957	0.958	0.984	0.964	0.966	0.964	1.000	0.964	0.960	0.959
TinyLlama-1.1B-Chat	0.979	0.984	0.999	0.989	0.973	0.989	1.000	0.989	0.991	0.992

Appendix

Table 9: Compressibility of sign bits across compression algorithms (ratio = compressed / raw).

Figure 17: Empirical distribution of the effective flip count

K

. Most weights exhibit

k=0

, with a rapidly decaying tail for

k\geq 1

Figure 18: Semi-log plot of tail probabilities

{\mathbb{P}}[K\geq k]

for different learning rates. Dashed lines show geometric fits

{\mathbb{P}}[K\geq k]\approx h\,g^{k-1}

, indicating an approximately geometric tail whose heaviness increases with learning rate.

Panel of Figure 19 — Figure 19: Estimated lock-in parameters over gap-init and regularization on vision task. Left: $\hat{h}$ (initial-hit factor) as a function of the gap threshold $a_{\text{init}}$ for different log-barrier weights $\lambda$ . Right: $\hat{g}$ (re-entry ratio) as a function of $\lambda$ (log scale) for different $a_{\text{init}}$ . Points show mean $\pm$ std over three seeds.

Figure 20: Validation error vs. mean flip rate. Baseline is shown as a single point, and Gap+reg curves connect runs with the same

a

while varying

\lambda

. Error bars denote mean

\pm

std over seeds. The plot highlights both the trade-off regime (very low flip rate can hurt validation error for small

a

) and regimes where flip rates are drastically reduced with little or no degradation in validation error. Curves connect points with the same gap threshold

a_{\text{init}}

and the log barrier weight decreases from left to right as

0.5,0.3,0.1,0.05,0.01,0.001,0.0001

Panel of Figure 21 — Figure 21: Sign vs. magnitude low-rank compressibility. Relative Frobenius error $E_{r}(M)=\|M-M_{r}\|_{F}/\|M\|_{F}$ as a function of rank ratio $q=r/d$ (log scale), for the sign matrix $S=\mathrm{sign}(W)$ and magnitude matrix $A=|W|$ , evaluated on the final trained weights under (a) baseline, (b) gap initialization only, and (c) gap initialization with outer-drift regularization.

Panel of Figure 22 — Figure 22: Performance vs. effective bits per weight ( $\mathrm{bpw}_{\mathrm{eff}}$ ) under knowledge distillation. Markers indicate the mean over three seeds and error bars show one standard deviation. Lower is better for perplexity (CharLM-KD, Text8-Char-KD), while higher is better for accuracy (DBPedia14-KD). These panels complement the main benchmark results in Figure 8 .

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 122 papers

Jul 31, 2026

Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

SignMuon compresses the Muon update to one bit per parameter by taking its elementwise sign, providing the most direct way to run a matrix-aware optimizer under an extremely low communication budget. It outperforms SignSGD in practice, yet it can ascend even on a linear function. Signing the gradient before the Linear Minimization Oracle (LMO), rather than after, does not repair this: we construct a small explicit instance on which sign-before (MuonUSign) and sign-on-both-sides (MuonSign) ascend as well, so no placement of the sign around the oracle descends in general. Error feedback, the standard remedy for a biased compressor, does not rescue SignMuon: when applied to Muon's output, error feedback can fail for every smoothness constant, step size, and momentum. Applied to the gradient, error feedback does work, and EF21-MuonUSign and EF21-MuonSign attain the standard

\mathcal{O}(T^{-1/2})

rate for the squared gradient norm on smooth nonconvex problems, the latter at one bit in each direction. Experiments then reverse the ordering: across centralized CIFAR-10, federated CIFAR-10, and the nanoGPT speedrun, the strongest compressed method is consistently sign-after-the-LMO, precisely the placement we prove divergent, with the provably convergent variants trailing it. Compressing after the LMO, a heuristic, matters more at these scales than the guarantee does.

Maria Smirnova, Alexey Kravatskiy

MIRIAI Moscow, Russia

Sign-Based Optimization Communication-Efficient Distributed Training Error Feedback Gradient Compression Muon Optimizer Communication Complexity

Ask PDF

Feb 2, 2026

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational resources. We study training failures in a 5M-parameter NanoGPT model scaled via

μ

P, identifying two key phenomena preceding collapse: (1) rapid decline in weight matrix stable rank (ratio of squared Frobenius norm to squared spectral norm), and (2) increasing alignment between adjacent layer Jacobians. We prove theoretically that these two conditions jointly cause exponential gradient norm growth with network depth. To break this instability mechanism, we propose MSign, a new optimizer that periodically applies matrix sign operations to restore stable rank. Experiments on models from 5M to 3B parameters demonstrate that MSign effectively prevents training failures with a computational overhead of less than 7.0%.

Lianhai Ren, Yucheng Ding, Xiao Liu +2

Microsoft SIGMA Team, Microsoft Research.

Language Model Pretraining LLM Training

Ask PDF

Apr 28, 2026

Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy

SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.

Haoran Chen, Wentao Wang

´Ecole polytechnique Palaiseau, France

Sign-Based Optimization Stochastic Optimization Convergence Gradient Quantization Stochastic Gradient Descent

Ask PDF

Abstract

Model Compression

Explore similar work

Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy