The Standard Deviation in VICReg Buys a Safety Margin

Hawar Dizayee

cs.LGOct 23, 2026

The Standard Deviation in VICReg Buys a Safety Margin

Authors: Hawar Dizayee

Organizations: Independent Researcher

Abstract

VICReg prevents collapse with a hinge on the standard deviation of each embedding dimension, and its authors argue from the gradient that a hinge on the variance would fail. To our knowledge, this claim has not been tested experimentally. We test it in a controlled setting (ResNet-18, CIFAR-10 and CIFAR-100), changing only the hinge and sweeping its weight $μ$ over five orders of magnitude. At the default weight ( $μ= 1$ ), the two hinges reach the same linear-probe accuracy on CIFAR-10 (about 89.5%). Both collapse below a threshold weight, but the standard-deviation hinge's threshold is 35-130 $\times$ lower. CIFAR-100 shows a consistent gap. In training, the variance hinge's gradient fades as the feature variance shrinks, while the standard-deviation hinge's does not until that variance is very small. The standard-deviation hinge therefore buys a safety margin: it keeps training at weights far below the point where the variance hinge collapses.

Figures & tables

Figure 1: CIFAR-10 linear-probe accuracy after 300 epochs versus the variance-term weight

\mu

, for the variance hinge (var; violet circles, solid line) and the standard-deviation hinge (std; green squares, dashed line). Each point is the mean over 3 seeds; error bars show the standard deviation and are often smaller than the marker. Shaded bands mark each hinge’s collapse threshold (std: 0.003–0.01; var: 0.35–0.4). At

\mu=100

training diverged for both hinges (NaN loss; the probe is at chance, 10%). Where the two hinges nearly coincide (

\mu=1

10

and

100

) the std marker is shifted slightly to the right for visibility.

$\mu$	var hinge	std hinge
0 (no var term)	18.79 $\pm$ 3.06
0.001	–	collapsed
0.003	–	collapsed
0.01	–	76.57 $\pm$ 0.21
0.1	collapsed	82.93 $\pm$ 0.61
0.2	collapsed	–

Table 1: Linear-probe accuracy (%) on CIFAR-10 at epoch 300 for the variance (var) and standard-deviation (std) hinge across the variance-term weight

\mu

. Mean over 3 seeds (1, 42, 1000), with the standard deviation in small type. Collapsed runs end between 16% and 23% (chance 10%), the level of the run with no variance term; Table 6 gives their values. ‡ training diverged (loss became NaN; probe at chance). – : not run.

Figure 2: The collapse threshold over the first 100 epochs of training (CIFAR-10). Colors as in Figure 1 . Left and middle: linear-probe accuracy per epoch on the two sides of each hinge’s threshold. Lines are the mean over 3 seeds and the shaded band spans the lowest to the highest seed; the paler curve is the weight that collapses. Right: escape delay, the first epoch at which accuracy exceeds 35%, against

\mu

(mean over 3 seeds). A cross marks a weight at which all three seeds collapse and never reach 35%.

$\mu$	var hinge	std hinge
0.003	–	collapsed
0.01	–	35.74 $\pm$ 0.29
0.1	collapsed	54.52 $\pm$ 0.54
0.3	collapsed	–
0.4	56.34 $\pm$ 0.55	–
0.5	58.90 $\pm$ 0.25	–

Table 2: Linear-probe accuracy (%) on CIFAR-100 at epoch 300, in the format of Table 1 . Collapsed runs end between 2% and 4% (chance 1%); Table 6 gives their values. – : not run.

Figure 3: Push/pull over training (CIFAR-10), on a logarithmic axis: the norm of the gradient of the weighted variance term divided by the norm of the gradient of the invariance term, both taken with respect to the projector outputs of the monitor batch. One box per weight

\mu

, holding every epoch of the 3 seeds (900 values). The box spans the middle half of the values, the line is the median and the whiskers reach the lowest and the highest value. Pale boxes are weights that collapse; colors as in Figure 1 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Term	VICReg paper	Official VICReg code	EB-JEPA
Invariance	summed over dimensions,	averaged over both	as the code
	averaged over images
Variance	the two views added	the two views averaged	the two views added
Covariance	squared off-diagonals	as the paper	averaged over the
	summed, divided by $D$		$D(D-1)$ off-diagonals

Appendix

Table 3: The three loss terms as written in the VICReg paper, as computed by the official VICReg code, and as computed by EB-JEPA.

D

is the embedding size.

VICReg $(\lambda,\mu,\nu)$	Ours $(\mu,\nu)$
$(25,25,1)$ , the default	$(0.5,41)$
$(10,10,1)$	$(0.5,102)$
about $(10,20,1)$ , not in their table	$(1,100)$ , our default

Appendix

Table 4: Weights of the VICReg paper and the same weights in our units, for

D=1024

Encoder	ResNet-18; first convolution $3\times 3$ with stride 1 and padding 2, no max-pooling, no classification layer; output size 512
Projector	three linear layers with hidden width 1024 and output width 1024; batch normalization and ReLU after the first two
Loss	computed on the projector outputs of the two views, under mixed precision
Optimizer	LARS, momentum 0.9, weight decay $10^{-4}$ , $\eta=0.02$ with learning-rate clipping; biases and normalization parameters are excluded from the LARS scaling and from weight decay
Learning rate	0.3, set once per epoch: linear warmup from $3\times 10^{-5}$ over 10 epochs, then cosine decay to 0
Batch size, epochs	256 (last incomplete batch dropped), 300
Precision	bfloat16 automatic mixed precision

Appendix

Table 5: Settings shared by all runs of the main grid. The loss weights are given in Section 3 .

	CIFAR-10		CIFAR-100
$\mu$	var hinge	std hinge	var hinge	std hinge
0 (no var term)	18.79 $\pm$ 3.06		–	–
0.001	–	16.97 $\pm$ 0.29	–	–
0.003	–	18.58 $\pm$ 1.52	–	3.37 $\pm$ 0.42
0.01	–	76.57 $\pm$ 0.21	–	35.74 $\pm$ 0.29
0.1	17.95 $\pm$ 0.76	82.93 $\pm$ 0.61	3.26 $\pm$ 0.35	54.52 $\pm$ 0.54

Appendix

Table 6: Linear-probe accuracy (%) at epoch 300 for every weight

\mu

, with the values of the runs that Tables 1 and 2 mark as collapsed. Mean over 3 seeds (1, 42, 1000), with the standard deviation in small type. Italic : collapsed run (CIFAR-10 chance 10%, CIFAR-100 chance 1%). ‡ training diverged (loss became NaN; probe at chance). – : not run.

Figure 4: RankMe ( Garrido et al., 2023a ) of the final model against the weight

\mu

(CIFAR-10): the exponential of the entropy of the normalized singular values of the embeddings of the 10,000 validation images, after subtracting their mean. Solid lines are the encoder output (512 dimensions, so at most 512) and dashed lines the projector output (1024 dimensions, at most 1024). Each point is the mean of the 3 seeds, which lie within 7 of each other at every point; colors as in Figure 1 . Weights at which the runs collapse (var hinge

\mu\leq 0.35

, std hinge

\mu\leq 0.003

) are left out: their embeddings are at numerical resolution and have no meaningful rank.

Self-Supervised Learning

Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.

See all 261 papers

Weak-SIGReg: Covariance Regularization for Stable Deep Learning
Mar 6, 2026Habibullah AkbarNeural Network Optimization Representation Collapse
Kreasof AI, Jakarta, Indonesia
VISReg: Variance-Invariance-Sketching Regularization for JEPA training
Jun 1, 2026Haiyu Wu, Randall Balestriero, Morgan LevineRepresentation Collapse Joint-Embedding Predictive Architecture
Altos Labs · Brown University
When Does Margin Clamping Affect Training Variance? Dataset-Dependent Effects in Contrastive Forward-Forward Learning
Mar 1, 2026Joshua SteierNeural Network Training Dynamics Supervised Contrastive Learning
Independent Researcher

Mar 6, 2026J/K move · Enter open · S save

Weak-SIGReg: Covariance Regularization for Stable Deep Learning

Habibullah Akbar

Kreasof AI, Jakarta, Indonesia

Neural Network Optimization Representation Collapse

Open paper AskPDF arXiv

Modern neural network optimization relies heavily on architectural priorssuch as Batch Normalization and Residual connectionsto stabilize training dynamics. Without these, or in low-data regimes with aggressive augmentation, low-bias architectures like Vision Transformers (ViTs) often suffer from optimization collapse. This work adopts Sketched Isotropic Gaussian Regularization (SIGReg), recently introduced in the LeJEPA self-supervised framework, and repurposes it as a general optimization stabilizer for supervised learning. While the original formulation targets the full characteristic function, a computationally efficient variant is derived, Weak-SIGReg, which targets the covariance matrix via random sketching. Inspired by interacting particle systems, representation collapse is viewed as stochastic drift; SIGReg constrains the representation density towards an isotropic Gaussian, mitigating this drift. Empirically, SIGReg recovers the training of a ViT on CIFAR-100 from a collapsed 20.73% to 72.02% accuracy without architectural hacks and significantly improves the convergence of deep vanilla MLPs trained with pure SGD. Code is available at github.com/kreasof-ai/sigreg.

Abstract

Self-Supervised Learning

Explore similar work

Weak-SIGReg: Covariance Regularization for Stable Deep Learning

VISReg: Variance-Invariance-Sketching Regularization for JEPA training

When Does Margin Clamping Affect Training Variance? Dataset-Dependent Effects in Contrastive Forward-Forward Learning