Soft Equivariance Regularization for Invariant Self-Supervised Learning

Joohyung Lee; Changhun Kim; Hyunsu Kim; Kwanhyung Lee; Juho Lee

cs.CVMar 4, 2026

Soft Equivariance Regularization for Invariant Self-Supervised Learning

Authors: Joohyung Lee, Changhun Kim, Hyunsu Kim, Kwanhyung Lee, Juho Lee

Organizations: AITRICS · Voronoi Inc. · Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations. While effective for recognition, enforcing strong invariance can suppress transformation-dependent structure that is useful for robustness to geometric perturbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-based SSL with equivariance objectives, but these objectives are often imposed on the same final representation. We empirically observe a trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k linear evaluation, motivating a layer-decoupled design. Motivated by this trade-off, we propose Soft Equivariance Regularization (SER), a plug-in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged on the final embedding, while softly encouraging equivariance on an intermediate spatial token map via analytically specified group actions $ρ_g$ applied directly in feature space. SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008x training FLOPs. On ImageNet-1k ViT-S/16 pretraining, SER improves MoCo-v3 by +0.84 Top-1 in linear evaluation under a strictly matched 2-view setting and consistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-1k linear-eval Top-1 among the compared invariance+equivariance add-ons. SER further improves ImageNet-C/P by +1.11/+1.22 Top-1 and frozen-backbone COCO detection by +1.7 mAP. Finally, applying the same layer-decoupling recipe to existing invariance+equivariance baselinesimproves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance.

Figures & tables

Figure 1: Overview of SER. For each image in $b_{2}$ , we sample two views from the equivariant-view policy $\mathcal{T}_{\mathrm{eq}}$ . We decompose each sampled transform into a geometric component $g\in\mathcal{G}$ and a photometric component (e.g., color jitter), and denote by $g_{1},g_{2}\in\mathcal{G}$ the geometric parts of the two views. We use the relative transform $g=g_{2}g_{1}^{-1}$ to align intermediate token maps in feature space. For clarity, the independently sampled photometric components in $\mathcal{T}_{\mathrm{eq}}$ are omitted in the diagram.

Figure 2: An overview of the training pipeline. The mini-batch is split into $b_{1}$ and $b_{2}$ : $b_{1}$ uses the baseline SSL augmentation policy $\mathcal{T}$ (including cropping), while $b_{2}$ uses an equivariant-view policy $\mathcal{T}_{\mathrm{eq}}$ that disables cropping and adds discrete $90^{\circ}$ rotations while retaining the baseline photometric jitter. Both $b_{1}$ and $b_{2}$ contribute to the baseline invariance loss. SER additionally applies an equivariance regularizer to an intermediate spatial token map using only the invertible geometric component of the $b_{2}$ transforms to define the feature-space action $\rho_{g}$ .

View	Algorithm	Param (M)	ImageNet-1k		ImageNet-Sketch		ImageNet-V2		ImageNet-R		3DIEBench
			Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
2 view	MoCo-v3	42.9	68.44 ±0.07	88.02 ±0.04	17.65	31.87	56.54	78.68	18.59	30.08	68.43	91.96
	$+$ AugSelf	43.7	67.55 ±0.05	87.62 ±0.05	13.30	25.35	53.74	76.68	17.62	28.66	64.97	90.73
	$+$ STL	62.2	65.49 ±0.12	85.91 ±0.08	15.40	28.96	55.43	78.02	17.22	28.49	-	-
	$+$ SER	43.4	69.28 ±0.01	88.79 ±0.02	17.68	32.54	56.95	79.29	18.95	30.72	70.17	92.78
3 view	$+$ EquiMod	43.3	68.95 ±0.02	88.87 ±0.01	14.81	28.11	56.31	79.93	16.54	27.32	67.97	91.97

Table 1: Top-1 and top-5 accuracy (%) under linear evaluation. Note that all equivariant representation learning methods use MoCo ( He et al., 2020 ) as their baseline, which outperformed DINO and BarlowTwins in our setting (see Table 2 ). Concatenated [CLS] tokens from the last 4 layers were used as an input to the linear classifier, following the feature-based evaluations in ( Devlin et al., 2019 ; Caron et al., 2021 ) . ‘View’ refers to the number of crops sampled per image (see Appendix A for more detail). ImageNet-1k scores are averaged over 3 runs.

Algorithm	ImageNet-1k		ImageNet-Sketch		ImageNet-V2		ImageNet-R
Algorithm	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
MoCo-v3	68.44 ±0.07	88.02 ±0.04	17.65	31.87	56.54	78.68	18.59	30.08
$+$ SER	69.28 ±0.01	88.79 ±0.02	17.68	32.54	56.95	79.29	18.95	30.72
DINO	67.37 ±0.02	87.55 ±0.01	17.13	32.09	55.00	77.38	18.28	30.38
$+$ SER	67.63 ±0.01	87.56 ±0.01	18.07	34.03	55.19	77.84	18.96	31.55
Barlow Twins	63.34 ±0.03	84.3 ±0.04	10.90	21.17	47.69	70.73	12.30	20.94

Table 2: Top-1 and top-5 accuracy (%) under linear evaluation with different baseline invariant self-supervised learning (SSL) methods. All methods use 2-view augmentation policy, and ImageNet-1k scores are averaged over 3 runs.

Figure 3: Ablation study on the location to regularize towards equivariance (left) and to insert the [CLS] token in the ViT encoder with fixed equivariance regularization layer at the 3rd layer (right). Both Top-1 (left) and Top-5 (right) accuracies peak when the equivariance loss and [CLS] is introduced near the middle of the network.

View	Algorithm	MLP		20-NN	Fine-tune
		Top-1	Top-5	Top-1	Top-1	Top-5
2 view	MoCo-v3	67.84	88.37	61.56	73.83	91.91
	$+$ AugSelf	63.24	85.86	60.63	73.50	91.67
	$+$ STL	65.74	86.32	57.34	73.90	91.49
	$+$ SER	68.04	88.37	61.64	74.33	91.79
3 view	$+$ EquiMod	68.33	88.50	58.28	74.08	91.75

Table 3: Nonlinear evaluations using ImageNet-1k with different equivariant representation learning methods. Note that all equivariant representation learning methods use MoCo as their baseline. View refers to the number of crops sampled per image.

Algorithm	Equivariance Loss Layer	ImageNet	Rotation $\uparrow$	Hflip $\uparrow$	Scale $\uparrow$
MoCo-v3	-	68.44	0.804	0.870	0.945
MoCo + SER	Layer 3	69.28	0.840	0.881	0.945
MoCo + SER	Layer 9	68.72	0.888	0.886	0.957
MoCo + SER	Layer 12	68.18	0.924	0.892	0.957

Table 4: Layer-wise ablation for the effect of equivariance regularization. Moving

\mathcal{L}_{\text{equiv}}

toward deeper layers increases the equivariance score of the final representation but reduces ImageNet-1k linear-eval accuracy, indicating a trade-off when equivariance is enforced closer to the final representation. Higher

\uparrow

indicates greater equivariance.

Method	ImageNet-1k Top-1 (%)
MoCo-v3 (baseline)	$68.44\pm 0.07$
+SER, $\lambda=0$ (control)	$68.82\pm 0.01$
+SER, $\lambda>0$ (full)	$\bm{69.28\pm 0.01}$

Table 5: Control experiment isolating the effect of the equivariance regularizer (3 seeds).

Method	Equiv. loss layer	ImageNet-1k Top-1 (%)
EquiMod ( Devillers and Lefort, 2023 )	12 (final)	$68.95\pm 0.02$
EquiMod ( Devillers and Lefort, 2023 )	3 (intermediate)	$\bm{69.51\pm 0.02}$
AugSelf ( Lee et al., 2021 )	12 (final)	$67.55\pm 0.05$
AugSelf ( Lee et al., 2021 )	3 (intermediate)	$\bm{68.23\pm 0.06}$

Table 6: Effect of applying layer-decoupling recipe to prior invariant+equivariant baselines using MoCo-v3 by moving equivariance objective from the final to an intermediate layer (3-seeds average).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Tasks	Methods	Layer 1	Layer 3	Layer 6	Layer 9	Layer 12
Rotation Prediction (%)	MoCo + AugSelf	81.55	96.71	99.71	99.13	68.74
	MoCo + STL	79.56	92.8	99.46	99.76	98.08
	MoCo + SER	79.97	93.34	99.52	99.73	96.59
HFlip Prediction (%)	MoCo + AugSelf	63.24	85.66	96.48	91.51	62.68
	MoCo + STL	62.91	82.09	88.83	97.22	83.27
	MoCo + SER	63.02	78.68	92.87	96.59	75.95

Appendix

Table 7: Transformation prediction. Evaluation of transformation label prediction from the learned representation of different layers (see Appendix A for more detail)

$r$	0 (MoCo-v3)	0.005	0.01	0.02
Top-1 (%)	$68.44\pm 0.07$	$69.08\pm 0.03$	$\bm{69.28\pm 0.01}$	$68.30\pm 0.05$

Appendix

Table 8: Sensitivity to the batch partition ratio

r

and equivariance loss weight

\lambda

(ImageNet-1k linear eval, 3 seeds).

Variant	ImageNet-1k Top-1 (%)
SER (full)	$\bm{69.28\pm 0.01}$
SER – Rot $90^{\circ}$	$69.17\pm 0.01$
SER – Rot $90^{\circ}$ – HFlip	$69.12\pm 0.12$
SER – Rot $90^{\circ}$ – Scale	$68.67\pm 0.01$

Appendix

Table 9: Ablating geometric group elements used in SER (3 seeds).

Variant	Top-1 (%)	Top-5 (%)
MoCo-v3	$68.44\pm 0.07$	$88.02\pm 0.04$
+SER (layer 3)	$\bm{69.28\pm 0.01}$	$\bm{88.79\pm 0.02}$
+SER (all layers)	$69.21\pm 0.05$	$88.78\pm 0.12$

Appendix

Table 10: Applying

\mathcal{L}_{\mathrm{equiv}}

to all transformer blocks vs. a single intermediate layer (3 seeds).

Metric	MoCo	MoCo + SER	MoCo + STL	MoCo + AugSelf
mAP	0.225	0.242	0.221	0.197
mAP@50	0.404	0.428	0.400	0.359
mAP@75	0.222	0.244	0.218	0.192

Appendix

Table 11: COCO object-detection results with a frozen backbone (higher is better).

Algorithm	Noise			Blur				Weather				Digital				Avg.
Algorithm	Gauss.	Shot	Impul.	Defo.	Glass	Motion	Zoom	Snow	Frost	Fog	Bright.	Cont.	Elas.	Pixel	JPEG	Avg.
MoCo-v3	39.18	37.81	36.09	33.51	13.85	31.49	25.86	30.61	30.03	35.02	62.81	52.00	54.65	55.78	53.37	39.47
$+$ AugSelf	34.91	31.81	31.44	35.06	17.12	34.28	27.67	31.99	28.50	35.01	61.99	50.64	55.48	54.88	53.17	38.93
$+$ STL	17.78	16.26	14.65	29.51	15.13	27.33	25.77	29.50	27.60	34.40	61.81	48.84	54.31	46.40	50.63	33.33
$+$ SER	39.42	38.30	36.85	36.23	15.91	34.90	27.35	30.71	30.12	36.04	63.60	52.81	55.85	56.63	54.03	40.58
$+$ EquiMod †	34.33	32.59	31.95	31.81	15.37	31.76	27.38	29.08	25.73	31.38	61.94	47.41	55.07	53.58	52.40	37.45

Appendix

Table 12: Top-1 accuracy comparison on ImageNet-C, including 15 types of common corruptions, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .

Algorithm	Noise			Blur			Weather			Digital					Avg.
Algorithm	Gau. N.	Shot	Speck.	Motion	Zoom	Gau. B.	Snow	Spatter	Bright.	Trans.	Rot.	Tilt	Scale	Shear	Avg.
MoCo-v3	67.85	67.85	67.97	57.31	68.30	68.28	56.57	66.57	63.43	68.05	64.55	67.58	45.18	65.32	63.91
$+$ AugSelf	67.44	67.47	67.47	57.51	67.76	67.81	56.77	66.15	60.77	67.39	64.41	67.10	47.15	64.96	63.58
$+$ STL	66.19	66.14	66.15	54.97	66.29	66.32	54.38	64.71	61.16	66.00	62.07	65.86	42.53	63.17	61.85
$+$ SER	68.97	69.01	69.00	59.09	69.46	69.35	58.02	67.86	64.58	69.04	65.76	68.60	46.78	66.34	65.13
$+$ EquiMod †	68.60	68.70	68.78	57.08	69.17	69.13	56.97	67.51	60.88	68.75	65.18	68.27	47.04	66.17	64.44

Appendix

Table 13: Top-1 accuracy comparison on ImageNet-P, including 14 perturbation types, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .

Algorithm	Noise			Blur			Weather		Digital / Geometric						Avg.
Algorithm	G.Nse	Shot	Spkl	Mot.	Zoom	G.Blr	Snow	Spat	Brt.	Tran	Rot	Tilt	Scal	Shear	Avg.
DINO	66.69	66.67	66.70	52.74	67.02	66.94	55.83	65.60	61.27	66.70	62.73	66.49	42.96	63.51	62.27
$+$ SER	67.39	67.31	67.41	54.41	67.69	67.66	56.19	66.06	62.13	67.22	63.33	67.02	42.76	64.31	62.92
Barlow Twins	60.09	60.08	60.12	42.25	60.53	60.53	46.37	58.84	52.92	60.22	55.47	59.37	33.15	56.79	54.77
$+$ SER	63.85	63.93	63.91	49.18	64.29	64.19	50.09	62.34	57.93	63.89	59.63	63.38	37.29	60.56	58.89

Appendix

Table 14: Experiments with various SSL algorithms. Top-1 accuracy (%) on ImageNet-P . All models are trained with the setting addressed in Section 4.1 . See Table 13 for the results from MoCo.

Algorithm	Noise			Blur				Weather				Digital				Avg.
Algorithm	Gauss.	Shot	Impul.	Defo.	Glass	Motion	Zoom	Snow	Frost	Fog	Bright.	Cont.	Elas.	Pixel	JPEG	Avg.
MoCo-v3	63.30	61.68	59.76	56.22	28.46	53.52	45.98	52.52	50.54	58.64	84.47	76.74	77.38	79.44	77.76	61.76
$+$ AugSelf	58.96	55.21	54.61	59.06	34.52	57.55	48.71	55.05	49.48	59.30	84.39	76.26	78.25	79.15	78.02	61.90
$+$ STL	37.10	34.36	32.04	52.71	31.08	48.45	46.35	50.96	47.60	59.05	84.07	74.74	76.96	71.37	75.66	54.83
$+$ SER	64.16	62.70	61.21	59.92	32.28	58.17	48.22	53.23	51.26	60.49	85.68	77.94	78.55	80.52	78.58	63.53
$+$ EquiMod †	58.63	56.28	55.34	55.28	31.48	54.45	48.20	51.11	45.88	55.49	84.63	73.66	78.51	78.46	77.62	60.33

Appendix

Table 15: Top-5 accuracy comparison on ImageNet-C, including 15 types of common corruptions, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .

Algorithm	Noise			Blur			Weather			Digital					Avg.
Algorithm	Gau. N.	Shot	Speck.	Motion	Zoom	Gau. B.	Snow	Spatter	Bright.	Trans.	Rot.	Tilt	Scale	Shear	Avg.
MoCo-v3	87.75	87.81	87.82	80.34	87.91	87.94	79.22	86.75	84.54	87.72	85.16	87.48	69.08	85.84	84.67
$+$ AugSelf	87.58	87.50	87.59	80.81	87.73	87.67	79.70	86.50	83.07	87.66	85.25	87.31	71.44	85.83	84.69
$+$ STL	86.67	86.65	86.66	78.33	86.82	86.77	77.62	85.59	83.21	86.50	83.81	86.50	66.31	84.75	83.30
$+$ SER	88.62	88.59	88.60	82.01	88.66	88.68	80.56	87.58	85.59	88.52	86.01	88.17	71.25	86.78	85.69
$+$ EquiMod †	88.78	88.79	88.75	80.86	88.93	88.98	80.16	87.95	83.41	88.68	86.38	88.51	71.89	87.13	85.66

Appendix

Table 16: Top-5 accuracy comparison on ImageNet-P, including 14 perturbation types, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .

Figure 4: t-SNE visualization of latent space features from 20 randomly sampled ImageNet-1k classes, comparing (a) MoCo-v3 (trained with invariance loss alone) and (b) MoCo-v3 + Ours. Our method promotes better class clustering, demonstrating that incorporating equivariance benefits downstream tasks requiring invariance.

Figure 5: t-SNE visualization of latent space features from 20 randomly sampled ImageNet-C classes under defocus blur corruption, comparing (a) MoCo-v3 (trained with invariance loss alone) and (b) MoCo-v3 + Ours. Our method maintains better class clustering under corruption, demonstrating robustness benefits of incorporating equivariance.

Method	Per-image FLOPs	Relative overhead
MoCo-v3	18.48G	1.0x
$+$ SER	18.63G	1.008x

Appendix

Table 17: Computation overhead. Measured FLOPs includes both forward and backward pass with a 2-view augmentation policy, and "Relative overhead" is the relative FLOPs to vanilla MoCo-v3. FLOPs for SER were measured for the overall mini-batch computation and divided by the mini-batch sample number (including both b1 and b2 as illustrated in Figure 2 )

Explore similar work

Tunable Soft Equivariance with Guarantees
Mar 27, 2026Md Ashiqur Rahman, Lim Jun Hao, Jeremiah Jiang +2Equivariant Neural Networks
Purdue University · DSO National Laboratories
Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning
May 11, 2026Yang Shen, Yusen Cai, Weronika Hryniewska-Guzik +2Visual Spatial Reasoning Self-Supervised Visual Representation Learning
Nanyang Technological University, Singapore · Warsaw University of Technology, Poland
VISReg: Variance-Invariance-Sketching Regularization for JEPA training
Jun 1, 2026Haiyu Wu, Randall Balestriero, Morgan LevineRepresentation Collapse Joint-Embedding Predictive Architecture
Altos Labs · Brown University

Mar 27, 2026J/K move · Enter open · S save

Tunable Soft Equivariance with Guarantees

Md Ashiqur Rahman, Lim Jun Hao, Jeremiah Jiang, Teck-Yian Lim+1

Purdue University · DSO National Laboratories

Equivariant Neural Networks

Open paper AskPDF arXiv

Equivariance is a fundamental property in computer vision models, yet strict equivariance is rarely satisfied in real-world data, which can limit a model's performance. Controlling the degree of equivariance is therefore desirable. We propose a general framework for constructing soft equivariant models by projecting the model weights into a designed subspace. The method applies to any pre-trained architecture and provides theoretical bounds on the induced equivariance error. Empirically, we demonstrate the effectiveness of our method on multiple pre-trained backbones, including ViT and ResNet, across image classification, semantic segmentation, and human-trajectory prediction tasks. Notably, our approach improves the performance while simultaneously reducing equivariance error on the competitive ImageNet benchmark.

Abstract

Explore similar work

Tunable Soft Equivariance with Guarantees

Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning

VISReg: Variance-Invariance-Sketching Regularization for JEPA training