Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations. While effective for recognition, enforcing strong invariance can suppress transformation-dependent structure that is useful for robustness to geometric perturbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-based SSL with equivariance objectives, but these objectives are often imposed on the same final representation. We empirically observe a trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k linear evaluation, motivating a layer-decoupled design. Motivated by this trade-off, we propose Soft Equivariance Regularization (SER), a plug-in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged on the final embedding, while softly encouraging equivariance on an intermediate spatial token map via analytically specified group actions ρg applied directly in feature space. SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008x training FLOPs. On ImageNet-1k ViT-S/16 pretraining, SER improves MoCo-v3 by +0.84 Top-1 in linear evaluation under a strictly matched 2-view setting and consistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-1k linear-eval Top-1 among the compared invariance+equivariance add-ons. SER further improves ImageNet-C/P by +1.11/+1.22 Top-1 and frozen-backbone COCO detection by +1.7 mAP. Finally, applying the same layer-decoupling recipe to existing invariance+equivariance baselinesimproves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance.
Figures & tables
Figure 1: Overview of SER. For each image in b2 , we sample two views from the equivariant-view policy Teq . We decompose each sampled transform into a geometric component g∈G and a photometric component (e.g., color jitter), and denote by g1,g2∈G the geometric parts of the two views. We use the relative transform g=g2g1−1 to align intermediate token maps in feature space. For clarity, the independently sampled photometric components in Teq are omitted in the diagram.
Figure 2: An overview of the training pipeline. The mini-batch is split into b1 and b2 : b1 uses the baseline SSL augmentation policy T (including cropping), while b2 uses an equivariant-view policy Teq that disables cropping and adds discrete 90∘ rotations while retaining the baseline photometric jitter. Both b1 and b2 contribute to the baseline invariance loss. SER additionally applies an equivariance regularizer to an intermediate spatial token map using only the invertible geometric component of the b2 transforms to define the feature-space action ρg .
View
Algorithm
Param (M)
ImageNet-1k
ImageNet-Sketch
ImageNet-V2
ImageNet-R
3DIEBench
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
2 view
MoCo-v3
42.9
68.44 ±0.07
88.02 ±0.04
17.65
31.87
56.54
78.68
18.59
30.08
68.43
91.96
+ AugSelf
43.7
67.55 ±0.05
87.62 ±0.05
13.30
25.35
53.74
76.68
17.62
28.66
64.97
90.73
+ STL
62.2
65.49 ±0.12
85.91 ±0.08
15.40
28.96
55.43
78.02
17.22
28.49
-
-
+ SER
43.4
69.28 ±0.01
88.79 ±0.02
17.68
32.54
56.95
79.29
18.95
30.72
70.17
92.78
3 view
+ EquiMod
43.3
68.95 ±0.02
88.87 ±0.01
14.81
28.11
56.31
79.93
16.54
27.32
67.97
91.97
Table 1: Top-1 and top-5 accuracy (%) under linear evaluation. Note that all equivariant representation learning methods use MoCo ( He et al., 2020 ) as their baseline, which outperformed DINO and BarlowTwins in our setting (see Table 2 ). Concatenated [CLS] tokens from the last 4 layers were used as an input to the linear classifier, following the feature-based evaluations in ( Devlin et al., 2019 ; Caron et al., 2021 ) . ‘View’ refers to the number of crops sampled per image (see Appendix A for more detail). ImageNet-1k scores are averaged over 3 runs.
Algorithm
ImageNet-1k
ImageNet-Sketch
ImageNet-V2
ImageNet-R
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
MoCo-v3
68.44 ±0.07
88.02 ±0.04
17.65
31.87
56.54
78.68
18.59
30.08
+ SER
69.28 ±0.01
88.79 ±0.02
17.68
32.54
56.95
79.29
18.95
30.72
DINO
67.37 ±0.02
87.55 ±0.01
17.13
32.09
55.00
77.38
18.28
30.38
+ SER
67.63 ±0.01
87.56 ±0.01
18.07
34.03
55.19
77.84
18.96
31.55
Barlow Twins
63.34 ±0.03
84.3 ±0.04
10.90
21.17
47.69
70.73
12.30
20.94
Table 2: Top-1 and top-5 accuracy (%) under linear evaluation with different baseline invariant self-supervised learning (SSL) methods. All methods use 2-view augmentation policy, and ImageNet-1k scores are averaged over 3 runs.
Figure 3: Ablation study on the location to regularize towards equivariance (left) and to insert the [CLS] token in the ViT encoder with fixed equivariance regularization layer at the 3rd layer (right). Both Top-1 (left) and Top-5 (right) accuracies peak when the equivariance loss and [CLS] is introduced near the middle of the network.
View
Algorithm
MLP
20-NN
Fine-tune
Top-1
Top-5
Top-1
Top-1
Top-5
2 view
MoCo-v3
67.84
88.37
61.56
73.83
91.91
+ AugSelf
63.24
85.86
60.63
73.50
91.67
+ STL
65.74
86.32
57.34
73.90
91.49
+ SER
68.04
88.37
61.64
74.33
91.79
3 view
+ EquiMod
68.33
88.50
58.28
74.08
91.75
Table 3: Nonlinear evaluations using ImageNet-1k with different equivariant representation learning methods. Note that all equivariant representation learning methods use MoCo as their baseline. View refers to the number of crops sampled per image.
Algorithm
Equivariance Loss Layer
ImageNet
Rotation ↑
Hflip ↑
Scale ↑
MoCo-v3
-
68.44
0.804
0.870
0.945
MoCo + SER
Layer 3
69.28
0.840
0.881
0.945
MoCo + SER
Layer 9
68.72
0.888
0.886
0.957
MoCo + SER
Layer 12
68.18
0.924
0.892
0.957
Table 4: Layer-wise ablation for the effect of equivariance regularization. Moving Lequiv toward deeper layers increases the equivariance score of the final representation but reduces ImageNet-1k linear-eval accuracy, indicating a trade-off when equivariance is enforced closer to the final representation. Higher ↑ indicates greater equivariance.
Method
ImageNet-1k Top-1 (%)
MoCo-v3 (baseline)
68.44±0.07
+SER, λ=0 (control)
68.82±0.01
+SER, λ>0 (full)
69.28±0.01
Table 5: Control experiment isolating the effect of the equivariance regularizer (3 seeds).
Method
Equiv. loss layer
ImageNet-1k Top-1 (%)
EquiMod ( Devillers and Lefort, 2023 )
12 (final)
68.95±0.02
EquiMod ( Devillers and Lefort, 2023 )
3 (intermediate)
69.51±0.02
AugSelf ( Lee et al., 2021 )
12 (final)
67.55±0.05
AugSelf ( Lee et al., 2021 )
3 (intermediate)
68.23±0.06
Table 6: Effect of applying layer-decoupling recipe to prior invariant+equivariant baselines using MoCo-v3 by moving equivariance objective from the final to an intermediate layer (3-seeds average).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Tasks
Methods
Layer 1
Layer 3
Layer 6
Layer 9
Layer 12
Rotation Prediction (%)
MoCo + AugSelf
81.55
96.71
99.71
99.13
68.74
MoCo + STL
79.56
92.8
99.46
99.76
98.08
MoCo + SER
79.97
93.34
99.52
99.73
96.59
HFlip Prediction (%)
MoCo + AugSelf
63.24
85.66
96.48
91.51
62.68
MoCo + STL
62.91
82.09
88.83
97.22
83.27
MoCo + SER
63.02
78.68
92.87
96.59
75.95
Appendix
Table 7: Transformation prediction. Evaluation of transformation label prediction from the learned representation of different layers (see Appendix A for more detail)
r
0 (MoCo-v3)
0.005
0.01
0.02
Top-1 (%)
68.44±0.07
69.08±0.03
69.28±0.01
68.30±0.05
Appendix
Table 8: Sensitivity to the batch partition ratio r and equivariance loss weight λ (ImageNet-1k linear eval, 3 seeds).
Variant
ImageNet-1k Top-1 (%)
SER (full)
69.28±0.01
SER – Rot 90∘
69.17±0.01
SER – Rot 90∘ – HFlip
69.12±0.12
SER – Rot 90∘ – Scale
68.67±0.01
Appendix
Table 9: Ablating geometric group elements used in SER (3 seeds).
Variant
Top-1 (%)
Top-5 (%)
MoCo-v3
68.44±0.07
88.02±0.04
+SER (layer 3)
69.28±0.01
88.79±0.02
+SER (all layers)
69.21±0.05
88.78±0.12
Appendix
Table 10: Applying Lequiv to all transformer blocks vs. a single intermediate layer (3 seeds).
Metric
MoCo
MoCo + SER
MoCo + STL
MoCo + AugSelf
mAP
0.225
0.242
0.221
0.197
mAP@50
0.404
0.428
0.400
0.359
mAP@75
0.222
0.244
0.218
0.192
Appendix
Table 11: COCO object-detection results with a frozen backbone (higher is better).
Algorithm
Noise
Blur
Weather
Digital
Avg.
Gauss.
Shot
Impul.
Defo.
Glass
Motion
Zoom
Snow
Frost
Fog
Bright.
Cont.
Elas.
Pixel
JPEG
MoCo-v3
39.18
37.81
36.09
33.51
13.85
31.49
25.86
30.61
30.03
35.02
62.81
52.00
54.65
55.78
53.37
39.47
+ AugSelf
34.91
31.81
31.44
35.06
17.12
34.28
27.67
31.99
28.50
35.01
61.99
50.64
55.48
54.88
53.17
38.93
+ STL
17.78
16.26
14.65
29.51
15.13
27.33
25.77
29.50
27.60
34.40
61.81
48.84
54.31
46.40
50.63
33.33
+ SER
39.42
38.30
36.85
36.23
15.91
34.90
27.35
30.71
30.12
36.04
63.60
52.81
55.85
56.63
54.03
40.58
+ EquiMod †
34.33
32.59
31.95
31.81
15.37
31.76
27.38
29.08
25.73
31.38
61.94
47.41
55.07
53.58
52.40
37.45
Appendix
Table 12: Top-1 accuracy comparison on ImageNet-C, including 15 types of common corruptions, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .
Algorithm
Noise
Blur
Weather
Digital
Avg.
Gau. N.
Shot
Speck.
Motion
Zoom
Gau. B.
Snow
Spatter
Bright.
Trans.
Rot.
Tilt
Scale
Shear
MoCo-v3
67.85
67.85
67.97
57.31
68.30
68.28
56.57
66.57
63.43
68.05
64.55
67.58
45.18
65.32
63.91
+ AugSelf
67.44
67.47
67.47
57.51
67.76
67.81
56.77
66.15
60.77
67.39
64.41
67.10
47.15
64.96
63.58
+ STL
66.19
66.14
66.15
54.97
66.29
66.32
54.38
64.71
61.16
66.00
62.07
65.86
42.53
63.17
61.85
+ SER
68.97
69.01
69.00
59.09
69.46
69.35
58.02
67.86
64.58
69.04
65.76
68.60
46.78
66.34
65.13
+ EquiMod †
68.60
68.70
68.78
57.08
69.17
69.13
56.97
67.51
60.88
68.75
65.18
68.27
47.04
66.17
64.44
Appendix
Table 13: Top-1 accuracy comparison on ImageNet-P, including 14 perturbation types, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .
Algorithm
Noise
Blur
Weather
Digital / Geometric
Avg.
G.Nse
Shot
Spkl
Mot.
Zoom
G.Blr
Snow
Spat
Brt.
Tran
Rot
Tilt
Scal
Shear
DINO
66.69
66.67
66.70
52.74
67.02
66.94
55.83
65.60
61.27
66.70
62.73
66.49
42.96
63.51
62.27
+ SER
67.39
67.31
67.41
54.41
67.69
67.66
56.19
66.06
62.13
67.22
63.33
67.02
42.76
64.31
62.92
Barlow Twins
60.09
60.08
60.12
42.25
60.53
60.53
46.37
58.84
52.92
60.22
55.47
59.37
33.15
56.79
54.77
+ SER
63.85
63.93
63.91
49.18
64.29
64.19
50.09
62.34
57.93
63.89
59.63
63.38
37.29
60.56
58.89
Appendix
Table 14: Experiments with various SSL algorithms. Top-1 accuracy (%) on ImageNet-P . All models are trained with the setting addressed in Section 4.1 . See Table 13 for the results from MoCo.
Algorithm
Noise
Blur
Weather
Digital
Avg.
Gauss.
Shot
Impul.
Defo.
Glass
Motion
Zoom
Snow
Frost
Fog
Bright.
Cont.
Elas.
Pixel
JPEG
MoCo-v3
63.30
61.68
59.76
56.22
28.46
53.52
45.98
52.52
50.54
58.64
84.47
76.74
77.38
79.44
77.76
61.76
+ AugSelf
58.96
55.21
54.61
59.06
34.52
57.55
48.71
55.05
49.48
59.30
84.39
76.26
78.25
79.15
78.02
61.90
+ STL
37.10
34.36
32.04
52.71
31.08
48.45
46.35
50.96
47.60
59.05
84.07
74.74
76.96
71.37
75.66
54.83
+ SER
64.16
62.70
61.21
59.92
32.28
58.17
48.22
53.23
51.26
60.49
85.68
77.94
78.55
80.52
78.58
63.53
+ EquiMod †
58.63
56.28
55.34
55.28
31.48
54.45
48.20
51.11
45.88
55.49
84.63
73.66
78.51
78.46
77.62
60.33
Appendix
Table 15: Top-5 accuracy comparison on ImageNet-C, including 15 types of common corruptions, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .
Algorithm
Noise
Blur
Weather
Digital
Avg.
Gau. N.
Shot
Speck.
Motion
Zoom
Gau. B.
Snow
Spatter
Bright.
Trans.
Rot.
Tilt
Scale
Shear
MoCo-v3
87.75
87.81
87.82
80.34
87.91
87.94
79.22
86.75
84.54
87.72
85.16
87.48
69.08
85.84
84.67
+ AugSelf
87.58
87.50
87.59
80.81
87.73
87.67
79.70
86.50
83.07
87.66
85.25
87.31
71.44
85.83
84.69
+ STL
86.67
86.65
86.66
78.33
86.82
86.77
77.62
85.59
83.21
86.50
83.81
86.50
66.31
84.75
83.30
+ SER
88.62
88.59
88.60
82.01
88.66
88.68
80.56
87.58
85.59
88.52
86.01
88.17
71.25
86.78
85.69
+ EquiMod †
88.78
88.79
88.75
80.86
88.93
88.98
80.16
87.95
83.41
88.68
86.38
88.51
71.89
87.13
85.66
Appendix
Table 16: Top-5 accuracy comparison on ImageNet-P, including 14 perturbation types, for our method and other equivariant representation learning methods built upon the invariant representation learning baseline MoCo ( He et al., 2020 ) .
Figure 4: t-SNE visualization of latent space features from 20 randomly sampled ImageNet-1k classes, comparing (a) MoCo-v3 (trained with invariance loss alone) and (b) MoCo-v3 + Ours. Our method promotes better class clustering, demonstrating that incorporating equivariance benefits downstream tasks requiring invariance.
Figure 5: t-SNE visualization of latent space features from 20 randomly sampled ImageNet-C classes under defocus blur corruption, comparing (a) MoCo-v3 (trained with invariance loss alone) and (b) MoCo-v3 + Ours. Our method maintains better class clustering under corruption, demonstrating robustness benefits of incorporating equivariance.
Method
Per-image FLOPs
Relative overhead
MoCo-v3
18.48G
1.0x
+ SER
18.63G
1.008x
Appendix
Table 17: Computation overhead. Measured FLOPs includes both forward and backward pass with a 2-view augmentation policy, and "Relative overhead" is the relative FLOPs to vanilla MoCo-v3. FLOPs for SER were measured for the overall mini-batch computation and divided by the mini-batch sample number (including both b1 and b2 as illustrated in Figure 2 )
Equivariance is a fundamental property in computer vision models, yet strict equivariance is rarely satisfied in real-world data, which can limit a model's performance. Controlling the degree of equivariance is therefore desirable. We propose a general framework for constructing soft equivariant models by projecting the model weights into a designed subspace. The method applies to any pre-trained architecture and provides theoretical bounds on the induced equivariance error. Empirically, we demonstrate the effectiveness of our method on multiple pre-trained backbones, including ViT and ResNet, across image classification, semantic segmentation, and human-trajectory prediction tasks. Notably, our approach improves the performance while simultaneously reducing equivariance error on the competitive ImageNet benchmark.
Existing self-supervised learning (SSL) methods primarily learn object-invariant representations but often neglect the spatial structure and relationships among object parts. To address this limitation, we introduce Spatial Prediction (SP), a spatially aware pretext regression task that predicts the relative position and scale between a pair of disentangled local views from the same image. By modeling part-to-part relationships in a continuous geometric space, SP encourages representations to capture fine-grained spatial dependencies beyond invariant categorical semantics, thereby learning the compositional structure of visual scenes. SP is implemented as a decoupled plug-in and can be seamlessly integrated into diverse SSL frameworks. Extensive experiments show consistent improvements across image recognition, fine-grained classification, semantic segmentation, and depth estimation, as well as substantial gains in out-of-distribution robustness for object recognition. To evaluate spatial reasoning, we introduce (1) a position and scale prediction task on image patch pairs and (2) a jigsaw understanding task requiring patch reordering and recognition after reconstruction. Strong performance on these tasks indicates improved spatial structure and geometric awareness. Overall, explicitly modeling spatial information provides an effective inductive bias for SSL, leading to more structured representations and better generalization. Code and models will be released.
Yang Shen, Yusen Cai, Weronika Hryniewska-Guzik +2
Nanyang Technological University, Singapore · Warsaw University of Technology, Poland
Self-supervised learning methods prevent embedding collapse via modeling heuristics or explicit regularization of the embedding space. Among the latter, VICReg decomposes regularization into variance and covariance objectives, offering flexibility and interpretability. However, covariance captures only second-order statistics -- encouraging decorrelation but failing to enforce the full distributional shape needed for stable training. Sketching-based methods such as SIGReg address this by aligning embeddings to an isotropic Gaussian, but lack flexibility and suffer from vanishing gradients under collapse. We propose Variance-Invariance-Sketching Regularization (VISReg), which replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape, while retaining a variance term for scale control. By decoupling scale and shape, VISReg combines VICReg's flexibility with the distributional rigor of sketching methods, providing robust gradients even under collapse. We show that VISReg scales linearly, outperforms existing regularization on low-quality datasets, and is resilient to long-tailed and low-rank regimes. Pre-trained on ImageNet-1K, VISReg achieves state-of-the-art performance on out-of-distribution datasets. Pre-trained on ImageNet-22K, it matches DINOv2's OOD performance despite the latter using 10x more data (LVD-142M). Project and code: https://haiyuwu.github.io/visreg.