Tunable Soft Equivariance with Guarantees

Md Ashiqur Rahman; Lim Jun Hao; Jeremiah Jiang; Teck-Yian Lim; Raymond A. Yeh

cs.CVMar 27, 2026

Tunable Soft Equivariance with Guarantees

Authors: Md Ashiqur Rahman, Lim Jun Hao, Jeremiah Jiang, Teck-Yian Lim, Raymond A. Yeh

Organizations: Purdue University · DSO National Laboratories

Abstract

Equivariance is a fundamental property in computer vision models, yet strict equivariance is rarely satisfied in real-world data, which can limit a model's performance. Controlling the degree of equivariance is therefore desirable. We propose a general framework for constructing soft equivariant models by projecting the model weights into a designed subspace. The method applies to any pre-trained architecture and provides theoretical bounds on the induced equivariance error. Empirically, we demonstrate the effectiveness of our method on multiple pre-trained backbones, including ViT and ResNet, across image classification, semantic segmentation, and human-trajectory prediction tasks. Notably, our approach improves the performance while simultaneously reducing equivariance error on the competitive ImageNet benchmark.

Figures & tables

Figure 2: Tunable softness results (cAcc & iErr). Compared to RPP, ours achieves comparable performance with better iErr across two groups.

	Arch.	ViT [ 12 ]				DINOv2 [ 43 ]				ResNet-50 [ 21 ]
		Acc( $\Delta$ ) $\uparrow$	aAcc $\uparrow$	cAcc $\uparrow$	iErr $\downarrow$	Acc( $\Delta$ ) $\uparrow$	aAcc $\uparrow$	cAcc $\uparrow$	iErr $\downarrow$	Acc( $\Delta$ ) $\uparrow$	aAcc $\uparrow$	cAcc $\uparrow$	iErr $\downarrow$
CIFAR10	Base	97.79 (00.00)	96.62	97.20	07.21	98.81 (00.00)	97.27	98.04	08.16	96.32 (00.00)	91.15	93.70	27.29
	Canon.	93.23 (-04.56)	92.62	92.92	10.97	97.16 (-01.65)	96.88	97.02	05.58	91.11 (-05.21)	90.66	90.89	10.32
	Ours	97.79 (00.00)	96.69	97.24	07.02	98.82 (00.01)	97.43	98.12	07.03	96.61 (00.29)	91.28	93.91	24.20
CIFAR100	Base	86.36 (00.00)	84.09	85.22	23.81	91.01 (00.00)	85.52	88.22	43.02	83.10 (00.00)	73.73	78.28	59.36
	Canon.	78.42 (-07.94)	76.83	77.62	24.55	84.39 (-06.62)	83.66	84.02	20.86	72.90 (-10.20)	72.11	72.50	19.07

Table 1: Performance on CIFAR10 and CIFAR100 across various backbones. Acc(

\Delta

) is top-1 accuracy with

\Delta

vs. Base; aAcc is accuracy under augmentation; cAcc is the combined accuracy; iErr is invariance error (

\times 10^{-2}

, lower is better). We observe that Ours offers a strong performance while maintaining low invariance error.

Arch.	ViT [ 12 ]				DINOv2 [ 43 ]				ResNet-50 [ 21 ]
	Acc( $\Delta$ ) $\uparrow$	aAcc $\uparrow$	cAcc $\uparrow$	iErr $\downarrow$	Acc( $\Delta$ ) $\uparrow$	aAcc $\uparrow$	cAcc $\uparrow$	iErr $\downarrow$	Acc( $\Delta$ ) $\uparrow$	aAcc $\uparrow$	cAcc $\uparrow$	iErr $\downarrow$
Base	81.67 (00.00)	77.29	79.40	00.36	84.27 (00.00)	82.82	83.52	00.13	77.91 (00.00)	75.12	76.48	00.24
Canon.	76.51 (-05.16)	75.81	76.15	00.15	83.22 (-01.06)	82.54	82.87	00.07	72.07 (-05.84)	70.89	71.46	00.24
Ours	82.28 (00.61)	80.56	81.40	00.15	85.31 (01.04)	84.44	84.87	00.05	77.96 (00.06)	75.52	76.72	00.11

Table 2: ImageNet results with various backbones. Acc(

\Delta

) is top-1 accuracy with

\Delta

vs. Base; aAcc is accuracy under augmentation; cAcc is the combined accuracy; iErr is invariance error (

\times 10^{-2}

, lower is better).

Arch.	ViT [ 12 ]			DINOv2 [ 43 ]			ResNet-50 [ 21 ]
	$15^{\circ}$	$45^{\circ}$	$60^{\circ}$	$15^{\circ}$	$45^{\circ}$	$60^{\circ}$	$15^{\circ}$	$45^{\circ}$	$60^{\circ}$
Base	79.04	74.83	72.66	83.63	81.78	80.37	75.77	73.42	70.70
Canon.	75.91	76.01	75.92	82.57	82.57	82.42	70.93	71.01	70.78
Ours	81.23	79.36	77.55	84.80	84.08	82.74	76.47	73.70	71.27

Table 3: Aug. accuracy (aAcc) with various angles on ImageNet.

Arch.	ViT [ 12 ]				DINOv2 [ 43 ]				SegFormer [ 71 ]
	mIoU( $\Delta$ ) $\uparrow$	aIoU $\uparrow$	cIoU $\uparrow$	eErr $\downarrow$	mIoU( $\Delta$ ) $\uparrow$	aIoU $\uparrow$	cIoU $\uparrow$	eErr $\downarrow$	mIoU( $\Delta$ ) $\uparrow$	aIoU $\uparrow$	cIoU $\uparrow$	eErr $\downarrow$
Base	73.40 (00.00)	70.09	71.73	12.31	89.57 (00.00)	88.10	88.83	04.06	65.34 (00.00)	61.17	63.22	11.03
Canon.	65.36 (-08.03)	61.93	63.62	20.39	83.48 (-06.09)	82.73	83.11	44.93	57.50 (-07.84)	55.23	56.36	24.82
Ours	74.78 (01.38)	71.61	73.18	11.12	89.48 (-00.09)	88.70	89.09	03.70	66.34 (00.99)	62.52	64.40	10.64

Table 4: Segmentation Performance on PASCAL VOC [ 15 ] . mIoU (

\Delta

) is the mean intersection over union with

\Delta

vs. Base; aIoU is the augmented mIoU; cIoU is the combined mIoU; eErr is equivariance error (

\times 10^{-2}

, lower is better.)

Scenes	ETH			UNIV			ZARA1			ZARA2			HOTEL
	cADE $\downarrow$	cFDE $\downarrow$	eErr $\downarrow$	cADE $\downarrow$	cFDE $\downarrow$	eErr $\downarrow$	cADE $\downarrow$	cFDE $\downarrow$	eErr $\downarrow$	cADE $\downarrow$	cFDE $\downarrow$	eErr $\downarrow$	cADE $\downarrow$	cFDE $\downarrow$	eErr $\downarrow$
Base [ 19 ]	4.73	6.15	1.68	7.91	8.16	0.73	3.61	4.68	1.14	3.17	3.79	1.17	5.84	6.67	2.50
EqAuto [ 10 ]	5.40	7.33	0.00	8.16	8.33	0.00	3.66	4.98	0.00	2.94	3.63	0.00	6.16	6.78	0.00
Ours	4.58	6.23	1.42	7.85	8.07	0.69	3.40	4.67	0.39	2.91	3.60	0.24	5.69	6.26	0.41

Table 5: Prediction Error on Human Trajectory Datasets. cADE and cFDE are combined metrics. eErr is equivariance error in

\times 10^{-2}

Size		4×4	6×6	8×8	10×10	12×12	14×14
SVD		$1.7\!\cdot\!10^{-1}$	$5.0\!\cdot\!10^{-1}$	$1.0\!\cdot\!10^{1}$	$4.5\!\cdot\!10^{1}$	$2.8\!\cdot\!10^{2}$	$8.9\!\cdot\!10^{2}$
Schur		$\bf 4.0\!\cdot\!10^{-3}$	$\bf 1.0\!\cdot\!10^{-2}$	$\bf 2\!\cdot\!10^{-2}$	$\bf 5\!\cdot\!10^{-2}$	$\bf 1.3\!\cdot\!10^{-1}$	$\bf 2.5\!\cdot\!10^{-1}$

Table 6: Runtime (s) of Schur vs. SVD to compute the projection operator across input sizes.

Hard Threshold				Soft Threshold
mIoU $\uparrow$	aIoU $\uparrow$	cIoU $\uparrow$	eErr $\downarrow$	mIoU $\uparrow$	aIoU $\uparrow$	cIoU $\uparrow$	eErr $\downarrow$
73.92	69.70	71.78	11.74	74.78	71.61	73.18	11.12

Table 7: Hard vs. Soft threshold performance for image segmentation on PASCAL VOC [ 15 ] .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of condition of strict equivariance condition on the weight

\Theta^{\prime}

which is described in Lemma 1 .

Figure 6: Distribution of the norm of the Jacobian and the relative equivariance metric (Eq. ( 6 )) from a pre-trained ResNet-18 model on ImageNet. The relative equivariance error does not collapse or explode.

Method	ETH						UNIV						ZARA1						ZARA2						HOTEL
	ADE $\downarrow$	aADE $\downarrow$	cADE $\downarrow$	FDE $\downarrow$	aFDE $\downarrow$	cFDE $\downarrow$	ADE $\downarrow$	aADE $\downarrow$	cADE $\downarrow$	FDE $\downarrow$	aFDE $\downarrow$	cFDE $\downarrow$	ADE $\downarrow$	aADE $\downarrow$	cADE $\downarrow$	FDE $\downarrow$	aFDE $\downarrow$	cFDE $\downarrow$	ADE $\downarrow$	aADE $\downarrow$	cADE $\downarrow$	FDE $\downarrow$	aFDE $\downarrow$	cFDE $\downarrow$	ADE $\downarrow$	aADE $\downarrow$	cADE $\downarrow$	FDE $\downarrow$	aFDE $\downarrow$	cFDE $\downarrow$
Base [ 19 ]	4.67	4.80	4.73	6.04	6.27	6.15	7.90	7.92	7.91	8.16	8.17	8.16	3.57	3.65	3.61	4.63	4.73	4.68	3.18	3.15	3.17	3.83	3.76	3.79	5.83	5.85	5.84	6.64	6.71	6.67
EqAuto [ 10 ]	5.40	5.40	5.40	7.33	7.33	7.33	8.16	8.16	8.16	8.33	8.33	8.33	3.66	3.66	3.66	4.98	4.98	4.98	2.94	2.94	2.94	3.63	3.63	3.63	6.16	6.16	6.16	6.78	6.78	6.78
Ours	4.50	4.66	4.58	6.10	6.35	6.23	7.82	7.88	7.85	8.04	8.09	8.07	3.39	3.41	3.40	4.65	4.68	4.67	2.91	2.91	2.91	3.60	3.60	3.60	5.69	5.69	5.69	6.26	6.25	6.26

Appendix

Table 8: Human Trajectory Prediction Results (ETH/UCY). All metrics are lower is better.

Model	$\gamma=0.3$	$\gamma=0.4$	$\gamma=0.5$
EMLP [ 17 ]	0.78	0.88	1.00
EMLP-RPP [ 16 ]	0.79	0.81	0.84
MLP	1.21	1.20	1.29
Ours	0.72	0.74	0.81

Appendix

Table 9: Relative MSE (

10^{-1}

) for varying

\gamma

on Synthetic

O(5)

benchmark.

Table 13

Optimization
Hyperparameter	Value
Optimizer	AdamW
Learning Rate	$1e-4$
Weight Decay	0.05-0.04
Epochs	15
Gradient Clipping	1.0

Appendix

Table 14: Training Hyperparameters for ImageNet-1k

Figure 7: Sample trajectories from the ETH/UYC dataset. Solid circles indicate observed past positions, and hollow circles show the future. Square markers denote starting points. Each line with a distinct color corresponds to a different pedestrian in the scene.

Explore similar work

Soft Equivariance Regularization for Invariant Self-Supervised Learning
Mar 4, 2026Joohyung Lee, Changhun Kim, Hyunsu Kim +2Self-Supervised Visual Representation Learning Equivariant Representation Learning
AITRICS · Voronoi Inc. · Korea Advanced Institute of Science and Technology (KAIST)
Normalization Equivariance for Arbitrary Backbones, with Application to Image Denoising
May 5, 2026Youssef Saied, François FleuretImage Corruption Robustness Neural Network Robustness
Department of Computer Science, University of Geneva, Switzerland · Meta FAIR.
Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group
Jun 2, 2026Hongbo WangWorld Models for Robotics World Models
Department of Mathematics, Stony Brook University, Stony Brook, NY 11794, USA

Mar 4, 2026J/K move · Enter open · S save

Soft Equivariance Regularization for Invariant Self-Supervised Learning

Joohyung Lee, Changhun Kim, Hyunsu Kim, Kwanhyung Lee+1

AITRICS · Voronoi Inc. · Korea Advanced Institute of Science and Technology (KAIST)

Self-Supervised Visual Representation Learning Equivariant Representation Learning Invariant Representation Learning Equivariant Neural Networks

Open paper AskPDF arXiv

Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations. While effective for recognition, enforcing strong invariance can suppress transformation-dependent structure that is useful for robustness to geometric perturbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-based SSL with equivariance objectives, but these objectives are often imposed on the same final representation. We empirically observe a trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k linear evaluation, motivating a layer-decoupled design. Motivated by this trade-off, we propose Soft Equivariance Regularization (SER), a plug-in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged on the final embedding, while softly encouraging equivariance on an intermediate spatial token map via analytically specified group actions

ρ_g

applied directly in feature space. SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008x training FLOPs. On ImageNet-1k ViT-S/16 pretraining, SER improves MoCo-v3 by +0.84 Top-1 in linear evaluation under a strictly matched 2-view setting and consistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-1k linear-eval Top-1 among the compared invariance+equivariance add-ons. SER further improves ImageNet-C/P by +1.11/+1.22 Top-1 and frozen-backbone COCO detection by +1.7 mAP. Finally, applying the same layer-decoupling recipe to existing invariance+equivariance baselinesimproves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance.

Abstract

Explore similar work

Soft Equivariance Regularization for Invariant Self-Supervised Learning

Normalization Equivariance for Arbitrary Backbones, with Application to Image Denoising

Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group