Latent Equivariant Operators for Robust Object Recognition: Promises and Challenges

Minh Dinh; Stéphane Deny

cs.CVFeb 20, 2026

Latent Equivariant Operators for Robust Object Recognition: Promises and Challenges

Authors: Minh Dinh, Stéphane Deny

Organizations: Department of Computer Science, Dartmouth College, Hanover, NH 03755, USA · Departments of Computer Science & Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland

Abstract

Despite the successes of deep learning in computer vision, difficulties persist in recognizing objects that have undergone group-symmetric transformations rarely seen during training $\unicode{x2013}$ for example objects seen in unusual poses, scales, positions, or combinations thereof. Equivariant neural networks are a solution to the problem of generalizing across symmetric transformations, but require knowledge of transformations a priori. An alternative family of architectures proposes to learn equivariant operators in a latent space, from examples of symmetric transformations. Here, using simple datasets of rotated and translated noisy MNIST, we illustrate how such architectures can successfully be harnessed for out-of-distribution classification, thus overcoming the limitations of both traditional and equivariant networks. While conceptually enticing, we discuss challenges ahead on the path of scaling these architectures to more complex datasets. Our code is available at https://github.com/BRAIN-Aalto/equivariant_operator.

Figures & tables

Figure 1: Pipeline for handling single transformation. Two transformed views of the same input are encoded by a shared encoder $f_{E}$ and mapped to a canonical representation using inverse shift operators $\varphi^{-k_{1}}$ and $\varphi^{-k_{2}}$ , yielding embeddings $Z_{1}$ and $Z_{2}$ . The embedding $Z_{1}$ is used for classification via a MLP $f_{\text{CLF}}$ optimized with the cross-entropy loss $\mathcal{L}_{\mathrm{CE}}$ , while a representation consistency loss $\mathcal{L}_{\mathrm{reg}}$ encourages alignment between $Z_{1}$ and $Z_{2}$ that both correspond to the canonical pose. When the operator is learned, we add an extra term $\mathcal{L}_{\mathrm{op}}$ to the loss.

Panel of Figure 2 — Figure 2: Classification accuracy as a function of transformations on MNIST. The shaded region denotes the range of translations observed during training.

Panel of Figure 3 — Figure 3: Test accuracy heatmaps under joint horizontal (rows) and vertical (columns) translations. The bordered cross indicates transformations observed during training.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Pipeline of operator-based classification under compound transformations. Top (Training): A canonical input is transformed along individual axes to generate augmented views. Shared encoders map each view into a representation space, where inverse operators $\varphi^{-k}$ align embeddings back to a canonical pose. Bottom (Inference): Given an input undergoing a compound transformation $T_{x,y}(k_{1},k_{2})$ , the model encodes the input and applies the corresponding inverse operators to recover a canonical representation for classification.

	Degree
Operator	Degree given	k	-12	-10	-8	-6	-4	-2	0	2	4	6	8	10	12	14
Translation on y-axis
$\times$	–	–	18.151	18.519	21.299	45.145	78.490	83.016	83.294	83.072	79.613	51.129	23.357	13.602	15.182	16.717
Fixed	✓	–	95.885	95.985	95.996	95.952	96.018	96.185	96.063	96.074	95.751	96.018	95.807	95.707	95.629	95.707
Fixed	$\times$	1	93.816	93.872	93.816	93.849	94.061	94.216	94.094	94.183	93.894	93.861	93.638	93.616	93.872	93.683
Fixed	$\times$	3	93.594	93.849	93.938	93.861	94.150	94.261	94.606	93.894	93.282	93.582	93.427	93.638	93.538	93.293

Appendix

Table 1: Classification accuracy (%) under translation extrapolation. Results for horizontal (x-axis) and vertical (y-axis) shifts. “Degree given” indicates whether the true translation degree is provided at test time. Blue-shaded columns denote degrees seen during training; unshaded columns are unseen.

		Degree
Operator	Degree given	k	-144	-108	-72	-36	0	36	72	108	144	180
$\times$	–	–	25.170	30.620	74.497	78.456	77.322	79.001	75.075	31.721	26.148	25.648
Fixed	✓	–	95.707	95.751	95.785	95.963	95.918	95.707	95.607	95.696	95.618	95.840
Fixed	$\times$	1	86.042	85.841	86.753	86.965	86.842	86.976	86.720	85.975	85.875	86.564
Fixed	$\times$	3	86.920	87.521	88.644	89.523	89.890	86.831	84.763	84.051	84.740	85.864
Fixed	$\times$	10	87.298	87.443	87.721	88.099	88.511	89.834	88.377	87.821	87.843	87.810

Appendix

Table 2: Classification accuracy (%) under rotation extrapolation. “Degree given” indicates whether the true rotation degree is provided at test time. Blue-shaded columns denote degrees seen during training; unshaded columns are unseen.

Panel of Figure 5 — Figure 5: Extrapolation behavior of operator-based models on MNIST. Solid lines use ground-truth transformation degrees; dashed lines “(auto)” use k-NN pose inference. Shaded regions denote training degrees.

	$N$
$k$	100		200		500		1000		2000		5000
	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$
GT	100.000	95.759	100.000	95.759	100.000	95.759	100.000	95.759	100.000	95.759	100.000	95.759
1	54.196	76.136	60.339	80.185	65.292	83.375	68.912	85.473	71.543	87.029	74.159	88.726
3	51.829	73.990	59.669	79.097	66.285	83.136	70.421	85.512	73.273	87.171	75.949	88.940
10	53.287	75.149	61.767	80.054	68.687	84.356	72.950	86.649	75.746	88.117	78.735	89.771

Appendix

Table 3: k-NN performance on rotated MNIST.

\mathrm{Acc}_{\mathrm{pose}}

(

\uparrow

) and

\mathrm{Acc}_{\mathrm{cls}}

(

\uparrow

) denote pose prediction and downstream classification accuracy, respectively. GT reports the reference performance with ground-truth poses.

	$N$
$k$	100		200		500		1000		2000		5000
	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$
GT	100.000	95.097	100.000	95.097	100.000	95.097	100.000	95.097	100.000	95.097	100.000	95.097
1	65.661	86.931	71.705	89.458	76.762	91.406	79.852	92.351	82.816	93.049	85.395	93.652
3	61.803	84.460	69.530	88.018	76.752	90.774	80.566	91.964	83.767	92.776	86.307	93.501
10	59.041	82.852	68.694	87.245	77.251	90.438	81.514	91.884	84.657	92.752	87.464	93.488

Appendix

Table 4: k-NN performance on x-translated MNIST. Acc

{}_{\text{pose}}

(\uparrow)

and Acc

{}_{\text{cls}}

(\uparrow)

denote pose prediction and downstream classification accuracy, respectively. GT reports the reference performance when ground-truth poses were given. The highest score for each

N

is shown in bold .

	$N$
$k$	100		200		500		1000		2000		5000
	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$	$\mathrm{Acc}_{\text{pose}}$	$\mathrm{Acc}_{\text{cls}}$
GT	100.000	95.913	100.000	95.913	100.000	95.913	100.000	95.913	100.000	95.913	100.000	95.913
1	66.732	87.442	72.056	90.092	77.052	92.161	79.872	93.020	82.513	93.792	85.342	94.532
3	63.211	84.416	71.428	88.416	77.668	91.441	80.967	92.725	83.756	93.651	86.227	94.502
10	62.334	84.603	71.036	88.648	78.322	91.380	82.019	92.753	84.673	93.655	87.319	94.521

Appendix

Table 5: k-NN performance on y-translated MNIST. Acc

{}_{\text{pose}}

(\uparrow)

and Acc

{}_{\text{cls}}

(\uparrow)

denote pose prediction and downstream classification accuracy, respectively. GT reports the reference performance when ground-truth poses were given. The highest score for each

N

is shown in bold .

Explore similar work

Generalized multi-object classification and tracking with sparse feature resonator networks
Mar 23, 2026Lazar Supic, Alec Mullen, E. Paxon FradyObject-Centric Representation Learning Multi-Object Tracking
Redwood Center for Theoretical Neuroscience, UC Berkeley · Neuromorphic Computing Lab, Intel
Measuring Learned Equivariance: An Information Theoretic Approach
Oct 12, 2026Claire Schlesinger, Wolfgang Gatterbauer, Robin WaltersEquivariant Representation Learning
Khoury College of Computer Sciences Northeastern University Boston, MA 02115, USA
Rotational Equivariance in Machine Learning: A Comprehensive Tutorial
Aug 31, 2026Peter Lippmann, Fred A. HamprechtEquivariant Neural Networks Geometric Representation Learning
Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, 69120 Heidelberg, Germany

Mar 23, 2026J/K move · Enter open · S save

Generalized multi-object classification and tracking with sparse feature resonator networks

Lazar Supic, Alec Mullen, E. Paxon Frady

Redwood Center for Theoretical Neuroscience, UC Berkeley · Neuromorphic Computing Lab, Intel

Object-Centric Representation Learning Multi-Object Tracking Invariant Representation Learning Equivariant Representation Learning

Open paper AskPDF arXiv

In visual scene understanding tasks, it is essential to capture both invariant and equivariant structure. While neural networks are frequently trained to achieve invariance to transformations such as translation, this often comes at the cost of losing access to equivariant information - e.g., the precise location of an object. Moreover, invariance is not naturally guaranteed through supervised learning alone, and many architectures generalize poorly to input transformations not encountered during training. Here, we take an approach based on analysis-by-synthesis and factoring using resonator networks. A generative model describes the construction of simple scenes containing MNIST digits and their transformations, like color and position. The resonator network inverts the generative model, and provides both invariant and equivariant information about particular objects. Sparse features learned from training data act as a basis set to provide flexibility in representing variable shapes of objects, allowing the resonator network to handle previously unseen digit shapes from the test set. The modular structure provides a shape module which contains information about the object shape with translation factored out, allowing a simple classifier to operate on centered digits. The classification layer is trained solely on centered data, requiring much less training data, and the network as a whole can identify objects with arbitrary translations without data augmentation. The natural attention-like mechanism of the resonator network also allows for analysis of scenes with multiple objects, where the network dynamics selects and centers only one object at a time. Further, the specific position information of a particular object can be extracted from the translation module, and we show that the resonator can be designed to track multiple moving objects with precision of a few pixels.

Abstract

Explore similar work

Generalized multi-object classification and tracking with sparse feature resonator networks

Measuring Learned Equivariance: An Information Theoretic Approach

Rotational Equivariance in Machine Learning: A Comprehensive Tutorial