REViT-v2: Hierarchical Windowed Roto-reflection Equivariant ViT for Equivariant Feature Extraction

Sheir A. Zaheer; Jihwan Moon; Chan Y. Park

cs.CVOct 6, 2026

REViT-v2: Hierarchical Windowed Roto-reflection Equivariant ViT for Equivariant Feature Extraction

Authors: Sheir A. Zaheer, Jihwan Moon, Chan Y. Park

Organizations: KC Machine Learning Lab, Rep. of Korea · NFOCZ Inc and Seoul National University, Rep. of Korea

Abstract

We propose a scalable roto-reflection-group-equivariant vision transformer based on windowed group-convolutional self-attention and a hierarchical feature architecture. We demonstrate that our approach can be scaled to group-equivariant vision transformers (ViTs) with millions of parameters and large datasets with practically sized images, i.e., ImageNet. The code and pretrained weights for the proposed Hierarchical Windowed Roto-reflection Equivariant ViTs (REViT-v2) are available at https://github.com/kc-ml2/revit.

Figures & tables

Model	Top-1 Accuracy (%)	Top-5 Accuracy (%)	Params.
ViT-S w/ aug	72.08	89.54	22 M
RE-ResNet	77.37	93.74	11 M
REViT-v2-T	72.58	90.88	5 M
REViT-v2-S	79.27	94.45	18 M
REViT-v2-S	80.9	95.1	47 M

Table 1: Performance Comparison on ImageNet-1K.

Attention	Params.	Forward FLOPs	Latency vs global	Peak Training Memory	Accuracy (%)
Global	97.7 K	1.69 GFLOPs	1 $\times$	429.7 MB	98.23
Global w/downsampling	97.7 K	333.6 MFLOPs	0.75 $\times$	50.98 MB	98.28
Windowed	102.9 K	80.1 MFLOPs	0.61 $\times$	26.47 MB	98.26

Table 2: Comparison with global G-CSA on Rotated MNIST.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Group	Lifting Stem Equivariance Err. $\downarrow$	Pre-Class Equivariance Err. $\downarrow$
$p4$	0.000142 $\pm$ 0.000017	0.00218 $\pm$ 0.000471
$p4m$	0.001316 $\pm$ 0.000614	0.00008 $\pm$ 0.000034
$p8$	0.001474 $\pm$ 0.000366	0.000102 $\pm$ 0.000013

Appendix

Table 3: Equivariance error analysis

Explore similar work

REViT: Roto-reflection Equivariant Convolutional Vision Transformer
Jun 24, 2026Sheir A. Zaheer, Alexander C. Holston, Chan Y. ParkSelf-Supervised Vision Transformers Rotation Equivariance
1KC Machine Learning Lab, Seoul, Rep. of Korea.
A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$
Jun 26, 2026Tīkun Ông, Georg BökmanSelf-Supervised Vision Transformers Vision Transformer
Independent Researcher · University of Amsterdam
Quick ViTs: Speeding up Vision Transformers through Equivariance
May 21, 2025David Nordström, Johan Edstedt, Fredrik Kahl +1Self-Supervised Vision Transformers Vision Transformer
Chalmers University of Technology · Linköping University · University of Amsterdam

Jun 24, 2026, cs.CVJ/K move · Enter open · S save

REViT: Roto-reflection Equivariant Convolutional Vision Transformer

Sheir A. Zaheer, Alexander C. Holston, Chan Y. Park

1KC Machine Learning Lab, Seoul, Rep. of Korea.

Self-Supervised Vision Transformers Rotation Equivariance Convolutional Block Attention Module Equivariant Neural Networks Equivariance

Open paper AskPDF arXiv

In this paper, we propose a discrete roto-reflection group equivariant vision transformer with convolutional attention. Roto-reflection equivariant networks preserve the rotational, flip and positional symmetry in feature maps, making them useful for tasks where orientation of the inputs is relevant to the model outputs. In image classification and object detection, most of the studies on roto-reflection equivariant models have focused on using convolutional neural networks rather than vision transformers. In this paper, we examine the challenges involved in achieving equivariance in vision transformers, and we propose a simpler way to implement a discretized roto-reflection group equivariant vision transformer. The experimental results demonstrate that our approach outperforms the existing approaches for developing discrete roto-reflection group equivariant neural networks for image classification.

Abstract

Explore similar work

REViT: Roto-reflection Equivariant Convolutional Vision Transformer

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of O(2)\mathrm{O}(2)O(2)

Quick ViTs: Speeding up Vision Transformers through Equivariance

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$