Vision Tiny Recursion Model (ViTRM): Parameter-Efficient Image Classification via Recursive State Refinement

Ange-Clément Akazan; Abdoulaye Koroko; Verlon Roel Mbingui; Choukouriyah Arinloye; Hassan Fifen; Rose Bandolo

cs.CVMar 19, 2026

Vision Tiny Recursion Model (ViTRM): Parameter-Efficient Image Classification via Recursive State Refinement

Authors: Ange-Clément Akazan, Abdoulaye Koroko, Verlon Roel Mbingui, Choukouriyah Arinloye, Hassan Fifen, Rose Bandolo

Organizations: ΣηiGmα Research Group · AIMS RIC · SaH Analytics International

Abstract

The success of deep learning in computer vision has been driven by models of increasing scale, from deep Convolutional Neural Networks (CNN) to large Vision Transformers (ViT). While effective, these architectures are parameter-intensive and demand significant computational resources, limiting deployment in resource-constrained environments. Inspired by Tiny Recursive Models (TRM), which show that small recursive networks can solve complex reasoning tasks through iterative state refinement, we introduce the Vision Tiny Recursion Model (ViTRM): a parameter-efficient architecture that replaces the $L$ -layer ViT encoder with a single tiny $k$ -layer block ( $k{=}3$ ) applied recursively $N$ times. Despite using up to $6 \times$ and $84 \times$ fewer parameters than CNN based models and ViT respectively, ViTRM maintains competitive performance on CIFAR-10 and CIFAR-100. This demonstrates that recursive computation is a viable, parameter-efficient alternative to architectural depth in vision.

Figures & tables

Figure 1: ViTRM: Recursive Reasoning with Working Memory and Deep Supervision. Top: At each reasoning step $t$ , the model alternates between two phases sharing the same transformer weights $\theta$ . In the Refine Memory phase, the concatenation of visual tokens $x$ , answer token $y$ , and memory tokens $z$ is processed by the shared transformer $T^{\theta}$ for $M$ iterations, retaining only the updated memory $z$ . In the Update Answer phase, the concatenation of $y$ and $z$ is fed to $T^{\theta}$ to produce a new answer token $y$ . This process is repeated $T$ times and then passed to two shallow MLP heads: a classification head producing class logits $\hat{y}_{t}$ , and a halting head producing a halting probability $q_{t}$ . Inference stops when $q_{t}>\tau$ . Bottom: During training, deep supervision is applied at each of the $N$ reasoning steps. At step $n=1,\cdots,N$ , leveraging previous $z$ and $y$ (detached from gradients), the predicted output $\hat{y}_{n}$ is supervised with a combined loss $L_{n}=L^{n}_{\text{cls}}+L^{n}_{\text{halt}}$ , using stop-gradient to prevent interference between steps.

Model	#Params	BS=64	BS=128	BS=256	BS=512	BS=1024
ViTRM (ours)	3.6M	92.6	93.1	92.8	92.1	91.2
ViT-Small	16.9M	94.0	93.9	93.3	92.8	91.1
ViT-Base	85.1M	93.7	93.8	94.3	94.1	94.0
ViT-Large	302.4M	80.8	73.4	82.0	83.2	OOM
ResNet-18	11.2M	90.5	89.7	88.8	87.8	86.1
ResNet-34	21.3M	90.8	90.1	89.7	87.9	86.3

Table 1: Top-1 test accuracy (%) on CIFAR-10 across batch sizes. ViTRM achieves competitive accuracy with

\sim

3.6M parameters-4.7

\times

fewer than ViT-Small and 23

\times

fewer than ViT-Base. Best result per column in bold ; best ViTRM result underlined .

Model	#Params	BS=64	BS=128	BS=256	BS=512	BS=1024
ViTRM (ours)	3.7M	71.8	72.1	72.0	71.3	69.8
ViT-Small	16.9M	76.1	75.2	74.8	72.7	71.2
ViT-Base	85.2M	72.4	74.6	75.4	76.4	75.1
ViT-Large	302.5M	61.1	58.9	61.7	64.7	OOM
ResNet-18	11.2M	64.6	63.4	62.1	60.5	58.2
ResNet-34	21.3M	65.4	63.8	62.0	58.8	56.7

Table 2: Top-1 test accuracy (%) on CIFAR-100 across batch sizes. ViTRM maintains strong performance on the more challenging 100-class task while using a fraction of baseline parameters.

Figure 2: Ablation results for supervision depth $N_{\text{supervision}}$ on CIFAR-10.

Figure 3: Ablation results for reasoning depth $n_{\text{latent\_steps}}$ on CIFAR-10.

Explore similar work

Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
May 11, 2026Michal Byra, Pawel Olszowiec, Grzegorz Stefanski +2Vision Transformer Recurrent Transformers
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers
Jul 1, 2026Sang In Lee, Jihun ParkVision Transformer Recurrent Transformers
Chungnam National University Daejeon, South Korea
Random Recursive Models
Sep 30, 2026Jama Hussein Mohamud, Mirco RavanelliEfficient Neural Network Inference LLM Reasoning
Mila – Quebec AI Institute · Université de Montréal · Concordia University

May 11, 2026J/K move · Enter open · S save

Investigating Single-Block Recurrence in Vision Transformers for Image Recognition

Michal Byra, Pawel Olszowiec, Grzegorz Stefanski, Grzegorz Gruszczynski+1

Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

Vision Transformer Recurrent Transformers Efficient ViTs Image Classification

Open paper AskPDF arXiv

Vision Transformers (ViTs) implement depth by stacking independently parameterized blocks, but it remains unclear how much of this parameterization is necessary and how much can be replaced by recurrent reuse. We study this question with bViT, a single-block recurrent ViT that repeatedly applies the same transformer block while preserving the iterative computation of a deep model. On ImageNet-1K, bViT-B reaches 0.779 validation accuracy compared with 0.789 for ViT-B under the same training recipe and computational budget, while using 8.6M rather than 86.6M parameters. This correspondence becomes stronger with model width, while narrow recurrent models exhibit a substantial performance gap. Beyond classification, the single-block formulation provides a controlled testbed for studying how transformer computation evolves with depth, since the same heads, neurons, and weight matrices can be tracked across recurrent steps. Analyses of attention, activation patterns, and step-conditioned spectral pruning reveal temporally organized behavior and step-dependent utilization of the shared parameters. bViT also transfers competitively to downstream tasks while enabling highly parameter-efficient adaptation. Our work shows that much of the performance associated with independently parameterized ViT depth can be recovered through recurrent reuse of a single sufficiently wide transformer block.

Abstract

Explore similar work

Investigating Single-Block Recurrence in Vision Transformers for Image Recognition

Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers

Random Recursive Models