A Computationally Efficient Multidimensional Vision Transformer

Alaa El Ichi; Khalide Jbilou

cs.LGFeb 23, 2026

A Computationally Efficient Multidimensional Vision Transformer

Authors: Alaa El Ichi, Khalide Jbilou

Organizations: Universit´e du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France.

Abstract

Vision Transformers have achieved state-of-the-art performance in a wide range of computer vision tasks, but their practical deployment is limited by high computational and memory costs. In this paper, we introduce a novel tensor-based framework for Vision Transformers built upon the Tensor Cosine Product (Cproduct). By exploiting multilinear structures inherent in image data and the orthogonality of cosine transforms, the proposed approach enables efficient attention mechanisms and structured feature representations. We develop the theoretical foundations of the tensor cosine product, analyze its algebraic properties, and integrate it into a new Cproduct-based Vision Transformer architecture (TCP-ViT). Numerical experiments on standard classification and segmentation benchmarks demonstrate that the proposed method achieves a uniform 1/C parameter reduction (where C is the number of channels) while maintaining competitive accuracy.

Figures & tables

Symbol	Description
$H_{\mathrm{img}},W_{\mathrm{img}}$	Image height and width
$C$	Number of image channels (tube dimension)
$P$	Patch side length
$N=H_{\mathrm{img}}W_{\mathrm{img}}/P^{2}$	Number of patches (tokens)
$d=P^{2}$	Spatial embedding dimension per frontal slice
$d_{\mathrm{eff}}=d\cdot C$	Effective (flattened) embedding dimension

Table 1: Summary of notation used throughout the paper.

Projection	TCP-ViT weight tensor	Std-ViT weight matrix
$\mathcal{W}_{Q,h}$	$\mathbb{R}^{d\times d_{h}\times C}$	$\mathbb{R}^{dC\times d_{h}C}$
$\mathcal{W}_{K,h}$	$\mathbb{R}^{d\times d_{h}\times C}$	$\mathbb{R}^{dC\times d_{h}C}$
$\mathcal{W}_{V,h}$	$\mathbb{R}^{d\times d_{h}\times C}$	$\mathbb{R}^{dC\times d_{h}C}$
$\mathcal{W}_{O}$	$\mathbb{R}^{d\times d\times C}$	$\mathbb{R}^{dC\times dC}$

Table 2: MHSA weight dimensions. Each projection is listed per head; the total count sums over all

H

heads.

Projection	TCP-ViT weight tensor	Std-ViT weight matrix
$\mathcal{W}_{1}$	$\mathbb{R}^{d\times d_{\mathrm{ff}}\times C}$	$\mathbb{R}^{dC\times d_{\mathrm{ff}}C}$
$\mathcal{W}_{2}$	$\mathbb{R}^{d_{\mathrm{ff}}\times d\times C}$	$\mathbb{R}^{d_{\mathrm{ff}}C\times dC}$

Table 3: FFN weight dimensions for TCP-ViT and standard ViT.

Component	TCP-ViT tensor size	TCP-ViT params	Std params	Ratio
$\mathcal{W}_{Q,h}$ ( $\times H$ )	$\mathbb{R}^{d\times d_{h}\times C}$	$d^{2}C$	$d^{2}C^{2}$	$1/C$
$\mathcal{W}_{K,h}$ ( $\times H$ )	$\mathbb{R}^{d\times d_{h}\times C}$	$d^{2}C$	$d^{2}C^{2}$	$1/C$
$\mathcal{W}_{V,h}$ ( $\times H$ )	$\mathbb{R}^{d\times d_{h}\times C}$	$d^{2}C$	$d^{2}C^{2}$	$1/C$
$\mathcal{W}_{O}$	$\mathbb{R}^{d\times d\times C}$	$d^{2}C$	$d^{2}C^{2}$	$1/C$
MHSA total	—	$4\,d^{2}\,C$	$4\,d^{2}\,C^{2}$	$\mathbf{1/C}$
$\mathcal{W}_{1}$	$\mathbb{R}^{d\times d_{\mathrm{ff}}\times C}$	$r_{\mathrm{ff}}\,d^{2}\,C$	$r_{\mathrm{ff}}\,d^{2}\,C^{2}$	$1/C$

Table 4: Per-layer parameter comparison between TCP-ViT and standard ViT. Input tensor

\mathcal{X}\in\mathbb{R}^{N\times d\times C}

, standard input

X\in\mathbb{R}^{N\times dC}

. Expansion ratio

r_{\mathrm{ff}}

H

heads,

d_{h}=d/H

d_{\mathrm{ff}}=r_{\mathrm{ff}}d

Dataset	Std-ViT (%)	TCP-ViT (%)	$\bm{\Delta}$	Std Params	TCP-ViT Params
CIFAR-10	61.8	63.6	$+1.8$	119,194	43,114
SVHN	49.8	63.8	$+14.0$	119,194	43,114
STL-10	52.5	55.9	$+3.4$	119,194	43,114
Parameter ratio				$0.362\times$ (theoretical: $0.333\times$ )

Table 5: Classification accuracy on subsampled datasets (10K train / 2K test). TCP-ViT uses

0.362\times

the parameters of Std-ViT (theoretical:

1/C=0.333\times

Dataset	Std-ViT (%)	TCP-ViT (%)	$\bm{\Delta}$	Param ratio
CIFAR-10 (full)	78.7	73.4	$-5.3$	$0.362\times$

Table 6: Classification accuracy on full CIFAR-10 (50K train / 10K test, 200 epochs). TCP-ViT retains 93.3% of performance with 36.2% of the parameters.

Component	Std-ViT	TCP-ViT	Ratio
Total	1,570,978	751,586	$0.478\times$
Backbone	1,275,072	455,680	$0.357\times$
Transformer Encoder(s)	1,188,480	402,048	$0.338\times$
Embeddings & Tokens	86,592	53,632	$0.619\times$
CNN Decoder	295,906	295,906	$1.000\times$

Table 7: Parameter breakdown for segmentation. The Transformer encoder achieves

0.338\times

, closely matching the theoretical

1/C=0.333\times

Fig. 1: Parameter distribution. (a) Total parameters: TCP-ViT uses $0.478\times$ of Std-ViT. (b) Stacked breakdown. (c) Component-level comparison.

Metric	Std-ViT	TCP-ViT	$\Delta$
mIoU (%)	87.8	86.8	$-1.0$
mDice (%)	93.5	92.9	$-0.6$
Pixel Acc. (%)	94.2	93.6	$-0.6$
IoU: Background	84.0	82.7	$-1.3$
IoU: Foreground (Pet)	91.7	90.9	$-0.8$
Dice: Background	91.3	90.5	$-0.8$

Table 8: Segmentation performance on Oxford-IIIT Pet. TCP-ViT retains 98.9% of the mIoU with

2.09\times

fewer parameters.

\Delta

denotes the absolute difference (TCP-ViT

-

Std-ViT).

Fig. 3: Normalized confusion matrices. Both models show strong diagonal dominance.

Fig. 4: Qualitative results. Columns: input, ground truth, Std-ViT, TCP-ViT. Green overlay: foreground. Per-image mIoU shown on each prediction.

Fig. 5: Error maps. Green = correct, red = error, gray = ignored. Errors are concentrated at boundaries for both models.

Fig. 6: Efficiency trade-off: parameters vs. mIoU. The dashed arrow indicates the compression direction.

Explore similar work

RAViT: Resolution-Adaptive Vision Transformer
Feb 27, 2026Martial Guidez, Stefan Duffner, Christophe GarciaEfficient ViTs Vision Transformer
INSA Lyon, CNRS, Université Claude Bernard Lyon 1, LIRIS, UMR5205, 69621 Villeurbanne, France
Multidimensional Task Learning: A Unified Tensor Framework for Computer Vision Tasks
Feb 26, 2026Alaa El Ichi, Khalide JbilouMulti-Task Learning
Université du Littoral Cote d’Opale, LMPA, 50 rue F. Buisson, 62228 Calais-Cedex, France.
JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
May 26, 2026Dongyun Zou, Zhuoyang Zhang, Junyu Chen +8Efficient Transformer Inference Vision Foundation Models
MIT · NVIDIA · University of Pennsylvania +1

Feb 27, 2026J/K move · Enter open · S save

RAViT: Resolution-Adaptive Vision Transformer

Martial Guidez, Stefan Duffner, Christophe Garcia

INSA Lyon, CNRS, Université Claude Bernard Lyon 1, LIRIS, UMR5205, 69621 Villeurbanne, France

Efficient ViTs Vision Transformer Image Classification

Open paper AskPDF arXiv

Vision transformers have recently made a breakthrough in computer vision showing excellent performance in terms of precision for numerous applications. However, their computational cost is very high compared to alternative approaches such as Convolutional Neural Networks. To address this problem, we propose a novel framework for image classification called RAViT based on a multi-branch network that operates on several copies of the same image with different resolutions to reduce the computational cost while preserving the overall accuracy. Furthermore, our framework includes an early exit mechanism that makes our model adaptive and allows to choose the appropriate trade-off between accuracy and computational cost at run-time. For example in a two-branch architecture, the original image is first resized to reduce its resolution, then a prediction is performed on it using a first transformer and the resulting prediction is reused together with the original-size image to perform a final prediction on a second transformer with less computation than a classical Vision transformer architecture. The early-exit process allows the model to make a final prediction at intermediate branches, saving even more computation. We evaluated our approach on CIFAR-10, Tiny ImageNet, and ImageNet. We obtained an equivalent accuracy to the classical Vision transformer model with only around 70% of FLOPs.

Abstract

Explore similar work

RAViT: Resolution-Adaptive Vision Transformer

Multidimensional Task Learning: A Unified Tensor Framework for Computer Vision Tasks

JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search