TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edge

Kurt M Wilson; Mohaiminul Al Nahian; Abeer Matar A. Almalky; Sadat Shahriyar; Souvik Kundu; Zhishan Guo; Abdullah Al Arafat; Adnan Siraj Rakin

cs.CRAug 24, 2026

TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edge

Authors: Kurt M Wilson, Mohaiminul Al Nahian, Abeer Matar A. Almalky, Sadat Shahriyar, Souvik Kundu, Zhishan Guo, Abdullah Al Arafat, Adnan Siraj Rakin

Organizations: North Carolina State University · Binghamton University (SUNY) · Florida International University · Intel

Abstract

Despite their remarkable success, machine learning models, particularly in vision applications, are alarmingly vulnerable to a range of security threats. One key factor in the attack landscape is the distinction between white-box and black-box threat models, as the latter poses challenges that limit attack effectiveness when access to model information is limited. As a result, using Trusted Execution Environments (TEEs) enhances security for machine learning applications by protecting model confidentiality and execution integrity, effectively shifting the execution environment from the white-box to the black-box side of the threat model spectrum. While adopting TEEs for large vision models, e.g., Vision Transformers (ViTs), is crucial for enhancing security and privacy, significant challenges related to memory constraints and increased computational latency must be addressed, especially in time-sensitive edge applications where safety and privacy are paramount. The objective of this work is to enable large vision models to be fully hosted within TEEs, achieving GPU-level inference latency for time-sensitive edge vision applications while maintaining performance. To this end, we propose TEE-X, a TEE-aware acceleration framework that introduces a sensitivity-aware modularization technique and enables vectorization in TEE inference. This design is validated on OP-TEE for Arm TrustZone, configured to optimize performance on the NVIDIA Jetson AGX Xavier for efficient edge vision applications using ViT models. The findings reveal that TEE-X delivers an effective TEE-aware acceleration framework that achieves minimal accuracy-latency trade-offs while ensuring fast and secure edge inference for vision models.

Figures & tables

Figure 1: Overview of attack types categorized according to different threat model settings.

Figure 2: Comparison of DeiT-small inference in: GPU, TEE, and accelerated using TEE-X.

Figure 3: Overview of TEE-X’ modularization steps.

Figure 4: Overview of Vectorization Step: Vectorizing

C=A\times B

Figure 5: Inference of two consecutive blocks in the TA. The input gets sent to the TA as a pointer to shared memory. Working memory and block results are kept in secure memory, except for the outputs of the last block, which gets placed into shared memory. The shared memory pointer gets passed to the host as the output, which can read the result.

Method	Model Size (MB)	Accuracy	Avg Layer Time (ms)	Avg Time (ms)	Peak Mem. (MB)
Baseline Host (GPU)	82.66	97.29	82.62	1022.35	154.78
TEE-X $(\mathcal{C}=0.5)$	3.48 (23 $\times$ )	96.48	58.24 (1.42 $\times$ )	1082.67	21.91
TEE-X $(\mathcal{C}=0.4)$	2.93 (28 $\times$ )	96.40	51.76 (1.60 $\times$ )	1006.74	18.14
TEE-X $(\mathcal{C}=0.2)$	1.75 (47 $\times$ )	94.64	48.02 (1.72 $\times$ )	957.13	10.13

Table 1: Model size, Accuracy, inference time, and memory of DeiT-Small (Original vs Inside TEE) on CIFAR-10 under different computation reduction factors

\mathcal{C}

. Accuracy (%) is reported. Peak Mem. (MB) is the peak memory utilization inside a TEE at inference time.

Method	Latency (ms)
Naive full model in TEE	1792.42
Model Partitioning 33 ; 46 ; 23 ; 49	1463.80
Operation-aware Execution 14 ; 35	3127.73
TEE-X	1082.67

Table 2: Impact of TEE-X’s modularization compared to SOTA TEE-based model partitioning. Since this is a software optimization comparison, we applied vectorization acceleration to the competition as well.

Model	Acc.(%)	TEE Ope. Support
Baseline (84.1 MB)	79.72	–
Pruning 57 (33 MB)	60.51	$\times$
2-bit Quantization 24 (6.18 MB)	71.90	$\times$
TEE-X (4.63 MB)	76.44	✓

Table 3: TEE-X and acceleration methods on DeiT-small on ImageNet.

Type	Setting	ASR (%)
Adversarial Attacks	WB 32	99.40
Adversarial Attacks	BB 64	60.00
Bit-Flip Attack	WB 40	100.0
Bit-Flip Attack	BB	0.0

Table 4: White-box (WB) and black-box (BB) attacks on DeiT-B. Attack success rate (ASR, %) is reported.

Model	Original		$\boldsymbol{\mathcal{C}=0.66}$		$\boldsymbol{\mathcal{C}=0.5}$
Model	Model Size (MB)	Acc.	Model Size (MB)	Acc.	Model Size (MB)	Acc.
DeiT-Small	84.11	79.72	4.63	76.44	3.81	75.79

Table 5: Accuracy and model size comparison (Original vs Inside TEE) on ImageNet for DeiT-Small under different computation reduction factors

\mathcal{C}

. Accuracy (%) is reported.

Model	Original		$\boldsymbol{\mathcal{C}=0.5}$		$\boldsymbol{\mathcal{C}=0.4}$		$\boldsymbol{\mathcal{C}=0.2}$
Model	Model Size (MB)	Acc.	Model Size (MB)	Acc.	Model Size (MB)	Acc.	Model Size (MB)	Acc.
DeiT-Base	327.33	97.55	12.83	97.22	10.47	97.04	5.80	95.57

Table 6: Accuracy and model size comparison (Original vs Inside TEE) on CIFAR-10 for DeiT-Base under different computation reduction factors

\mathcal{C}

. Accuracy (%) is reported.

Method	Acc.(%)	Inference Time (s)
Baseline (Naive full model TEE)	97.29	4.43
TEE-X (w/o modularization)	97.29	1.80
TEE-X (w/o vectorization)	96.48	1.77
TEE-X	96.48	1.08
Host (GPU)	97.29	1.02

Table 7: Impact of each component of TEE-X on DeiT-S on the CIFAR-10 dataset.

Method	$\boldsymbol{\mathcal{C}=0.2}$	$\boldsymbol{\mathcal{C}=0.4}$
Uniform Budget	91.27	93.55
TEE-X (Ours)	94.64	96.40

Table 8: Effect of Different Computation Budget Assignment Strategy in TEE for Different

\mathcal{C}

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 468 papers

Jun 30, 2026

FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers

Deploying Vision Transformer (ViT) models on edge platforms remains challenging due to their high computational demands and the architectural heterogeneity of modern hybrid ViT models, which incorporate both fully connected and convolutional layers. This heterogeneity leads to significant variation in tensor shapes, requiring flexible and efficient FPGA-based acceleration. In this paper, we present FlexViT, a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. Built on the SECDA-TFLite framework, FlexViT employs a hardware-software co-design approach that maps both fully connected and convolutional layers onto a unified high-throughput INT8 GEMM engine using a runtime im2col transformation. To efficiently support diverse layer configurations, we propose a dual-mode dataflow that dynamically switches between input and weight reuse by reconfiguring the compute array at runtime. We further introduce a depth-first tiling strategy that completes accumulation in a single pass, eliminating off-chip partial-sum transfers and reducing memory bandwidth requirements. We implement FlexViT on a PYNQ-Z2 FPGA and evaluate it across a representative set of ViT models. FlexViT achieves up to 2.74x speedup on accelerator-executed layers, translating into up to 1.40x end-to-end speedup compared to CPU-only execution. The code is available at: https://github.com/gicLAB/FlexViT

Hubert Dymarkowski, Xingjian Fu, Rappy Saha +2

School of Computing Science, University of Glasgow, Scotland, UK

Hybrid CNN-Transformer Architectures Edge Inference Efficient ViTs FPGA-Based Neural Network Acceleration

Ask PDF

May 11, 2026

MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers

The Vision Transformer (ViT) achieves remarkable accuracy across visual tasks but remains computationally expensive for edge deployment. This paper presents MicroViTv2, a lightweight Vision Transformer optimized for real-device efficiency. Built upon the original MicroViT, the proposed model is designed based on reparameterized design, specifically Reparameterized Patch Embedding (RepEmbed) and Reparameterized Depth-Wise convolution mixer (RepDW) for faster inference, and introduces the Single Depth-Wise Transposed Attention (SDTA) to capture long-range dependencies with minimal redundancy. Despite slightly higher FLOPs, MicroViTv2 improves accuracy up to 0.5% compared to its predecessor and surpassing MobileViTv2, EdgeNeXt, and EfficientViT while maintaining fast inference and high energy efficiency on Jetson AGX Orin. Experiments on ImageNet-1K and COCO demonstrate that hardware-aware design and structural re-parameterization are key to achieving high accuracy and low energy consumption, validating the need to evaluate efficiency beyond FLOPs. Code is available at https://github.com/novendrastywn/MicroViT.

Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu +2

Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering, University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan +1

Energy-Efficient ML Edge Inference Efficient ViTs Image Classification

Ask PDF

Feb 26, 2025

Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies

In recent years, vision transformers (ViTs) have emerged as powerful and promising techniques for computer vision tasks such as image classification, object detection, and segmentation. Unlike convolutional neural networks (CNNs), which rely on hierarchical feature extraction, ViTs treat images as sequences of patches and leverage self-attention mechanisms. However, their high computational complexity and memory demands pose significant challenges for deployment on resource-constrained edge devices. To address these limitations, extensive research has focused on model compression techniques and hardware-aware acceleration strategies. Nonetheless, a comprehensive review that systematically categorizes these techniques and their trade-offs in accuracy, efficiency, and hardware adaptability for edge deployment remains lacking. This survey bridges this gap by providing a structured analysis of model compression techniques, software tools for inference on edge, and hardware acceleration strategies for ViTs. We discuss their impact on accuracy, efficiency, and hardware adaptability, highlighting key challenges and emerging research directions to advance ViT deployment on edge platforms, including graphics processing units (GPUs), application-specific integrated circuit (ASICs), and field-programmable gate arrays (FPGAs). The goal is to inspire further research with a contemporary guide on optimizing ViTs for efficient deployment on edge devices.

Shaibal Saha, Lanyu Xu

Department of Computer Science and Engineering, Oakland University, , Rochester Hills, 48309, Michigan, USA

Efficient ViTs Neural Network Compression Edge Inference Neural Network Hardware Acceleration

Ask PDF

Abstract

Efficient Neural Network Inference

Explore similar work

FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers

MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers

Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies