EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation

Longfei Liu; Yongjie Hou; Yang Li; Qirui Wang; Youyang Sha; Yongjun Yu; Yinzhi Wang; Peizhe Ru

cs.CVMar 19, 2026

EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation

Authors: Longfei Liu, Yongjie Hou, Yang Li, Qirui Wang, Youyang Sha, Yongjun Yu, Yinzhi Wang, Peizhe Ru,

Organizations: Intellindust AI Lab

Abstract

Deploying high-performance dense prediction models on resource-constrained edge devices remains challenging due to strict computation and memory budgets. In practice, lightweight systems for object detection, instance segmentation, and pose estimation are still dominated by CNN-based architectures such as YOLO, while compact Vision Transformers (ViTs) often struggle to achieve comparable accuracy-efficiency trade-offs, even with large-scale pretraining. We argue that this gap arises primarily from insufficient task-specific representation learning in small-scale ViTs, rather than from an inherent mismatch between ViTs and edge dense prediction. To address this issue, we introduce EdgeCrafter, a unified compact ViT framework for edge dense prediction centered on ECDet, a detection model built on a distilled compact backbone and an edge-friendly encoder-decoder design. The resulting detection-distilled representation transfers directly to instance segmentation and human pose estimation through lightweight task-specific prediction modules. Without using task annotations beyond COCO, ECDet-S achieves 51.7 box AP with fewer than 10M parameters, while ECInsSeg-X and ECPose-X reach 48.4 mask AP and 74.8 keypoint AP, respectively. As a complementary but more compute-intensive setting, Objects365 detection pretraining consistently improves performance across all scales, with the X variants reaching 59.9 box AP, 49.8 mask AP, and 75.9 keypoint AP. These results show that compact ViTs, when combined with task-specialized distillation and edge-aware design, can be a practical and competitive solution for edge dense prediction. Code is available at: https://intellindust-ai-lab.github.io/projects/EdgeCrafter

Figures & tables

Panel of Figure 1 — Figure 1: Comprehensive evaluation of EdgeCrafter. (a) Accuracy–efficiency comparison across object detection, human pose estimation, and instance segmentation on COCO ( Lin et al., 2014 ) . The top and bottom rows plot model parameters and FLOPs, respectively, against AP. Methods marked with ∗ use Objects365 pretraining ( Shao et al., 2019 ) . (b) Training dynamics and downstream AP for ECDet-T under different backbone initialization strategies. ViT-T follows the ImageNet-21K ( Ridnik et al., 2021 ) training strategy of Steiner et al. (2022) . Supervised ImageNet-21K pretraining is weaker than training from scratch for this compact model, while task-specialized distillation yields substantially stronger downstream performance.

Figure 2: Overview of the EdgeCrafter pipeline. Stage 1: A pretrained DINOv3 backbone ( Siméoni et al., 2025 ) is adapted to object detection to create a task-specialized teacher within the ECDet formulation. Stage 2: The resulting teacher distills its detection-oriented representation into compact ECViT student backbones through feature alignment on a large image collection. Stage 3: The distilled students are used to instantiate the ECDet model family at different scales (S/M/L/X), and the same distilled backbone and encoder are further reused for instance segmentation and human pose estimation with lightweight task-specific heads. The key idea is that detection serves as the representation-learning stage, while the learned backbone transfers directly to other dense prediction tasks.

Figure 3: Architecture of ECDet. A frozen detection-specialized teacher distills the compact ECViT backbone, whose convolutional stem replaces standard large-stride patch embedding. The final two transformer features are fused, converted into stride- $8$ , stride- $16$ , and stride- $32$ feature maps, refined by the hybrid encoder, and decoded into object predictions.

Teacher Arch.	COCO Pre-training	Distillation Dataset	AP (%)
Teacher capacity
DINOv3-S	✓	IN-1K + COCO	54.0
DINOv3-B	✓	IN-1K + COCO	54.3
DINOv3-L	✓	IN-1K + COCO	52.6
Teacher task adaptation
DINOv3-B	✗	IN-1K + COCO	53.5

Table 5: Unified ablation on teacher design and distillation data. All experiments use ECViT-T+ as the student. The teacher is initialized from DINOv3 ( Siméoni et al., 2025 ) , and “COCO pre-training” indicates that the teacher is first adapted as the ECDet backbone on COCO before distillation.

Table 5

Patch Embedding	Params (M)	FLOPs (G)	Latency (ms)	AP	AP S	AP M	AP L
Vanilla	19.0	82.5	7.87	53.5	33.7	58.2	72.8
ConvStem ( $d{=}1$ )	19.2	84.9	7.98	54.3	35.9	59.1	72.7
ConvStem ( $d{=}2$ )	19.2	84.9	-	53.0	33.7	57.9	72.5
ConvStem ( $d{=}3$ )	19.2	84.9	-	53.6	33.2	58.6	72.6

Table 8: Effect of patch embedding design and stem receptive field. Vanilla uses a single

16\times 16

stride-16 convolution. ConvStem follows the convolutional-stem design used in the final model and consists of four

3\times 3

stride-2 convolutions ( Xiao et al., 2021 ) . Here,

d

denotes the dilation rate of the final stem convolution, which controls the effective receptive field of the patch embedding stage. All results are reported on ECDet-M.

Fusion Method	Fused Layers	Params (M)	FLOPs (G)	Latency (ms)	AP
Mean	$L_{11}$	19.2	84.9	7.98	54.1
Mean	$L_{10\to 11}$	19.2	84.9	7.98	54.3
Mean	$L_{9\to 11}$	19.2	84.9	7.98	54.3
Mean	$L_{6\to 11}$	19.2	84.9	8.11	54.1
Concat	$L_{10\to 11}$	19.5	87.0	8.05	54.2
STA ( Huang et al., 2025a )	$L_{5}$ , $L_{7}$ , $L_{11}$	19.4	86.3	8.48	54.3

Table 9: Effect of layer fusion strategy. We compare simple mean fusion over different late-layer ranges with concatenation and the STA fusion module from DEIMv2 ( Huang et al., 2025a ) . The gray row marks the adopted default setting in ECDet-M.

Student	Teacher	AP
ECViT-T	–	46.6
	Co-DETR-Swin-S ( )	50.4
	DINOv3-S	50.7
	ECTeacher-S	51.7

Table 10: Comparison of different teacher models. We additionally evaluate Co-DETR with a Swin-S backbone ( Zong et al., 2023 ; Liu et al., 2021 ) as a strong detection teacher, while keeping the ECDet-S student architecture and overall training budget unchanged.

Method	#Epochs	Detection	Distillation	Total GPU Hours $\downarrow$
\rowfont YOLO26-M	80	$\sim$ 24	–	$\sim$ 24
YOLO11-M	500	$\sim$ 133	–	$\sim$ 133
RT-DETRv4-M	102	$\sim$ 190	–	$\sim$ 190
ECDet-M	62	$\sim$ 96	$\sim$ 34	$\sim$ 130

Table 11: Comparison of training cost on COCO. All models are trained on 4

\times

NVIDIA RTX 4090 GPUs . Gray results indicate pre-training on Objects365 ( Shao et al., 2019 ) followed by fine-tuning on COCO.

Table 10

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	ECDet-S	ECDet-M	ECDet-L	ECDet-X
Optimizer	AdamW	AdamW	AdamW	AdamW
Backbone LR	2.5e-5	2.5e-5	5e-6	2.5e-6
Base LR	5e-4	5e-4	5e-4	5e-4
LR decay	0.5	0.5	0.5	0.5
Weight Decay	1e-4	1e-4	1.25e-4	1.25e-4
Epochs (w/ + w/o Aug.)	72 + 2	60 + 2	48 + 2	48 + 2

Appendix

Table 14: Hyperparameters for ECDet.

Setting	ECPose-S	ECPose-M	ECPose-L	ECPose-X
Optimizer	AdamW	AdamW	AdamW	AdamW
Backbone LR	2.5e-5	2.5e-5	2.5e-6	2.5e-6
Base LR	5e-4	5e-4	5e-4	5e-4
Weight Decay	1e-4	1e-4	1.25e-4	1.25e-4
Epochs (w/ + w/o Aug.)	90 + 2	90 + 2	72 + 2	72 + 2
Prob ${}_{\text{mosaic}}$	0.5	0.5	0.5	0.5

Appendix

Table 15: Hyperparameters for ECPose.

Dense Prediction

Dense prediction maps images or other structured inputs to spatially aligned outputs such as labels, depth, motion, or physical fields.

See all 31 papers

MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers
May 11, 2026Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu +2Energy-Efficient ML Edge Inference
Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering, University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan +1
Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
Feb 26, 2025Shaibal Saha, Lanyu XuEfficient ViTs Neural Network Compression
Department of Computer Science and Engineering, Oakland University, , Rochester Hills, 48309, Michigan, USA
UtVAA: Ultra-tiny Vision Transformer with Affix Attention for Mobile Image Classification
Jun 2, 2026Romiyal George, Sathiyamohan Nishankar, Selvarajah Thuseethan +1Vision Transformer Edge Inference
University of Peradeniya, Sri Lanka · Charles Darwin University, Australia

May 11, 2026J/K move · Enter open · S save

MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers

Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo+1

Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering, University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan+1

Energy-Efficient ML Edge Inference Efficient ViTs Image Classification

Open paper AskPDF arXiv

The Vision Transformer (ViT) achieves remarkable accuracy across visual tasks but remains computationally expensive for edge deployment. This paper presents MicroViTv2, a lightweight Vision Transformer optimized for real-device efficiency. Built upon the original MicroViT, the proposed model is designed based on reparameterized design, specifically Reparameterized Patch Embedding (RepEmbed) and Reparameterized Depth-Wise convolution mixer (RepDW) for faster inference, and introduces the Single Depth-Wise Transposed Attention (SDTA) to capture long-range dependencies with minimal redundancy. Despite slightly higher FLOPs, MicroViTv2 improves accuracy up to 0.5% compared to its predecessor and surpassing MobileViTv2, EdgeNeXt, and EfficientViT while maintaining fast inference and high energy efficiency on Jetson AGX Orin. Experiments on ImageNet-1K and COCO demonstrate that hardware-aware design and structural re-parameterization are key to achieving high accuracy and low energy consumption, validating the need to evaluate efficiency beyond FLOPs. Code is available at https://github.com/novendrastywn/MicroViT.

Abstract

Dense Prediction

Explore similar work

MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers

Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies

UtVAA: Ultra-tiny Vision Transformer with Affix Attention for Mobile Image Classification