ButterflyViT: 354$\times$ Expert Compression for Edge Vision Transformers

In current Mixture of Experts (MoE) architectures, linear memory scaling is present, the memory grows as the number of experts increases. $N$ independent expert weight matrices require $\mathcal{O}(N \cdot d^2)$ memory which exceeds the memory budget of edge devices. Current compression methods like quantization, pruning, and low-rank factorization reduce constant factors, but the scaling bottleneck is still unresolved. We introduce ButterflyMoE, a method which treats experts not as independent matrices but as geometric reorientations of a shared quantized substrate. Diversity amongst the experts arises from viewing different angles of the shared capacity and not from redundant storage. Learned rotations are applied to a shared ternary prototype. With this, each expert yields $\mathcal{O}(d^2 + N \cdot d \log d)$ memory-reducing per-expert cost from $\mathcal{O}(d^2)$ to $\mathcal{O}(d \log d)$. The key insight is that training these rotations with quantization reduces activation outliers and stabilizes extreme low-bit training where other static methods collapse. Across language modeling benchmarks, ButterflyMoE achieves 80$\times$ memory reduction at 8 experts with a highly favorable memory-accuracy tradeoff.At this 80x compression ButterflyMoE outperforms an equal memory dense baseline, showing that orbital parameterization extracts fundamentally more utility per byte. When scaled up to 256 experts, ButterflyMoE asymptotically compresses the memory by 150 $\times$. ButterflyMoE reduces the constant factor of linear scaling with compression ratio growing with the expert count.

The Vision Transformer (ViT) achieves remarkable accuracy across visual tasks but remains computationally expensive for edge deployment. This paper presents MicroViTv2, a lightweight Vision Transformer optimized for real-device efficiency. Built upon the original MicroViT, the proposed model is designed based on reparameterized design, specifically Reparameterized Patch Embedding (RepEmbed) and Reparameterized Depth-Wise convolution mixer (RepDW) for faster inference, and introduces the Single Depth-Wise Transposed Attention (SDTA) to capture long-range dependencies with minimal redundancy. Despite slightly higher FLOPs, MicroViTv2 improves accuracy up to 0.5% compared to its predecessor and surpassing MobileViTv2, EdgeNeXt, and EfficientViT while maintaining fast inference and high energy efficiency on Jetson AGX Orin. Experiments on ImageNet-1K and COCO demonstrate that hardware-aware design and structural re-parameterization are key to achieving high accuracy and low energy consumption, validating the need to evaluate efficiency beyond FLOPs. Code is available at https://github.com/novendrastywn/MicroViT.

We present CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights. The pipeline (i) scores feed-forward network (FFN) layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis and per-cluster residual experts using k-means clustering; (iii) trains lightweight routers supervised by cluster labels; and (iv) dispatches tokens through pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of the dense model's accuracy (86.70 +/- 0.02% versus 86.73%). Extensive ablation studies reveal a consistent empirical finding: the shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). Accuracy also remains stable across different SVD ranks, expert counts (2-8), calibration set sizes (50-500), and random seeds. This behavior generalizes across five ViT backbones (DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Small, and ViT-Base), covering models from 5.7M to 86.6M parameters, with accuracy differences <= 0.10 percentage points from their dense counterparts. On a GTX 960 GPU, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense implementation. A dispatch microbenchmark further shows that routing is an order of magnitude more memory-bound than expert matrix multiplications, identifying fused dispatch kernels as a promising direction for future optimization.