TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings

Bibin Wilson

cs.CVFeb 24, 2026

TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings

Authors: Bibin Wilson

Organizations: Independent Researcher

Abstract

Zero-shot object detection enables recognising novel objects without task-specific training, but current approaches rely on large vision language models (VLMs) like CLIP that require hundreds of megabytes of memory - far exceeding the constraints of micro controller units (MCUs). We present TinyVLM, the first framework enabling zero-shot object detection on resource-constrained MCUs with less than 1MB of memory. Our approach introduces three key innovations: (1) a decoupled architecture that separates visual inference from text encoding, allowing precomputed class embeddings to be stored in flash memory; (2) Matryoshka distillation that trains nested embeddings at multiple dimensions (16-256), enabling flexible accuracy-memory trade-offs; and (3) quantized embedding storage that reduces class prototype memory by 4x with minimal accuracy loss. Trained on Conceptual Captions 3M (CC3M), TinyVLM achieves competitive zero-shot accuracy on COCO, Flowers102, and Food101 while requiring only 285KB of RAM and 892KB of flash memory for the deployed vision encoder. We demonstrate real-time inference at 26 FPS on STM32H7 and over 1,000 FPS on MAX78000 with its CNN accelerator, enabling practical zero-shot detection on edge devices for the first time.

Figures & tables

Panel of Figure 1 — Figure 1: TinyVLM enables zero-shot detection on MCUs. While existing VLMs require 18–350MB, TinyVLM achieves zero-shot capability within MCU memory constraints ( $<$ 1MB flash for the deployed vision encoder) through Matryoshka distillation and decoupled architecture.

Method	Params	Memory	MCU
	(M)	(MB)	Compatible
CLIP ViT-B/32 [ 26 ]	151	350	✗
CLIP ViT-L/14 [ 26 ]	428	812	✗
OpenCLIP ViT-G/14 [ 8 ]	1843	2100	✗
TinyCLIP ViT-S [ 34 ]	39	78	✗
MobileCLIP-S2 [ 31 ]	18	36	✗

Table 1: Comparison of VLM approaches. TinyVLM is the first to fit within MCU constraints.

Panel of Figure 2 — Figure 2: TinyVLM overview. (Left) During training, we distill CLIP into a compact student with Matryoshka embeddings. (Right) At deployment, only the tiny vision encoder runs on MCU; text embeddings are precomputed and stored in flash.

Panel of Figure 3 — Figure 3: Memory layout for TinyVLM on STM32H7.

Teacher models
Method	Memory	COCO	Flowers	Food	Avg.
CLIP ViT-B/32	350MB	56.4	66.1	83.7	68.7
CLIP ViT-L/14	812MB	64.8	76.2	92.1	77.7
OpenCLIP ViT-G/14	2.1GB	68.3	79.5	93.8	80.5
Efficient VLMs
TinyCLIP ViT-S	78MB	45.2	55.3	74.2	58.2

Table 2: Zero-shot classification accuracy (%) on standard benchmarks. TinyVLM achieves competitive accuracy with orders-of-magnitude smaller memory. Numbers for TinyVLM will be updated after full training.

Panel of Figure 4 — Figure 4: Relative accuracy vs. embedding dimension on COCO (compared to CLIP ViT-B/32). Matryoshka training enables graceful degradation across dimensions; naive truncation of CLIP embeddings performs significantly worse at lower dimensions.

Method	Memory	MCU	COCO	LVIS
		Deploy	AP50	AP50
OWL-ViT [ 21 ]	580MB	✗	34.2	25.6
YOLO-World-S [ 7 ]	52MB	✗	37.1	24.3
Grounding DINO [ 20 ]	890MB	✗	42.5	32.1
TinyVLM -Det (64d)	1.2MB	✓	12.4	8.6
TinyVLM -Det (128d)	1.5MB	✓	15.2	10.8

Table 3: Zero-shot detection results (AP50). TinyVLM enables zero-shot detection on MCUs for the first time.

Platform	Freq.	Flash	SRAM	Latency	Energy	FPS
	(MHz)	(KB)	(KB)	(ms)	(mJ)
STM32H7	480	892	285	38	2.1	26
MAX78000 †	100	133	6	0.86	0.016	1,160
GAP9	400	892	312	18	0.8	55
ESP32-S3	240	892	165	52	3.2	19

Table 4: MCU deployment benchmarks for TinyVLM (64-dim configuration).

$\mathcal{L}_{\text{contrastive}}$	$\mathcal{L}_{\text{emb}}$	$\mathcal{L}_{\text{mat}}$	64-dim	256-dim
✓			24.2	28.5
✓	✓		27.8	31.2
✓		✓	26.5	30.8
✓	✓	✓	29.5	33.8

Table 5: Ablation of loss components on COCO zero-shot accuracy.

Weighting	16-dim	32-dim	64-dim	256-dim
Equal	19.8	25.2	29.5	33.8
Inverse ( $1/d$ )	21.2	24.8	28.6	32.1
Log ( $\log(d_{\max}/d)$ )	20.5	25.6	29.2	33.2

Table 6: Effect of dimension weighting in Matryoshka loss.

Teacher	Teacher Acc.	Student Acc.	Retention
CLIP ViT-B/32	63.2	29.5	46.7%
CLIP ViT-L/14	75.5	32.8	43.4%
OpenCLIP ViT-G/14	80.1	34.2	42.7%

Table 7: Effect of teacher model on student accuracy (64-dim).

Precision	Memory (KB)	COCO Acc.	vs. Float32
Float32	20.0	33.8	—
Float16	10.0	33.7	-0.3%
Int8	5.0	33.4	-1.2%
Int4	2.5	32.1	-5.0%

Table 8: Effect of text embedding quantization (80 COCO classes, 64-dim).

Figure 5: Zero-shot accuracy across Matryoshka embedding dimensions. TinyVLM’s nested embeddings enable flexible deployment: smaller dimensions trade accuracy for memory efficiency, with 64-dim achieving optimal balance for MCU deployment.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-3}$
Weight decay	0.01
LR schedule	Cosine annealing
Warmup epochs	10
Total epochs	100

Appendix

Table 9: Training hyperparameters for TinyVLM .

Figure 6: Per-class zero-shot accuracy comparison between TinyVLM (256-dim) and CLIP teacher (512-dim) on COCO val. Classes are sorted by teacher accuracy.

Figure 7: t-SNE visualization of TinyVLM image embeddings (64-dim) on COCO val, colored by class (top-10 most frequent). Embeddings show meaningful cluster structure despite aggressive dimensionality reduction.

Figure 8: Zero-shot accuracy (256-dim) across evaluation datasets compared to random baselines. Numbers above bars indicate improvement factor over random chance.

Figure 9: Qualitative zero-shot predictions from TinyVLM (256-dim) on sample images from each evaluation dataset. Correct predictions shown in green, incorrect in red, with confidence scores.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 605 papers

Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression
Feb 24, 2026Bibin WilsonContinual Object Detection Object Detection
Independent Researcher
AHC: Meta-Learned Adaptive Compression for Continual Object Detection on Memory-Constrained Microcontrollers
Feb 24, 2026Bibin WilsonContinual Object Detection Object Detection
Independent Researcher
MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes
May 14, 2026Luca Bompani, Manuele Rusci, Luca Benini +2Edge Inference Efficient Object Detection
Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy. · Department of Electrical Engineering (ESAT), KU Leuven, Belgium. · Integrated Systems Laboratory (IIS), ETH Zürich, Switzerland. +1

Feb 24, 2026J/K move · Enter open · S save

Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression

Bibin Wilson

Independent Researcher

Continual Object Detection Object Detection Replay-Based Continual Learning Edge Inference On-Device Inference

Open paper AskPDF arXiv

Deploying object detection on microcontrollers (MCUs) enables intelligent edge devices but current models cannot learn new object categories after deployment. Existing continual learning methods require storing raw images far exceeding MCU memory budgets of tens of kilobytes. We present Latent Replay Detection (LRD), the first framework for continual object detection under MCU memory constraints. Our key contributions are: 1. Task-Adaptive Compression: Unlike fixed PCA, we propose learnable compression with FiLM (Feature-wise Linear Modulation) conditioning, where task specific embeddings modulate the compression to preserve discriminative features for each task's distribution; 2. Spatial-Diverse Exemplar Selection: Traditional sampling ignores spatial information critical for detection - we select exemplars maximizing bounding box diversity via farthest-point sampling in IoU space, preventing localization bias in replay; 3. MCU-Deployable System: Our latent replay stores 150 bytes per sample versus >10KB for images, enabling a 64KB buffer to hold 400+ exemplars. Experiments on CORe50 (50 classes, 5 tasks) demonstrate that LRD achieves mAP@50 on the initial task and maintains strong performance across subsequent tasks - a significant improvement over naive fine-tuning while operating within strict MCU constraints. Our task-adaptive FiLM compression and spatial diverse exemplar selection work synergistically to preserve detection capabilities. Deployed on STM32H753ZI, ESP32-S3, and MAX78000 MCUs, LRD achieves 4.9-97.5ms latency per inference within a 64KB memory budget-enabling practical continual detection on edge devices for the first time.

Abstract

Object Detection

Explore similar work

Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression

AHC: Meta-Learned Adaptive Compression for Continual Object Detection on Memory-Constrained Microcontrollers

MR2-ByteTrack: CNN and Transformer-based Video Object Detection for AI-augmented Embedded Vision Sensor Nodes