Efficient Universal Perception Encoder

Chenchen Zhu; Saksham Suri; Cijo Jose; Maxime Oquab; Marc Szafraniec; Wei Wen; Yunyang Xiong; Patrick Labatut

cs.CVMar 23, 2026

Efficient Universal Perception Encoder

Authors: Chenchen Zhu, Saksham Suri, Cijo Jose, Maxime Oquab, Marc Szafraniec, Wei Wen, Yunyang Xiong, Patrick Labatut,

Organizations: Meta Reality Labs · FAIR at Meta

Abstract

Running AI models on smart edge devices can unlock versatile user experiences, but presents challenges due to limited compute and the need to handle multiple tasks simultaneously. This requires a vision encoder with small size but powerful and versatile representations. We present our method, Efficient Universal Perception Encoder (EUPE), which offers both inference efficiency and universally good representations for diverse downstream tasks. We achieve this by distilling from multiple domain-expert foundation vision encoders. Unlike previous agglomerative methods that directly scale down from multiple teachers to an efficient encoder, we demonstrate the importance of first scaling up to a large proxy teacher and then scaling down from this single teacher. Experiments show that EUPE achieves on-par or better performance than individual domain experts of the same size on diverse task domains and also outperforms previous agglomerative encoders. We release the full family of EUPE models and the code to foster future research.

Figures & tables

Panel of Figure 1 — Figure 1: Applying our distillation recipe (EUPE) to ViT-B gives a well-balanced universal encoder that excels at diverse task domains compared to both ViT-B domain experts and existing agglomerative ViT-Bs. Left : Performance on benchmarks across three task domains, higher the better. IN1k-ZS and IN1k-KNN are image understanding benchmarks on ImageNet1k. TextVQA, SQA, Realworld, GQA, POPE are vision-language modeling tasks. SPair and ADE20k are dense prediction tasks. We omit the IN1k-ZS score for models without text encoder (PEspatial-B, DINOv3-ViT-B, DUNE-B) and the IN1k-KNN score for models without class token output (PEspatial-B). Right : Visualization of EUPE-ViT-B’s feature by PCA projection into RGB space.

Figure 2: Multi-stage distillation pipeline (scaling up $\rightarrow$ scaling down). In Stage 1 we distill from multiple foundation models into a heavy proxy model. For Stage 2 the distillation happens from the proxy model into the target efficient encoder. In Stage 3 we finetune from Stage 2 models at multiple resolutions. The image pyramid indicates the multi-resolution inputs.

Figure 3: Per teacher distillation flow. Snowflake symbol indicates frozen parameters and flame symbol indicates trainable parameters. 2D interpolation is applied to the patch tokens in case the student’s output and the teacher’s output are of different spatial dimensions.

Domain Experts
	Image under.		VLM OCR	VLM know.	VLM vision		VLM general		Dense prediction
Model	IN1k-ZS	IN1k-KNN	TextVQA	SQA	Realworld	POPE	GQA	MMEp	SPair	NYUv2 $\downarrow$	ADE20k
PEcore-B Bolya et al. (2025)	78.4	79.7	50.8	70.0	52.9	85.8	65.6	1375.5	25.9	0.641	37.4
PEspatial-B Bolya et al. (2025)	no cls	no cls	42.8	69.0	51.6	84.4	63.3	1279.6	42.2	0.389	45.5
SigLIP2-B Tschannen et al. (2025)	78.2	83.2	51.6	69.8	52.5	85.0	65.2	1389.5	32.8	0.512	41.6
DINOv3-ViT-B Siméoni et al. (2025)	no txt	83.0	42.7	69.3	52.6	85.7	65.9	1368.0	51.3	0.373	51.8

Table 1: Comparison with representative domain experts and agglomerative encoders across image understanding, VLM, and dense prediction benchmarks. Best results are indicated in bold . Numbers in brackets indicate the gap with the best domain expert. “no txt” means no text encoder. “no cls” means no class token output. ∗ The discrepancy with results from Sarıyıldız et al. (2025) is due to benchmarking only the encoder part without adapter head.

Task domain	Benchmark	Stage 2 only	Stage 1&2	Stage 1&3	Stage 1&2&3
Image	IN1k-ZS	79.6	79.5	80.0	79.7
Image	IN1k-KNN	84.0	84.0	84.3	84.1
VLM	TextVQA	46.8	48.3	49.5	50.4
	SQA	69.6	69.3	69.2	69.7
	Realworld	53.5	55.1	55.1	55.5
	POPE	85.3	85.3	84.6	85.9

Table 2: Ablation on the necessity of the three-stage pipeline. “Stage 2 only” means direct distillation from multiple teachers into the target efficient encoder. Best results are indicated in bold .

Task domain	Benchmark	SOTA	PEc&Dv3	PEc&Dv3&S2	PEc&Dv3&PEl
Image	IN1k-ZS	78.4	79.9	78.8	79.7
Image	IN1k-KNN	83.2	84.3	84.2	84.1
VLM	TextVQA	51.6	48.6	44.8	50.4
	SQA	70.0	69.7	70.2	69.7
	Realworld	54.3	55.1	52.9	55.5
	POPE	85.8	85.8	84.7	85.9

Table 3: Ablation on the choice of teacher foundation encoders in Stage 1. SOTA is the best per-benchmark performance among all existing vision encoders as a reference. PEc = PEcore-G. PEl = PElang-G. S2 = SigLIP2-G. Dv3 = DINOv3-H+. Best results are in bold .

Task domain	Benchmark	Dv3	PEc	PEc&Dv3	PEc&Dv3&S2	PEc&Dv3&PEl
Image	IN1k-ZS	no text	85.4	85.0	85.3	84.8
Image	IN1k-KNN	85.4	87.2	87.0	87.2	87.0
VLM	TextVQA	49.8	54.7	56.2	53.2	58.6
	SQA	69.0	72.2	70.9	70.2	70.6
	Realworld	53.9	56.9	59.24	54.2	60.4
	POPE	87.2	85.9	87.3	87.0	87.3

Table 4: Proxy model performance with different teachers sets used to train the Stage-1 proxy. Also include teachers (PEcore-G, DINOv3-H+) performance as a reference. PEc = PEcore-G. PEl = PElang-G. S2 = SigLIP2-G. Dv3 = DINOv3-H+. Best results are in bold .

Figure 4: Comparison of dense features by projecting the patch tokens using PCA into RGB space. From left to right: EUPE-ViT-B16 (Ours), DINOv3-ViT-B16, DUNE-B14, RADIOv2.5-B16, SigLIP2-B16, PEcore-B16. Best viewed in color.

Figure 5: Comparison of dense features PCA components for encoders trained with stage variants. Input is the same as the image in Fig. 4 row 1. “Stage 2 only” means direct distillation from multiple teachers into the target efficient encoder. Best viewed in color.

Model	Image Under.		VLM OCR	VLM Know.	VLM Vision		VLM General		Dense Prediction
Model	IN1k-ZS	IN1k-KNN	TextVQA	SQA	Realworld	POPE	GQA	MMEp	SPair	NYUv2 $\downarrow$	ADE20k
PEcore-T	52.9	61.7	44.3	68.6	47.4	82.2	60.9	1221.1	20.2	0.756	22.0
PEspatial-T	no cls	no cls	40.7	68.0	48.7	80.7	59.0	1128.0	18.4	0.589	28.5
EUPE-ViT-T	50.5	66.3	42.0	69.5	50.0	82.4	61.4	1258.0	37.2	0.571	36.7
PEcore-S	62.8	71.9	47.0	69.4	48.4	83.2	63.5	1339.2	28.6	0.599	33.7
PEspatial-S	no cls	no cls	42.5	67.6	49.0	82.8	62.9	1245.4	30.8	0.464	38.6

Table 5: ViT family evaluation across image understanding, VLM, and dense prediction tasks. “no cls” means no class token output. “no txt” means no text encoder.

Model	VLM OCR	VLM Know.	VLM Vision		VLM General		Dense Prediction
Model	TextVQA	SQA	Realworld	POPE	GQA	MMEp	SPair	NYUv2 $\downarrow$	ADE20k
DINOv3-ConvNext-T	41.6	69.8	44.7	81.9	60.3	1226.0	35.7	0.448	42.7
EUPE-ConvNext-T	43.7	68.8	47.9	83.4	63.0	1278.1	41.3	0.430	43.5
DINOv3-ConvNext-S	42.6	68.8	49.3	83.7	63.1	1321.5	34.7	0.432	44.8
EUPE-ConvNext-S	45.0	68.9	50.5	84.0	64.7	1284.2	40.1	0.388	46.8
DINOv3-ConvNext-B	42.7	69.1	46.6	84.4	63.7	1278.8	35.0	0.420	46.3

Table 6: ConvNext family evaluation across VLM and dense prediction tasks. We omit results on image understanding tasks as the models do not have class token output.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

		PEc&Dv3	PEc&Dv3&S2	PEc&Dv3&PEl	PEc&Dv3-7B&PEl
Task domain	Benchmark	(1.9B)	(1.9B)	(1.9B)	(7B)
Image	IN1k-ZS	85.0	85.3	84.8	85.1
Image	IN1k-KNN	87.0	87.2	87.0	87.2
VLM	TextVQA	56.2	53.2	58.6	59.7
	SQA	70.9	70.2	70.6	71.6
	Realworld	59.24	54.2	60.4	60.2

Appendix

Table 7: Proxy model performance with different teachers sets used to train the Stage-1 proxy. PEc = PEcore-G. PEl = PElang-G. S2 = SigLIP2-G. Dv3 = DINOv3-H+. Dv3-7B = DINOv3-7B. The number in brackets is the parameter size of the proxy model. Best results are in bold .

		Image under.		VLM OCR	VLM know.	VLM vision		VLM general		Dense prediction
DINOv3 size	Proxy size	IN1k-ZS	IN1k-KNN	TextVQA	SQA	Realworld	POPE	GQA	MMEp	SPair	NYUv2 $\downarrow$	ADE20k
ViT-H+	ViT-G	79.7	84.1	50.4	69.7	55.5	85.9	67.3	1374.5	51.3	0.391	52.4
ViT-7B	ViT-7B	80.2	83.9	48.5	69.8	53.9	85.3	66.6	1345.2	52.0	0.390	52.5

Appendix

Table 8: Impact of scaling up the DINOv3 teacher in Stage 1 and the proxy model in Stage 2&3. Reported performance is from the final ViT-B student after Stage 3.

	Image under.		VLM OCR	VLM know.	VLM vision		VLM general		Dense prediction
Training data	IN1k-ZS	IN1k-KNN	TextVQA	SQA	Realworld	POPE	GQA	MMEp	SPair	NYUv2 $\downarrow$	ADE20k
90% MetaCLIP + 10% IN1k	79.3	83.7	48.5	69.7	54.2	83.9	66.8	1327.8	49.0	0.393	52.6
90% LVD + 10% IN1k	79.9	84.3	48.6	69.7	55.1	85.8	66.7	1375.2	51.5	0.384	52.5

Appendix

Table 9: Impact of training on LVD-1689M with 1689M images versus MetaCLIP with 2.5B images

	Image	VLM OCR	VLM know.	VLM vision		VLM general		Dense
$\gamma$	IN1k-KNN	TextVQA	SQA	Realworld	POPE	GQA	MMEp	SPair@224	ADE20k
0.0	79.7	51.8	71.4	54.3	84.9	66.4	1362.8	23.4	28.7
1.0	80.2	50.9	69.9	55.2	84.9	66.4	1375.4	29.0	31.9
2.0	80.3	50.1	68.8	54.0	86.2	66.6	1380.9	31.3	32.9

Appendix

Table 10: Impact of varying the patch loss weight for DINOv3 in Eq. 7 . SPair@224 means the benchmark is done at

224\times 224

resolution.

Model	#Params	Res. 256	Res. 512	Res. 256	Res. 512
		Inference GFLOPs		Inference latency (ms)
ConvNext-Tiny	29M	5	20	22.4	82.4
ConvNext-Small	50M	11	46	38.5	141.9
ConvNext-Base	89M	20	81	59.3	222.7
ConvNext-Large ∗	198M	38	152	112.5	447.2
ViT-T	6M	4	17	6.8	38.3

Appendix

Table 11: Model size and inference cost comparison. We present per model the number of parameters and the cost measured by FLOPs and latency on images of size

256\times 256

and

512\times 512

. Latency is measured on iPhone 15 Pro CPU. ∗ models not included in our EUPE family but to show their incompatibility for running on edge devices.

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 224 papers

IronViT: Toward Efficient Generalist Visual Representation Learning
Sep 24, 2026Jiaxi Huang, Yueqi Hu, Xin Zhu +13Visual Representation Learning Efficient ViTs
Robotics Foundation Model Team, Xpeng Inc.
MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
Aug 18, 2026Bonan Zhang, Shiyu Dong, Quan Hung Tran +9Efficient ViTs VLM Inference
Meta · Work done while at Meta
Dual-Domain Representation Alignment: Bridging 2D and 3D Vision via Geometry-Aware Architecture Search
Mar 20, 2026Haoyu Zhang, Zhihao Yu, Rui Wang +3Neural Architecture Search Hardware-Aware NAS
Hangzhou Normal University, Hangzhou, P.R. China · Trustworthy and General AI Lab, School of Engineering, Westlake University, Hangzhou, P.R. China · Department of Data Science and Artificial Intelligence, and the Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China +1

Sep 24, 2026J/K move · Enter open · S save

IronViT: Toward Efficient Generalist Visual Representation Learning

Jiaxi Huang, Yueqi Hu, Xin Zhu, Xiaopeng Zhang+12

Robotics Foundation Model Team, Xpeng Inc.

Visual Representation Learning Efficient ViTs Knowledge Distillation

Open paper AskPDF arXiv

A generalist vision encoder must capture semantic, spatial, language-aligned, and action-relevant cues within a unified representation, yet softmax attention underlying today's most capable visual backbones becomes prohibitively expensive at high resolution. A natural attempt to address both challenges is to distill multiple specialist teachers directly into an efficient architecture. We find that directly coupling these objectives degrades representation quality, as the student must simultaneously reconcile heterogeneous capabilities and adapt them to a different token-mixing architecture. We introduce IronViT, built on a simple principle: consolidate capabilities before constraining computation. IronViT first distills complementary specialists into a softmax attention capability bridge, then progressively transfers the consolidated representation to a hybrid softmax-linear attention encoder. A purpose-built data pipeline further curates the distillation corpus for higher information density and broader domain coverage. Across recognition, retrieval, dense prediction, multimodal understanding, and robotic learning, IronViT is competitive with leading specialist and generalist vision encoders. The softmax bridge achieves the strongest aggregate performance in multimodal understanding and robotic learning among the evaluated backbones, while the hybrid encoder retains broad transfer performance with an efficiency advantage that grows with input resolution. Together, these results show that consolidating capabilities before architectural conversion can yield a generalist visual encoder without inheriting the prohibitive high-resolution cost of conventional softmax attention.

Abstract

Visual Representation Learning

Explore similar work

IronViT: Toward Efficient Generalist Visual Representation Learning

MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Dual-Domain Representation Alignment: Bridging 2D and 3D Vision via Geometry-Aware Architecture Search