SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

Zhuohang Jiang; Xu Yuan; Haohao Qu; Shanru Lin; Kanglong Liu; Wenqi Fan; Qing Li

cs.CVFeb 26, 2026

SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

Authors: Zhuohang Jiang, Xu Yuan, Haohao Qu, Shanru Lin, Kanglong Liu, Wenqi Fan, Qing Li

Organizations: The Hong Kong Polytechnic University

Abstract

The rapid advancement of AI-powered smart glasses-one of the hottest wearable devices-has unlocked new frontiers for multimodal interaction, with Visual Question Answering (VQA) over external knowledge sources emerging as a core application. Existing Vision Language Models (VLMs) adapted to smart glasses are typically trained and evaluated on traditional multimodal datasets; however, these datasets lack the variety and realism needed to reflect smart glasses usage scenarios and diverge from their specific challenges, where accurately identifying the object of interest must precede any external knowledge retrieval. To bridge this gap, we introduce SUPER- GLASSES, the first comprehensive VQA benchmark built on real-world data entirely collected by smart glasses devices. SUPERGLASSES comprises 2,422 egocentric image-question pairs spanning 14 image domains and 8 query categories, enriched with full search trajectories and reasoning annotations. We evaluate 26 representative VLMs on this benchmark, revealing significant performance gaps. To address the limitations of existing models, we further propose the SUPERLENS, a multimodal smart glasses agent that enables retrieval-augmented answer generation by integrating automatic object detection, query decoupling, and multimodal web search. SUPERLENS achieves state-of-the-art performance, outperforming GPT-4o by 2.19%, underscoring the need for task-specific solutions in smart glasses VQA. Our dataset is publicly available at https://huggingface.co/datasets/xandery/SuperGlasses.

Figures & tables

Figure 1: SuperGlasses contains 14 image domains and 8 query categories, with 2,422 question–answer pairs. Each example includes an egocentric image captured by smart glasses, a manually annotated question–answer pair, and associated multimodal search logs.

Figure 2: The A 4 Data Collection Pipeline , consisting of four stages: Acquirement , Annotatation , Assessment , and Analysis .

Figure 4: Overview of the proposed SuperLens , composed of a Demand-Adaptive Answerer and a Dual-Lens Knowledge Retriever. All modules marked in blue are powered by VLMs, while modules in green are external tools.

Dataset	Language	Hops	Image Source	Location	MM Search	Obj. Detect.	Search Log	#Domain	#Category
WebQA [ 7 ]	en	multi-hop	Web	✗	✗	✗	✗	-	2
InfoSeek [ 9 ]	en	single-hop	Web	✗	✗	✗	✗	9	-
MRAG-Bench [ 19 ]	en	single-hop	Web	✗	✓	✗	✗	9	3
LIVEVQA [ 17 ]	en	multi-hop	News	✗	✓	✗	✗	12	-
MMSearch [ 50 ]	en	multi-hop	Web	✗	✓	✗	✗	14	-
Dyn-VQA [ 26 ]	en/zh	multi-hop	Web	✗	✓	✗	✗	9	3

Table 2: Comparison with VQA and multimodal search benchmarks across multiple dimensions.

Vision Language Model	RAG Type	Difficulty			Reasoning Steps		Information-Seeking				All	Rank
Vision Language Model	RAG Type	Easy	Medium	Hard	1-hop	$\geq$ 2-hop	None	Image	Text	Both	All	Rank
Ray-Ban Meta Smart Glasses
LLaMA-3.2-11B [ 34 ]	Direct Answer	27.16	17.44	14.53	26.35	17.55	33.14	13.37	19.84	14.57	23.53	15
LLaMA-3.2-90B [ 34 ]	Direct Answer	34.99	25.93	22.91	34.55	25.29	39.9	23.39	28.78	22.61	31.59	9
RayNeo Smart Glasses
Qwen2.5-VL-3B [ 4 ]	Direct Answer	30.83	20.68	18.53	29.03	21.68	36.37	17.22	21.63	12.06	25.56	12

Table 3: SuperGlasses Leaderboard: Results of our SuperLens and 26 Vision Language Models across 3 Dimensions.

Setting	All	Decline
SuperLens	44.10	–
A. No Retrieval
w/o Search	32.82	$\downarrow$ 11.28
B. Mandatory Retrieval
Image Only (w/o Object Dectector)	19.45	$\downarrow$ 24.65
Text Only (w/o Query Decoupler)	16.39	$\downarrow$ 27.71

Table 4: Ablation on key designs of our SuperLens .

Figure 8: Distribution of error types.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Question/answer length distribution and question prefixes of SuperGlasses .

Panel of Figure 10 — Figure 10: The common topics in questions and answers of SuperGlasses .

Panel of Figure 11 — Figure 11: The prompt used for domain recognition.

Panel of Figure 12 — Figure 12: The prompt used for direct answer generation.

Model	Plant	Public Service	Food	Shopping	Translation	Transport	Culture	Navigation	Animal	Education
Meta-Rayben Smart Glasses
LLaMA-3.2-11B	22.49	24.58	18.82	28.27	25.11	17.24	25.13	30.46	18.35	30.00
LLaMA-3.2-90B	34.32	33.00	26.20	34.60	34.04	26.11	33.51	35.06	25.32	37.50
RayNeo Smart Glasses
Qwen2.5-VL-3B	26.04	28.96	20.30	29.96	25.11	20.20	25.13	33.91	10.76	40.00
Qwen2.5-VL-7B	31.07	34.01	22.51	35.44	39.57	25.12	26.70	48.28	25.95	45.00

Appendix

Table 5: Detailed scores on top-10 domains of SuperGlasses .

Panel of Figure 13 — Figure 13: The prompt used for answer evaluation.

Model	Aggregation	Comparison	Factual Knowledge	Multi-hop	Reasoning	Simple Recognition	Spatial Reasoning	Temporal Understanding
Meta-Rayben Smart Glasses
LLaMA-3.2-11B	19.42	21.23	18.09	16.79	32.38	24.53	33.94	21.54
LLaMA-3.2-90B	26.62	32.88	26.50	23.80	38.95	34.08	38.53	32.31
RayNeo Smart Glasses
Qwen2.5-VL-3B	22.30	28.08	17.52	17.52	31.03	30.43	31.19	24.62
Qwen2.5-VL-7B	30.22	34.93	21.94	23.94	41.48	39.33	34.86	30.77

Appendix

Table 6: Detailed scores on query categories of SuperGlasses .

Model	Serach Type	Dynamism			Reasoning Hops
Model	Serach Type	Static	Slow-Changing	Fast-Changing	1-hop	2-hop	3-hop	4-hop
Meta-Rayben Smart Glasses
LLaMA-3.2-11B	Direct Answer	25.21	13.66	13.92	26.35	23.53	16.90	4.35
LLaMA-3.2-90B	Direct Answer	32.80	22.36	26.29	34.55	29.41	24.80	17.39
RayNeo Smart Glasses
Qwen2.5-VL-3B	Direct Answer	42.59	25.77	18.81	29.02	21.85	17.69	13.04

Appendix

Table 7: Detailed scores on dynamism and hop dimensions of SuperGlasses.

Evaluator	Llama-3.1-8B	Gemma-3-27B	Qwen2.5-32B	Hybrid LLM	Human
Qwen2.5-VL-7B	41.37	39.05	32.82	29.36	32.81
Gemini 2.5 Pro	52.18	52.10	43.02	34.64	43.75
SuperLens ‡	54.75	51.11	44.10	38.03	45.31

Appendix

Table 8: Performance of Different Evaluators on Assessment.

Language	en (#1369)	zh (#880)	fr (#152)	jp (#63)
Qwen2.5-VL-7B	35.06	32.16	30.26	36.51
Gemini 2.5 Pro	44.63	46.82	37.50	50.79
SuperLens ‡	47.11	44.09	47.37	47.62

Appendix

Table 9: Different Language Performance Comparison.

Figure 14: Success case of using image search.

Figure 15: Success case of using text search.

Figure 16: Failure case caused by incorrect search tool selection.

Figure 17: Failure case caused by incorrect search query generation.

Panel of Figure 18 — Figure 18: Case Study in SuperGlasses : “Campbell’s Soup Can”

Panel of Figure 19 — Figure 19: Hop 1 and Hop 2 of “Campbell’s Soup Can”

Panel of Figure 20 — Figure 20: Hop 3 and Hop 4 of “Campbell’s Soup Can”

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses
Oct 8, 2026Zhimin Li, Pan Wang, Jingxian Chen +3Efficient VLM Inference VLM Inference
Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida
AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM Evaluation Agent Benchmarks
University of Maryland, College Park · Princeton University
IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models
Feb 18, 2026Parsa Madinei, Srijita Karmakar, Russell Cohen Hoffing +2Visual Question Answering Large Vision-Language Models
Department of Computer Science, UC Santa Barbara · Department of Psychological & Brain Sciences, UC Santa Barbara · DEVCOM Army Research Laboratory

Oct 8, 2026J/K move · Enter open · S save

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

Zhimin Li, Pan Wang, Jingxian Chen, Yuantao Tang+2

Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida

Efficient VLM Inference VLM Inference Privacy Protection in VLMs Vision-Language Models

Open paper AskPDF arXiv

Smart glasses are a promising platform for vision-language assistants, but private split inference remains expensive: protecting visual representations before transmission does not remove the cost of on-device vision encoding, wide protected tokens, or hardware-unaware execution. We present SAGE (Smart-glasses-Aware Guarded Encoding), a hardware-aware framework for efficient protected VLM inference on smart glasses. SAGE keeps raw images and unprotected visual tokens on the trusted device, reduces visual input cost, executes shape-specialized INT8 graphs, and compresses protected representations into compact 256-dimensional INT8 packets before transmission. Unlike efficiency methods optimized only for nominal model size, SAGE selects its operating point using downstream answer quality together with measured latency, memory, and thermal behavior on real Mentra Live glasses. On a matched Qwen workload, SAGE reduces glasses inference latency by 39.18%, device-sensor temperature rise by 21.91%, and protected packet size by 96.56%, while retaining 85.62% of the protected reference's relaxed utility and comparable prompted containment. Direct glasses-to-phone tests deliver all 192 packets and reduce median packet-plus-acknowledgment time from 71.3 to 10.0 ms. The compact protected interface also transfers across four additional VLM families. These results show that private wearable VLM inference must be optimized jointly across representation protection, communication, and real-device execution rather than through model compression alone.

Abstract

Visual Question Answering

Explore similar work

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models