SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

Zhimin Li; Pan Wang; Jingxian Chen; Yuantao Tang; Qian Lou; Jingtong Hu

cs.CROct 8, 2026

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

Authors: Zhimin Li, Pan Wang, Jingxian Chen, Yuantao Tang, Qian Lou, Jingtong Hu

Organizations: Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida

Abstract

Smart glasses are a promising platform for vision-language assistants, but private split inference remains expensive: protecting visual representations before transmission does not remove the cost of on-device vision encoding, wide protected tokens, or hardware-unaware execution. We present SAGE (Smart-glasses-Aware Guarded Encoding), a hardware-aware framework for efficient protected VLM inference on smart glasses. SAGE keeps raw images and unprotected visual tokens on the trusted device, reduces visual input cost, executes shape-specialized INT8 graphs, and compresses protected representations into compact 256-dimensional INT8 packets before transmission. Unlike efficiency methods optimized only for nominal model size, SAGE selects its operating point using downstream answer quality together with measured latency, memory, and thermal behavior on real Mentra Live glasses. On a matched Qwen workload, SAGE reduces glasses inference latency by 39.18%, device-sensor temperature rise by 21.91%, and protected packet size by 96.56%, while retaining 85.62% of the protected reference's relaxed utility and comparable prompted containment. Direct glasses-to-phone tests deliver all 192 packets and reduce median packet-plus-acknowledgment time from 71.3 to 10.0 ms. The compact protected interface also transfers across four additional VLM families. These results show that private wearable VLM inference must be optimized jointly across representation protection, communication, and real-device execution rather than through model compression alone.

Figures & tables

Figure 1: Smart-glasses motivation and protected-inference paths. Left: The camera view contains task-relevant road cues and a private license plate. Center: Previous on-device protection transforms tokens before transmission but retains a large visual input and wide protected tokens, increasing glasses-side computation and transmission burden. Right: SAGE retains the full 32-layer vision tower and the same protection transform, reduces the pixel budget, and sends a compact 256-dimensional INT8 packet per token, lowering measured glasses-side latency and sensor-temperature rise.

Figure 2: SAGE overview. (a) The trusted glasses execute visual encoding, resampling, the frozen TGAP protection transform ( Li et al., 2026 ) , and latent encoding. Only the protected INT8 packet and reconstruction metadata cross the trust boundary; downstream dequantization and decoding restore the LLM input. (b) Quality evaluation and glasses-side measurements select a fixed configuration, separately from per-request execution. Downstream QA runs on a host; the host phone is also used for the packet-delivery experiment in Appendix A.4 .

	Answer utility $\uparrow$		Prompted privacy $\downarrow$			Packet $\downarrow$
System	U.EM $\uparrow$	U.Rel $\uparrow$	P.Acc $\downarrow$	P.Cont $\downarrow$	P.WR $\downarrow$	KB $\downarrow$
Raw-HR (unprotected)	49.70 $\pm$ 0.89	59.32 $\pm$ 1.01	48.23 $\pm$ 1.08	36.67 $\pm$ 0.34	17.66 $\pm$ 0.43	229.642
TGAP-HR	50.08 $\pm$ 2.23	60.83 $\pm$ 1.60	39.30 $\pm$ 6.96	11.00 $\pm$ 2.02	18.08 $\pm$ 3.31	229.642
DistillVLM	17.72 $\pm$ 1.38	23.18 $\pm$ 2.02	8.78 $\pm$ 3.84	6.34 $\pm$ 1.21	5.21 $\pm$ 2.20	7.871
Switch-KD	16.42 $\pm$ 1.79	21.72 $\pm$ 3.08	10.39 $\pm$ 1.71	9.00 $\pm$ 0.40	4.84 $\pm$ 1.09	7.871
UPop	30.36 $\pm$ 1.53	38.30 $\pm$ 2.48	25.98 $\pm$ 7.14	8.29 $\pm$ 1.20	10.60 $\pm$ 3.01	7.871

Table 1: Qwen answer quality and prompted privacy over three 617-image splits (mean

\pm

sample standard deviation from three random seeds). Raw-HR is the unprotected high-resolution reference; TGAP-HR is the cited prior protected system ( Li et al., 2026 ) . The four adapted methods use feature or logit distillation, block pruning, or token selection and merging. Packet sizes coincide where the 256D INT8 codec and token count are shared; VisionZip shortens the sequence. TGAP-HR sends FP16 protected tokens.

	Footprint		Startup	On-device execution			Software $\uparrow$	Index $\uparrow$
System	Params (M)	Model (MB)	Load (s)	Latency (s)	HWM (MiB)	$\Delta T$ ( ∘ C)	$\bar{Q}$	$\mathcal{R}$
TGAP-HR	668.95	683.3	8.695	15.967	766.1	7.99	0.5415	1.00
DistillVLM	20.95	23.3	0.636	0.500	80.4	4.12	0.2171	0.64
Switch-KD	20.95	23.3	0.639	0.470	80.2	4.82	0.1977	0.58
UPop	590.67	601.3	7.631	8.280	661.7	4.48	0.3514	1.04
VisionZip	669.48	681.0	8.539	9.456	740.7	5.74	0.4357	1.28

Table 2: Qwen glasses-side cost for six methods measured on the same 32 paired grids, weighted by their frequency in the 617-image split. Model size averages the distinct grid-specialized graphs; HWM is weighted process high-water memory.

\bar{Q}

is the three-seed software utility–containment summary;

\mathcal{R}

combines it with device costs using Eq. 5 .

Figure 3: Qwen depth and precision ablation from a separate hardware run. Left: Glasses latency versus sensor-temperature rise. Right: Latency, serialized model size, process high-water memory (HWM), sensor rise, and accounted packet size relative to TGAP-HR (100%; lower is better). Measurements weight 32 grid pairs by their frequency in 617 images and are separate from Table 2 .

Backbone	U.Rel TGAP-HR $\rightarrow$ SAGE $\uparrow$	P.Cont TGAP-HR $\rightarrow$ SAGE $\downarrow$	$\Delta$ P.WR (pp) $\downarrow$	Payload $\downarrow$
MiniCPM-V	74.45 $\pm$ 1.98 $\rightarrow$ 63.21 $\pm$ 2.71	3.63 $\pm$ 2.99 $\rightarrow$ 3.59 $\pm$ 3.10	$-0.09$	96.87%
Phi-3.5-V	58.29 $\pm$ 3.28 $\rightarrow$ 55.81 $\pm$ 0.57	6.83 $\pm$ 1.82 $\rightarrow$ 5.44 $\pm$ 1.59	$-3.02$	95.83%
InternVL2.5-2B	53.43 $\pm$ 1.51 $\rightarrow$ 49.76 $\pm$ 0.99	2.52 $\pm$ 2.07 $\rightarrow$ 3.09 $\pm$ 2.55	$+0.45$	93.75%
GLM-4.1V-9B	51.81 $\pm$ 3.19 $\rightarrow$ 37.01 $\pm$ 2.45	4.57 $\pm$ 1.36 $\rightarrow$ 4.76 $\pm$ 1.34	$-2.80$	98.34%

Table 3: TGAP-HR to SAGE transfer on four non-Qwen VLM families (three-seed mean

\pm

sample SD).

\Delta

P.WR is SAGE minus TGAP-HR in percentage points; payload reduction compares FP16 protected tokens with INT8 codec packets, both including a 12-byte header.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

System	Params (M)	Model (MB)	Load (s)	Inference (s)	HWM (MiB)	$\Delta T$ ( ∘ C)
TGAP-HR	668.95	683.3	6.431	11.711	765.6	13.01
SAGE	669.48	680.9	6.158	6.944	739.8	10.59
SAGE-Depth	590.67	601.3	4.619	6.175	661.8	9.86
SAGE-INT4	590.67	378.7	3.500	11.442	463.8	12.71

Appendix

Table 4: Earlier four-configuration Qwen ablation run, separate from the six-method matched measurement in Table 2 . Compare the depth and precision variants within this table only. Model size is the mean serialized size across grid-specialized graphs; HWM is weighted process high-water memory.

Figure 4: Qwen utility (higher is better) and prompted containment (lower is better) on 617 images per split across three matched splits. Filled circles and whiskers show the mean and sample standard deviation over three random seeds; adjacent columns give their values. These are split-level variations, not confidence intervals. Raw-HR is unprotected; TGAP-HR is the cited protected reference. The four comparison methods use the common protected interface.

Figure 5: A shared visual-answer example (evaluation image 1itxxzl ). Both protected interfaces preserve the object answer across the five readers. Table cells show the saved response’s first line; any subsequent generated text is omitted from this display, not from evaluation.

Figure 6: Direct protected-packet delivery from Mentra Live to Galaxy S25+. (a) Packet bytes weighted by the 617-image grid frequencies. (b) Request-level glasses-clock time from TCP connect to the phone’s checksum acknowledgment. Dots are individual transfers; diamonds are medians and horizontal segments span the 10th–90th percentiles. Three alternating-order repetitions for each of 32 grid pairs yield 192/192 CRC-verified deliveries.

System	Measured stage	P10	Median	P90
TGAP-HR	TCP connect	4.25	5.12	8.73
	Glasses send	0.44	0.62	0.81
	Phone read + CRC	45.18	61.83	78.22
	Connect to ACK	55.94	71.28	92.28
SAGE	TCP connect	4.00	5.25	8.60
	Glasses send	0.23	0.35	0.48

Appendix

Table 5: Packet-delivery stages over 96 transfers per system (ms). Glasses-clock measurements cover TCP connection, sending, and the full connect-to-ACK interval; phone read and CRC use the phone’s clock and are therefore reported separately. All entries are request-level percentiles.

Group	TGAP-HR packet (KB)	TGAP-HR ACK	SAGE ACK	SAGE wins
Smallest 8 pairs	172.0–208.9	62.3	9.6	24/24
Next 8 pairs	213.0–221.2	69.8	10.2	24/24
Next 8 pairs	225.3–245.8	78.0	9.9	24/24
Largest 8 pairs	245.8–262.2	84.7	10.2	24/24

Appendix

Table 6: Matched transfer results across packet-size groups. We sort the 32 grid pairs by TGAP-HR packet bytes and place eight pairs in each group; each pair has three alternating-order transfers per system. Packet ranges use decimal KB. ACK entries are within-group medians (ms), and wins count paired requests for which SAGE receives the ACK first.

Method	Qwen adaptation	Training budget
DistillVLM	Compact visual student with aligned-feature, relation-attention, and token-contrastive transfer	Six epochs on all 1,973 fit records
Switch-KD	Compact visual student with visual-switch and bidirectional logit-difference distillation	Two epochs on all 1,973 fit records
UPop	Progressive vision-block gate search, followed by physical 28-block reconstruction	Two search epochs on 512 fit and 128 selection records; three retraining epochs on fit and selection partitions
VisionZip	Final-attention dominant-token selection and contextual merging (65% and 5% ratios)	Training-free token selection

Appendix

Table 7: Implementation and training of the adapted Qwen comparisons. “Fit” and “selection” refer to the partitions defined in Section 4.1 .

Figure 7: Measured hardware shape sensitivity. Top: Mentra Live latency, process high-water memory, and post-request peak sensor across 32 grid pairs per Qwen system; means are weighted by evaluation-image frequency, whiskers span repeated grid areas, and marker area denotes frequency. The fixed ONNX graphs use four CPU threads and batch one under the cooled sporadic-request protocol. Bottom: Direct glasses-to- Galaxy Wi-Fi packet-plus-ACK and phone read-plus-CRC times for 192 verified transfers; diamonds are per-system medians. These are measured packet costs, not phone LLM inference or electrical energy.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses
Feb 26, 2026Zhuohang Jiang, Xu Yuan, Haohao Qu +4Visual Question Answering Knowledge-Based VQA
The Hong Kong Polytechnic University
Defending Wearable VLMs Against Private Attribute Inference
Aug 26, 2026Zhimin Li, Pan Wang, Jingxian Chen +4Privacy Protection in VLMs Privacy-Utility Tradeoff
Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · College of Engineering, Michigan State University +1
GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction
Oct 20, 2026Trong-Thang Pham, Anh Nguyen, Hien Van Nguyen +1Egocentric Video QA Efficient VLM Inference

Feb 26, 2026J/K move · Enter open · S save

SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

Zhuohang Jiang, Xu Yuan, Haohao Qu, Shanru Lin+3

The Hong Kong Polytechnic University

Visual Question Answering Knowledge-Based VQA VLM Evaluation Tool-Using Vision-Language Agents

Open paper AskPDF arXiv

The rapid advancement of AI-powered smart glasses-one of the hottest wearable devices-has unlocked new frontiers for multimodal interaction, with Visual Question Answering (VQA) over external knowledge sources emerging as a core application. Existing Vision Language Models (VLMs) adapted to smart glasses are typically trained and evaluated on traditional multimodal datasets; however, these datasets lack the variety and realism needed to reflect smart glasses usage scenarios and diverge from their specific challenges, where accurately identifying the object of interest must precede any external knowledge retrieval. To bridge this gap, we introduce SUPER- GLASSES, the first comprehensive VQA benchmark built on real-world data entirely collected by smart glasses devices. SUPERGLASSES comprises 2,422 egocentric image-question pairs spanning 14 image domains and 8 query categories, enriched with full search trajectories and reasoning annotations. We evaluate 26 representative VLMs on this benchmark, revealing significant performance gaps. To address the limitations of existing models, we further propose the SUPERLENS, a multimodal smart glasses agent that enables retrieval-augmented answer generation by integrating automatic object detection, query decoupling, and multimodal web search. SUPERLENS achieves state-of-the-art performance, outperforming GPT-4o by 2.19%, underscoring the need for task-specific solutions in smart glasses VQA. Our dataset is publicly available at https://huggingface.co/datasets/xandery/SuperGlasses.

Abstract

Efficient VLM Inference

Explore similar work

SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

Defending Wearable VLMs Against Private Attribute Inference

GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction