No Frames, Just Frequencies: Bandwidth-Efficient Deployment-Time Adaptation for Smart Glasses

Francesco Corti; Moritz Becker; Olga Saukh

cs.CVOct 21, 2026

No Frames, Just Frequencies: Bandwidth-Efficient Deployment-Time Adaptation for Smart Glasses

Authors: Francesco Corti, Moritz Becker, Olga Saukh

Organizations: Graz University of Technology, Graz, Austria · Graz University of Technology, Complexity Science Hub, Graz, Vienna, Austria

Abstract

Deep neural networks deployed on smart glasses must adapt to changes in the distribution of incoming data while adhering to strict privacy and device constraints. We present No Frames, Just Frequencies, a bandwidth-efficient source-assisted adaptation framework that characterizes distribution shifts using compact frequency descriptors computed on-device. Rather than transmitting camera frames, the glasses compute and send frequency descriptors to a host device that contains a replica of the model. The host device maps the received frequency descriptor to a distribution shift operator that synthesizes out-of-distribution data and adapts a model replica using a pre-defined adaptation protocol. We evaluate the framework on mixtures of ImageNet-C corruptions using convolution-based (ConvNeXtV2-Femto), self-attention-based (DeiT-Tiny), and hybrid (TinyViT-5M) vision models, on person classification (Visual Wake Words), and on object detection (MS-COCO). Our results reveal that adaptation driven solely by the transmitted descriptors recovers $93-95\%$ of the supervised oracle adaptation gain on all three architectures, $82\%$ on Visual Wake Words and $94$ -- $98\%$ on MS-COCO. Notably, adaptation quality degrades gracefully as the descriptor resolution decreases from $224$ to $28$ pixels, consistently across architectures, enabling bandwidth reductions from $213\times$ to $784\times$ across the frequency descriptors.

Figures & tables

Figure 1. System overview of the proposed bandwidth-efficient TTA framework. An edge device ( i.e. , Smart Glasses) computes compact frequency descriptors (RAPSD, DCT, or DWT statistics) from incoming camera frames on-device. Only these descriptors are transmitted to the host device, keeping the per-frame uplink within a low-bandwidth budget instead of streaming raw frames. The host device maps the received descriptor to a synthetic corruption distribution through a sequential pipeline of training corruptions. The host device generates synthetic corrupted data batches from locally stored in-distribution images, and adapts a model replica using a pre-defined adaptation protocol ( Lee et al., 2023 ; Wang et al., 2021a ) . Finally, the updated parameter subset is transmitted back to the edge device.

Panel of Figure 2 — Figure 2. Per-architecture accuracy across four input resolutions ( $28$ , $56$ , $112$ , $224$ ), reported for each frequency descriptor (RAPSD, DCT and DWT). NFJF is compared against three resolution-independent references: No Adapt and Oracle - E baselines and the Oracle - CE empirical upper bound. Bars report the mean over the $63$ test DAGs of ImageNet-C; error bars indicate $\pm\sigma$ . Oracle - E accuracy markedly improves on NoAdapt only for ConvNeXtV2-Femto and destabilizes on DeiT-Tiny and TinyViT-5M, confirming the fragility of unsupervised entropy-based adaptation under distribution shift. NFJF accuracy improves on No Adapt across all three architectures for resolutions $>28$ and it surpasses Oracle - E at all resolutions on DeiT-Tiny and TinyViT-5M. For ConvNeXtV2-Femto, NFJF mostly surpasses Oracle - E at $112$ and $224$ input resolutions.

Panel of Figure 3 — Figure 3. Descriptor visualizations under a sample DAG (brightness $\rightarrow$ shot noise $\rightarrow$ contrast, each at severity 2; Section 5.1 ), normalized with clean-data statistics. Brightness amplifies the DC component across all descriptors and resolutions; shot noise amplifies high frequencies, an effect that diminishes at lower resolutions since the reduced-scale JPEG decode acts as a low-pass filter. Contrast reduces the mid-frequency bands left free by the other two corruptions.

Stage	Role	Pool $\mathcal{C}_{s}$	$\|\mathcal{C}_{s}\|$
$S_{1}$	Weather	{brightness}	1
$S_{2}$	Blur	{defocus, gaussian} blur	2
$S_{3}$	Noise	{gaussian, shot, speckle, impulse} noise	4
$S_{4}$	Digital	{saturate, pixelate, contrast}	3

Table 1. Four-stage ISP corruption DAG. Each DAG is restricted to one corruption per node drawn from the disjoint set

\mathcal{C}_{s}

, a subset of ImageNet-C ( Hendrycks and Dietterich, 2019 ) corruptions following ( Drenkow et al., 2026 ) . Each sample composes one corruption from each stage in the high-order degradation sequence following ( Wang et al., 2021b ) .

Panel of Figure 4 — Figure 4. Adaptation accuracy for three architecture paradigms with increasing input resolutions across $G_{test}$ . Left to right: $28$ , $112$ , $224$ input resolutions. The frequency descriptors DCT, DWT, and RAPSD (Section 4.1 ) are compared against the No Adapt and the Oracle baselines. The test graphs are sorted along the x-axis in ascending order of their mean NoAdapt accuracy across all architectures. As the input resolution increases the adaptation gap of NFJF against supervised cross-entropy Oracle adaptation narrows, with fewer $G_{test}$ shifts falling below No Adapt .

	SSDLite320-MNv3-L				YOLOX-Tiny				MobileNetV1-0.25, VWW
No Adapt	$7.4{\pm}5.1$				$8.8{\pm}7.1$				$65.3{\pm}4.2$
Oracle	$+4.7{\pm}1.4$ (CE + Reg.)				$+8.3{\pm}2.2$ (CE + Reg.)				$+9.1{\pm}3.2$ (CE); $-2.6{\pm}4.2$ (E)
Input Resolution	40	80	160	320	52	104	208	416	24	48	96
RAPSD	$+1.6{\pm}1.9$	$+2.0{\pm}1.6$	$+2.6{\pm}1.8$	$+3.7{\pm}1.4$	$+3.3{\pm}3.1$	$+4.8{\pm}2.5$	$+5.4{\pm}2.8$	$+6.8{\pm}2.2$	$+4.3{\pm}3.4$	$+5.8{\pm}3.5$	$+5.8{\pm}3.7$
DCT	$\mathbf{+2.1{\pm}1.5}$	$+2.8{\pm}1.6$	$+3.4{\pm}1.7$	$+4.4{\pm}1.5$	$+4.5{\pm}2.1$	$\mathbf{+6.1{\pm}2.9}$	$+6.9{\pm}2.5$	$+8.0{\pm}2.4$	$+5.8{\pm}3.6$	$+6.5{\pm}3.2$	$\mathbf{+7.5{\pm}3.2}$
DWT	$+1.6{\pm}1.8$	$\mathbf{+2.9{\pm}1.8}$	$\mathbf{+3.9{\pm}1.8}$	$\mathbf{+4.4{\pm}1.5}$	$\mathbf{+4.6{\pm}2.8}$	$+5.8{\pm}2.4$	$\mathbf{+7.0{\pm}2.6}$	$\mathbf{+8.1{\pm}2.3}$	$\mathbf{+6.4{\pm}3.2}$	$\mathbf{+7.1{\pm}3.4}$	$+7.5{\pm}3.8$

Table 2. Framework generality across person classification and object detection. Entries report the mean (box mAP on MS-COCO; accuracy [%] on Visual Wake Words) per-DAG paired gain (

\pm\sigma

) over the

63

test DAGs. The No Adapt row reports the absolute performance under shift, the common reference of every adaptation gain. The best per-column adaptation gain is reported in bold. The trends observed in Fig. 2 persist: every adaptation gain is positive and approaches the supervised Oracle as the resolution increases, whereas Oracle - E (VWW) collapses below No Adapt .

Panel of Figure 5 — Figure 5. Descriptors ablation at $224$ input resolution: the NFJF adaptation loop re-run with the frequency descriptor replaced by the model-based alternatives. ActStats denotes the first layer and block activation statistics and FeatFinal the pooled pre-logits embedding. Same layout as Fig. 2 . ActStats attains the highest adaptation accuracy, improving on DCT by $0.67$ pp on average and approaching Oracle - CE within $0.71$ pp. This adaptation gain costs on average $2.3\times$ the DCT payload (a fixed $708$ B versus architecture-dependent $768$ – $3072$ B). Computing ActStats requires executing the deployed model’s first layer and block, $\approx 100\times$ the number of DCT FLOPs ( $\sim 150$ - $180$ versus $1.6$ MFLOPs), whereas the frequency descriptors need no model execution.

Domain Adaptation

Domain adaptation transfers machine-learning models across changing data distributions, often using unlabeled or limited target-domain examples.

See all 252 papers

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses
Oct 8, 2026Zhimin Li, Pan Wang, Jingxian Chen +3Efficient VLM Inference VLM Inference
Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida
ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization
Mar 15, 2026Ronghao Zhang, Shuaicheng Niu, Qi Deng +3Test-Time Adaptation Zeroth-Order Optimization
South China University of Technology, Guangzhou, 510000, China. · Nanyang Technological University, 639798, Singapore. · Xi’an Jiaotong University, Xi’an, 710049, China. +1
OpenGlass: Ultra-Low-Power On-Device AI Eyewear with Event-based Vision
Jun 5, 2026Pietro Bonazzi, Julian Moosmann, Ahmet Celik +2Event-Based Vision Wearable Sensing
Department of Information Technology and Electrical Engineering, ETH Zürich, 8092 Zürich, Switzerland

Oct 8, 2026J/K move · Enter open · S save

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

Zhimin Li, Pan Wang, Jingxian Chen, Yuantao Tang+2

Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida

Efficient VLM Inference VLM Inference Privacy Protection in VLMs Vision-Language Models

Open paper AskPDF arXiv

Smart glasses are a promising platform for vision-language assistants, but private split inference remains expensive: protecting visual representations before transmission does not remove the cost of on-device vision encoding, wide protected tokens, or hardware-unaware execution. We present SAGE (Smart-glasses-Aware Guarded Encoding), a hardware-aware framework for efficient protected VLM inference on smart glasses. SAGE keeps raw images and unprotected visual tokens on the trusted device, reduces visual input cost, executes shape-specialized INT8 graphs, and compresses protected representations into compact 256-dimensional INT8 packets before transmission. Unlike efficiency methods optimized only for nominal model size, SAGE selects its operating point using downstream answer quality together with measured latency, memory, and thermal behavior on real Mentra Live glasses. On a matched Qwen workload, SAGE reduces glasses inference latency by 39.18%, device-sensor temperature rise by 21.91%, and protected packet size by 96.56%, while retaining 85.62% of the protected reference's relaxed utility and comparable prompted containment. Direct glasses-to-phone tests deliver all 192 packets and reduce median packet-plus-acknowledgment time from 71.3 to 10.0 ms. The compact protected interface also transfers across four additional VLM families. These results show that private wearable VLM inference must be optimized jointly across representation protection, communication, and real-device execution rather than through model compression alone.

Abstract

Domain Adaptation

Explore similar work

SAGE: Efficient Protected Vision-Language Inference on Smart Glasses

ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization

OpenGlass: Ultra-Low-Power On-Device AI Eyewear with Event-based Vision