Hardware-Aware Learned Representation Compression for Distributed In-Sensor Vision

Chengwei Zhou; Abu Masum; Xuming Chen; Mehran Moghadam; Sreetama Sarkar; Arnab Sanyal; Md Abdullah-Al Kaiser; M. Hassan Najafi

cs.LGSep 12, 2026

Hardware-Aware Learned Representation Compression for Distributed In-Sensor Vision

Authors: Chengwei Zhou, Abu Masum, Xuming Chen, Mehran Moghadam, Sreetama Sarkar, Arnab Sanyal, Md Abdullah-Al Kaiser, M. Hassan Najafi,

Organizations: Electrical, Computer, and Systems Engineering (ECSE) Department, Case Western Reserve University, Cleveland, OH 44106, USA · School of Computing and Informatics, University of Louisiana at Lafayette, Lafayette, LA 70503, USA · Department of Electrical and Computer Engineering (ECE), University of Southern California, Los Angeles, CA 90089, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA ·

Abstract

In-sensor computing reduces the cost of transmitting high-resolution image data by performing early-stage processing near the sensor. However, the logic chip integrated with a CMOS image sensor (CIS) is tightly constrained in compute and memory, limiting conventional deep neural network partitioning. We present OASIS, a distributed in-sensor vision framework that uses a lightweight encoder to generate compact, task-relevant representations before off-chip transmission. The encoder is trained end-to-end using task, entropy, and reconstruction objectives, while the decoder is used only during training. OASIS supports two complementary deployment paths. The first applies 4-bit quantization and Huffman coding while preserving the spatial structure required by classification and dense-prediction tasks. The second uses Sobol-based hyperdimensional computing (HDC) to transform the encoder latent into a fixed-dimensional binary hypervector for associative-memory classification. For the SwinViT-based VWW model, mapping a $3\times3\times8$ latent to a 64-dimensional hypervector provides an additional $1.77\times$ communication reduction with less than one percentage point of accuracy loss relative to the 128-dimensional configuration, yielding an overall $18{,}816\times$ reduction compared with raw 8-bit image transmission. We implement the digital near-sensor pipeline on an AMD Xilinx Zynq UltraScale+ FPGA and characterize it using direct board-level power measurements and Vivado post-implementation analysis, together with circuit-simulated CIS models and a 7-nm ASIC projection. Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces total system energy by approximately $2\times$ - $4.5\times$ while maintaining competitive accuracy, demonstrating a practical hardware-algorithm co-design path for communication-efficient in-sensor vision.

Figures & tables

Fig. 1: Overview of the dual-branch autoencoder-based in-sensor computing architecture. The system integrates an image sensor with an in-sensor processor that implements a lightweight encoder $\mathcal{E}$ . The output then is processed via two alternative paths: ① the default path, where quantization and Huffman coding compress the activations before transmission over MIPI to an off-chip task-specific processor that performs Huffman decoding, de-quantization, and task-specific inference; and ② the HDC path, where the encoder latent is instead mapped on-chip to a fixed-dimensional hypervector $h$ and transmitted directly to an off-chip associative memory that predicts the label $Y$ via similarity search against stored class prototypes, eliminating the need for a separate classifier network. A decoder branch, active only during training, reconstructs the input from the de-quantized encoder output to provide reconstruction supervision.

Fig. 2: Overview of an HDC model: encoding, training, and classification via similarity search.

Fig. 3: Probability distribution of the 4-bit quantized encoder output for input images from the evaluation sets of (a) the Visual Wake Words (VWW) dataset for image classification, and (b) the Stereo Hand Pose Tracking Benchmark (STB) dataset for hand tracking.

Fig. 4: Accuracy vs. Representation Dimension comparison on VWW classification task for ( Left ) ResNet and ( Right ) SwinViT based architecture, where $d$ denotes the channel dimension of the encoder output.

Fig. 5: Left: Comparison of 2D estimation AUC and PCK curves, truncated at 30mm, against the baseline. Right: Comparison of 3D estimation AUC and PCK curves, truncated within a deviation range of 20mm to 50mm. Baseline model is an encoder-only network with d $=$ 16, s $=$ 12.

Model	Inp. Dim.	Enc. Out Dim.	mIoU
EyeNet	(1, 400, 640)	(32, 25, 40)	0.988
EyeNet (w/ ROI)	(1, 240, 560)	(32, 15, 35)	0.967
Ours	(1, 400, 640)	(4, 25, 40)	0.974
Ours (w/ ROI)	(1, 240, 560)	(4, 15, 35)	0.939

TABLE I: Comparison of Eye segmentation results between EyeNet and our network on OpenEDS dataset w/ and w/o ROI prediction.

Fig. 6: (a) Normalized energy consumption comparison between our approach and baseline across different vision tasks and models. (b) Comparison of bandwidth reduction with respect to the input image between our approach and P 2 M. Our autoencoder-based system achieves substantial bandwidth reductions, which enables more efficient data transmission and contributes to the overall energy savings shown in (a). We report P²M bandwidth savings only for VWW tasks since these are the only common benchmarks in [ 7 ] . For fair comparison, we use identical convolutional and pooling strides in the first layer for both ResNet and SwinViT models.

Method	Task	Res.	Tech.	Network	TOPS/W	Acc.
Senputing [ 15 ]	MNIST	28²	180nm	2-layer MLP	4.7	93.76%
SCAMP [ 8 ]	MNIST	256²	180nm	2-layer CNN	0.535	93.0%
MR-PIPA [ 14 ]	MNIST	256²	180nm	3-layer CNN	1.89	97.26%
DPCE [ 16 ]	CIFAR10	32²	-	LeNet-5	11.49	87.20%
PIPSIM [ 17 ]	CIFAR10	64²	45nm	LeNet-5	4.12	90.05%
P 2 M [ 7 ]	VWW	224²	22nm	MobileNetV2	0.4	84.3%

TABLE II: Comparison of in-sensor computing approaches with varying DNNs, tasks, and CIS technology nodes.

Encoder	Module	Latency (ms)	Energy ( $\mu$ J)
ResNet ( $d$ =8, $s$ =2)	Bottleneck Encoder	123.42	33817.08
	Quant+Huff	0.06944	1.805
	HDC	0.03242	0.729
SwinViT ( $d$ =8, $s$ =3)	Attention Encoder	344.48	45421.06
	Quant+Huff	0.06827	1.775
	HDC	0.03156	0.520

TABLE III: Hardware-measured latency and energy of the proposed encoder, Quant+Huff, and HDC modules implemented on a Xilinx Zynq UltraScale+ MPSoC FPGA.

Hardware Module	LUT	FF	BRAM	DSP	Dynamic Power (W)
ResNet Encoder	20,942	20,943	177	66	0.277
SwinViT Encoder †	10,529	13,576	257	31	0.128
Quant + Huffman	5,031	4,960	2	0	0.025
HDC	3,258	4,103	2	0	0.029
† Peak resource usage across separately implemented SwinViT blocks.

TABLE IV: FPGA resource utilization and measured on-board dynamic power

Configuration	Acc. (%)	BW Red. ( $\times$ )
Encoder dim. reduction only	88.0	2,091
+ entropy loss ( $\beta$ )	87.9	2,091
+ 4-bit quantization	87.6	4,181
+ Huffman coding (default path)	87.6	10,653
+ HDC Encoding (HDC alter. path)	86.0	18,816

TABLE V: Ablation of OASIS components on VWW (SwinViT encoder). Bandwidth reduction is relative to the raw

224^{2}\times 3

input.

Bit-width	Acc. (%)	Eff. bits (post-Huffman)
8	87.9	3.77
4	87.6	1.57
2	70.3	1.46

TABLE VI: Encoder output bit-width sweep on VWW (ResNet encoder).

Encoder	Dim. (HDC)	Red. ( $\times$ )	Acc. w/o RT (%)	Acc. w/ RT (%)
SwinViT	64	4.50	85.87	86.02
SwinViT	128	2.25	86.06	86.89
ResNet	32	4.00	89.39	90.79
ResNet	64	2.00	89.78	90.81

TABLE VII: Inference accuracy and communication cost across HDC Dimension (VWW DATASET).

In-Sensor Computing

See all 10 papers

FrequencyFormer: A Co-Designed Sensor-to-Processor Pipeline for Frequency-Domain Vision Transformer Inference
Jun 17, 2026Chengwei Zhou, Ovishake Sen, Xuming Chen +5Edge Inference Efficient ViTs
Case Western Reserve University, USA · University of Florida, USA · New Jersey Institute of Technology, USA
Exploiting In-Sensor Computing for Energy-Efficient Earth Observation
May 31, 2026Luigi Capogrosso, Pietro Bonazzi, Loris Hoxhaj +1Remote Sensing Image Understanding Energy-Efficient ML
Interdisciplinary Transformation University of Austria · ETH Zurich · University of Verona
OptINC: Optical In-Network-Computing for Scalable Distributed Learning
Mar 30, 2026Sijie Fei, Grace Li Zhang, Bing Li +1Gradient Quantization Communication-Efficient Distributed Training
Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany

Jun 17, 2026J/K move · Enter open · S save

FrequencyFormer: A Co-Designed Sensor-to-Processor Pipeline for Frequency-Domain Vision Transformer Inference

Chengwei Zhou, Ovishake Sen, Xuming Chen, Rishith Paramasivam+4

Case Western Reserve University, USA · University of Florida, USA · New Jersey Institute of Technology, USA

Edge Inference Efficient ViTs In-Sensor Computing Visual Token Compression

Open paper AskPDF arXiv

Deploying vision transformers (ViTs) on sensor-edge systems is limited not only by on-device compute, but also by the energy and bandwidth required to transmit high-dimensional image data from the sensor to the processor. While in-sensor and near-sensor computing reduce this cost through early feature extraction, existing methods often provide only modest compression. We observe that the frequency domain provides a naturally compact representation of visual information and can be exploited at the sensor level to reduce sensor-to-processor data movement. Building on this insight, we present FrequencyFormer, a co-designed sensor-to-processor pipeline for efficient ViT inference. FrequencyFormer includes: (1) a multi-scale DCT tokenizer that compresses a 224x224 image into compact frequency-domain tokens, achieving up to 128x reduction in off-chip data volume with modest accuracy loss; (2) a LUT-based near-sensor hardware implementation that leverages fixed DCT coefficients for multiplier-free, energy- and area-efficient tokenization; and (3) a modified MIPI-based low-power communication architecture that further reduces transfer energy. FrequencyFormer serves as a drop-in replacement for standard ViT patch embedding and remains compatible with pretrained backbones across classification, detection, and segmentation tasks. The pipeline achieves 28.8 TOPS/W, reduces communication energy by 230x, and lowers total sensor-side energy by 2.22x, demonstrating frequency-domain tokenization as a scalable foundation for in-sensor ViT deployment.

Abstract

In-Sensor Computing

Explore similar work

FrequencyFormer: A Co-Designed Sensor-to-Processor Pipeline for Frequency-Domain Vision Transformer Inference

Exploiting In-Sensor Computing for Energy-Efficient Earth Observation

OptINC: Optical In-Network-Computing for Scalable Distributed Learning