Organizations: Electrical, Computer, and Systems Engineering (ECSE) Department, Case Western Reserve University, Cleveland, OH 44106, USA · School of Computing and Informatics, University of Louisiana at Lafayette, Lafayette, LA 70503, USA · Department of Electrical and Computer Engineering (ECE), University of Southern California, Los Angeles, CA 90089, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA · Department of Electrical and Computer Engineering at the University of Wisconsin–Madison, Madison, WI 53706, USA
In-sensor computing reduces the cost of transmitting high-resolution image data by performing early-stage processing near the sensor. However, the logic chip integrated with a CMOS image sensor (CIS) is tightly constrained in compute and memory, limiting conventional deep neural network partitioning. We present OASIS, a distributed in-sensor vision framework that uses a lightweight encoder to generate compact, task-relevant representations before off-chip transmission. The encoder is trained end-to-end using task, entropy, and reconstruction objectives, while the decoder is used only during training. OASIS supports two complementary deployment paths. The first applies 4-bit quantization and Huffman coding while preserving the spatial structure required by classification and dense-prediction tasks. The second uses Sobol-based hyperdimensional computing (HDC) to transform the encoder latent into a fixed-dimensional binary hypervector for associative-memory classification. For the SwinViT-based VWW model, mapping a 3×3×8 latent to a 64-dimensional hypervector provides an additional 1.77× communication reduction with less than one percentage point of accuracy loss relative to the 128-dimensional configuration, yielding an overall 18,816× reduction compared with raw 8-bit image transmission. We implement the digital near-sensor pipeline on an AMD Xilinx Zynq UltraScale+ FPGA and characterize it using direct board-level power measurements and Vivado post-implementation analysis, together with circuit-simulated CIS models and a 7-nm ASIC projection. Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces total system energy by approximately 2×-4.5× while maintaining competitive accuracy, demonstrating a practical hardware-algorithm co-design path for communication-efficient in-sensor vision.
Figures & tables
Fig. 1: Overview of the dual-branch autoencoder-based in-sensor computing architecture. The system integrates an image sensor with an in-sensor processor that implements a lightweight encoder E . The output then is processed via two alternative paths: ① the default path, where quantization and Huffman coding compress the activations before transmission over MIPI to an off-chip task-specific processor that performs Huffman decoding, de-quantization, and task-specific inference; and ② the HDC path, where the encoder latent is instead mapped on-chip to a fixed-dimensional hypervector h and transmitted directly to an off-chip associative memory that predicts the label Y via similarity search against stored class prototypes, eliminating the need for a separate classifier network. A decoder branch, active only during training, reconstructs the input from the de-quantized encoder output to provide reconstruction supervision.
Fig. 2: Overview of an HDC model: encoding, training, and classification via similarity search.
Fig. 3: Probability distribution of the 4-bit quantized encoder output for input images from the evaluation sets of (a) the Visual Wake Words (VWW) dataset for image classification, and (b) the Stereo Hand Pose Tracking Benchmark (STB) dataset for hand tracking.
Fig. 4: Accuracy vs. Representation Dimension comparison on VWW classification task for ( Left ) ResNet and ( Right ) SwinViT based architecture, where d denotes the channel dimension of the encoder output.
Fig. 5: Left: Comparison of 2D estimation AUC and PCK curves, truncated at 30mm, against the baseline. Right: Comparison of 3D estimation AUC and PCK curves, truncated within a deviation range of 20mm to 50mm. Baseline model is an encoder-only network with d = 16, s = 12.
Model
Inp. Dim.
Enc. Out Dim.
mIoU
EyeNet
(1, 400, 640)
(32, 25, 40)
0.988
EyeNet (w/ ROI)
(1, 240, 560)
(32, 15, 35)
0.967
Ours
(1, 400, 640)
(4, 25, 40)
0.974
Ours (w/ ROI)
(1, 240, 560)
(4, 15, 35)
0.939
TABLE I: Comparison of Eye segmentation results between EyeNet and our network on OpenEDS dataset w/ and w/o ROI prediction.
Fig. 6: (a) Normalized energy consumption comparison between our approach and baseline across different vision tasks and models. (b) Comparison of bandwidth reduction with respect to the input image between our approach and P 2 M. Our autoencoder-based system achieves substantial bandwidth reductions, which enables more efficient data transmission and contributes to the overall energy savings shown in (a). We report P²M bandwidth savings only for VWW tasks since these are the only common benchmarks in [ 7 ] . For fair comparison, we use identical convolutional and pooling strides in the first layer for both ResNet and SwinViT models.
Method
Task
Res.
Tech.
Network
TOPS/W
Acc.
Senputing [ 15 ]
MNIST
28²
180nm
2-layer MLP
4.7
93.76%
SCAMP [ 8 ]
MNIST
256²
180nm
2-layer CNN
0.535
93.0%
MR-PIPA [ 14 ]
MNIST
256²
180nm
3-layer CNN
1.89
97.26%
DPCE [ 16 ]
CIFAR10
32²
-
LeNet-5
11.49
87.20%
PIPSIM [ 17 ]
CIFAR10
64²
45nm
LeNet-5
4.12
90.05%
P 2 M [ 7 ]
VWW
224²
22nm
MobileNetV2
0.4
84.3%
TABLE II: Comparison of in-sensor computing approaches with varying DNNs, tasks, and CIS technology nodes.
Encoder
Module
Latency (ms)
Energy ( μ J)
ResNet ( d =8, s =2)
Bottleneck Encoder
123.42
33817.08
Quant+Huff
0.06944
1.805
HDC
0.03242
0.729
SwinViT ( d =8, s =3)
Attention Encoder
344.48
45421.06
Quant+Huff
0.06827
1.775
HDC
0.03156
0.520
TABLE III: Hardware-measured latency and energy of the proposed encoder, Quant+Huff, and HDC modules implemented on a Xilinx Zynq UltraScale+ MPSoC FPGA.
Hardware Module
LUT
FF
BRAM
DSP
Dynamic Power (W)
ResNet Encoder
20,942
20,943
177
66
0.277
SwinViT Encoder †
10,529
13,576
257
31
0.128
Quant + Huffman
5,031
4,960
2
0
0.025
HDC
3,258
4,103
2
0
0.029
† Peak resource usage across separately implemented SwinViT blocks.
TABLE IV: FPGA resource utilization and measured on-board dynamic power
Configuration
Acc. (%)
BW Red. ( × )
Encoder dim. reduction only
88.0
2,091
+ entropy loss ( β )
87.9
2,091
+ 4-bit quantization
87.6
4,181
+ Huffman coding (default path)
87.6
10,653
+ HDC Encoding (HDC alter. path)
86.0
18,816
TABLE V: Ablation of OASIS components on VWW (SwinViT encoder). Bandwidth reduction is relative to the raw 2242×3 input.
Bit-width
Acc. (%)
Eff. bits (post-Huffman)
8
87.9
3.77
4
87.6
1.57
2
70.3
1.46
TABLE VI: Encoder output bit-width sweep on VWW (ResNet encoder).
Encoder
Dim. (HDC)
Red. ( × )
Acc. w/o RT (%)
Acc. w/ RT (%)
SwinViT
64
4.50
85.87
86.02
128
2.25
86.06
86.89
ResNet
32
4.00
89.39
90.79
64
2.00
89.78
90.81
TABLE VII: Inference accuracy and communication cost across HDC Dimension (VWW DATASET).
Deploying vision transformers (ViTs) on sensor-edge systems is limited not only by on-device compute, but also by the energy and bandwidth required to transmit high-dimensional image data from the sensor to the processor. While in-sensor and near-sensor computing reduce this cost through early feature extraction, existing methods often provide only modest compression. We observe that the frequency domain provides a naturally compact representation of visual information and can be exploited at the sensor level to reduce sensor-to-processor data movement. Building on this insight, we present FrequencyFormer, a co-designed sensor-to-processor pipeline for efficient ViT inference. FrequencyFormer includes: (1) a multi-scale DCT tokenizer that compresses a 224x224 image into compact frequency-domain tokens, achieving up to 128x reduction in off-chip data volume with modest accuracy loss; (2) a LUT-based near-sensor hardware implementation that leverages fixed DCT coefficients for multiplier-free, energy- and area-efficient tokenization; and (3) a modified MIPI-based low-power communication architecture that further reduces transfer energy. FrequencyFormer serves as a drop-in replacement for standard ViT patch embedding and remains compatible with pretrained backbones across classification, detection, and segmentation tasks. The pipeline achieves 28.8 TOPS/W, reduces communication energy by 230x, and lowers total sensor-side energy by 2.22x, demonstrating frequency-domain tokenization as a scalable foundation for in-sensor ViT deployment.
Chengwei Zhou, Ovishake Sen, Xuming Chen +5
Case Western Reserve University, USA · University of Florida, USA · New Jersey Institute of Technology, USA
The rapid growth of the satellite industry has driven a significant increase in geospatial data acquisition, highlighting a critical bottleneck: the severe disparity between the volume of collected sensor data and the limited downlink bandwidth available to ground stations. While On-Board Computing (OBC) has helped address this by pre-processing data in orbit, this article further advances the paradigm by introducing an in-sensor computing framework. We present an optimized end-to-end Earth Observation (EO) pipeline tailored for strict computational constraints by integrating TinyML techniques with the Sony IMX500 Intelligent Vision Sensor. Specifically, our approach shifts processing directly to the sensor level, offloading the computation from the primary embedded device, and effectively mitigating the downlink transmission of noisy or irrelevant data. We evaluated several efficient Convolutional Neural Networks (ConvNets), i.e., SqueezeNet, ShuffleNetV2, and MCUNetV1, on the EuroSAT dataset. Experimental results show that, despite the optimizations required for deployment on the IMX500 platform, our models maintain a competitive 96.68% accuracy while operating within its 8 MB constraints. Specifically, the models reach an average processing throughput of 17.40 FPS with a latency of 27.43 ms. Furthermore, our system profile exhibits high energy efficiency, with a low energy footprint of 14.19 mJ per inference and an efficiency rating of 42.26 GMAC/J, demonstrating its viability for in-sensor deployment.
Luigi Capogrosso, Pietro Bonazzi, Loris Hoxhaj +1
Interdisciplinary Transformation University of Austria · ETH Zurich · University of Verona
Distributed learning is widely used for training large models on large datasets by distributing parts of the model or dataset across multiple devices and aggregating the computed results for subsequent computations or parameter updates. Existing communication algorithms for distributed learning such as ring all-reduce result in heavy communication overhead between servers. Since communication in large-scale systems uses optical fibers, we propose an Optical In-Network-Computing (OptINC) architecture to offload the computation in servers onto the optical interconnects. To execute gradient averaging and quantization in the optical domain, we incorporate optical devices such as Mach-Zehnder-Interferometers (MZIs) into the interconnects. Such a de facto optical neural network (ONN) can effectively reduce the communication overhead in existing distributed training solutions. To reduce dataset complexity for training this neural network, a preprocessing algorithm implemented in the optical domain is also proposed. Hardware cost is lowered by approximating the weight matrices of the optical neural network with unitary and diagonal matrices, while the accuracy is maintained by a proposed hardware-aware training algorithm. The proposed solution was evaluated on real distributed learning tasks, including ResNet50 on CIFAR-100, and a LLaMA-based network on Wikipedia-1B. In both cases, the proposed framework can achieve comparable training accuracy to the ring all-reduce baseline, while eliminating communication overhead.
Sijie Fei, Grace Li Zhang, Bing Li +1
Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany