A Generalized Optimization Engine (GOE) for Edge AI Inference Acceleration

Venkat R. Dasari; Jakob A. Adams; Vinod K. Mishra; Brian Jalaian

cs.AIAug 19, 2026

A Generalized Optimization Engine (GOE) for Edge AI Inference Acceleration

Authors: Venkat R. Dasari, Jakob A. Adams, Vinod K. Mishra, Brian Jalaian

Organizations: DEVCOM Army Research Laboratory Aberdeen Proving Ground, MD, USA · University of West Florida Pensacola, FL, USA

Abstract

Artificial intelligence (AI) models have demonstrated remarkable capabilities across various domains, yet their widespread deployment is impeded by significant computational costs, particularly on resource-constrained devices. This paper explores the theoretical underpinnings of various AI model optimization techniques, algorithms, and abstractions, discussing their potential to reduce computational complexity, memory footprint, latency, and power consumption. Furthermore, we propose a comprehensive hardware (HW) and model-agnostic generalized optimization architecture that integrates these techniques for improved efficiency. Our study underscores the critical role of such a generalized optimization system in preparing model deployment over resource-constrained heterogeneous hardware in a tactical environment. As a concrete demonstration, we show that GOE-compressed language models deploy and run on a GPU-less edge CPU, and that the choice of compression method, not merely its nominal bit-width, determines whether task accuracy survives deployment.

Figures & tables

Symbol	Meaning
$m$	Model from the repository (e.g., CNN variant, LLM family member).
$q$	Quantization bit-width / policy (e.g., 4-, 8-, 16-bit; potentially mixed-precision).
$p$	Pruning vector describing method and sparsity level(s).
$h$	Training hyperparameters (e.g., learning rate) relevant for fine-tuning/QAT.
$\rho$	Global pruning fraction (structured/unstructured).
$\delta$	Distillation indicator: $0$ = none, $1$ = apply KD/self-distillation.

Table 1: Notation and decision variables used in eqs. 3.1 , 3.2 and 3.3 .

Panel of Figure 1 — Figure 1: The GOE optimization engine: an input model is profiled by the Model Analyzer, compressed by architecture-aware operators, and compiled to a hardware target for deployment on edge devices.

Model	Accuracy (%)	Size (Mb)
ResNet 50	80.56	97.7
ViT B32	75.82	336.55

Table 2: Base Model Results

Model	(Min)	Epochs	Prune (%)	(%)	(Mb)
	Total			Pruned	Pruned
	Runtime	Fine-Tune		Accuracy	Size
ResNet 50	16	5	25	96.93	55
ResNet 50	19	5	50	96.87	24.52
ResNet 50	19	5	75	96.99	6.19
ViT B32	18	5	25	96.11	282.51

Table 3: Structured Pruning

Model	Base	Base	Quant	Quant
	Accuracy (%)	Latency	Accuracy (%)	Latency
ResNet 50	81.0	0.00139	81.0	0.00005
MobileNet v2	72.0	0.00059	72.0	0.00002
MobileNet v3 Small	68.0	0.00015	38.0	0.00001
MobileNet v3 Large	76.0	0.00041	74.0	0.00003

Table 4: Quantization with TorchAO

Model	Number of	Compression	Accuracy	Accuracy	Memory
	Parameters	Ratio	(%)	Change	Usage (Mb)
ofa-rn50	48,105,992	0	80.14	–	183.76
Subnet 0	23,514,408	2.05	85.43	+5.29	89.90
Subnet 1	20,423,600	2.36	84.19	+4.05	78.10
Subnet 2	20,597,080	2.34	84.18	+4.04	78.75
Subnet 3	23,106,688	2.08	84.00	+3.86	88.34

Table 5: Accuracy and Memory Size of Top-10 Sub-networks from ResNet 50 Search

Subnet	Quantized	Subnet	Quantized
Accuracy (%)	Accuracy (%)	Size (Mb)	Size (Mb)
78.42	78.34	93.08	85.26
77.15	77.09	71.14	63.32
78.07	78.10	79.73	73.47
76.60	76.62	82.67	76.41
77.33	77.36	77.78	71.52

Table 6: Accuracy and Memory Size Quantized Subnets - Supernet Accuracy: 80.14% and Model Size: 183.77 Mb

Figure 2: Edge-CPU deployment envelope on a GPU-less device (Intel Core Ultra 7, llama.cpp /GGUF). Proper GGUF quantization ( Q8_0 , Q4_K_M ) moves each model toward smaller size (left) and higher throughput (right) while holding accuracy near the F16 baseline; the pruning route trades accuracy for further speed, and a naive dynamic-int8 scheme of the same 8-bit width collapses into the near-chance band.

Variant	MB	tok/s	ARC-e	PIQA	HS	CSQA
F16	2365	23.5	.620	.760	.627	.620
Q8_0	1260	40.6	.613	.753	.627	.620
Q4_K_M	770	58.6	.613	.760	.600	.587
pruned+Q4	627	72.0	.440	.640	.500	.193

Table 7: Llama-3.2-1B on the edge CPU. Q8 / Q4 preserve accuracy; the pruning route trades it for speed.

Variant	MB	tok/s	ARC-e	PIQA	HS	CSQA
F16	2950	18.0	.747	.780	.667	.807
Q8_0	1570	30.5	.753	.773	.667	.807
Q4_K_M	940	43.3	.747	.747	.633	.733

Table 8: Qwen2.5-1.5B on the edge CPU. Same pattern: quantization is the accuracy-preserving route.

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 471 papers

Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment
Aug 16, 2026Subhransu Das, Jiaming Cheng, Arnav Kumar +6Edge AI Model Compression
The Ohio State University, Department of Computer Science and Engineering, Columbus, OH, USA
AIPC: Agent-Based Automation for AI Model Deployment with Qualcomm AI Runtime
Apr 16, 2026Jianhao Su, Zhanwei Wu, ShengTing Huang +1Edge ML
Qualcomm Technologies, Inc.
A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
Apr 16, 2026Nekane Fernandez, Ivan Valdes, Steven Van Vaerenbergh +2Efficient Inference Edge Inference
Nekane Fernandez, Ivan Valdes, Idoia de la Iglesia and Julen Arratibel are with the Dept. of Connected and Distributed Intelligence, Ikerlan Technology Research Centre, Arrasate, Spain · Nekane Fernandez and Steven Van Vaerenbergh are with the Dept. of Mathematics, Statistics and Computation, Universidad de Cantabria, Santander, Spain

Aug 16, 2026J/K move · Enter open · S save

Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

Subhransu Das, Jiaming Cheng, Arnav Kumar, Sadia Afrose+5

The Ohio State University, Department of Computer Science and Engineering, Columbus, OH, USA

Edge AI Model Compression On-Device Inference LLM Quantization Neural Network Pruning

Open paper AskPDF arXiv

Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation. What compresses well, however, need not deploy well. We survey dozens of recent works that report compression results on real hardware and extract practical deployment guidelines from them. Following these guidelines, we deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation. No single technique wins across tasks. For question answering, Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 at a 1% ratio. For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU. Pruning can even inflate the deployed artifact by 21-49% by breaking k-quant super-block alignment; combined with longer, less format-compliant outputs, this raises Raspberry Pi latency up to 3.4x. Compression can also manufacture the appearance of competence rather than destroy it visibly: one LoRA-recovered variant stays fully parseable and holds 71% strict BoolQ accuracy while sending 97 of 100 predictions to a single class, at 52.6% balanced accuracy. We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition, and condense them into task-specific deployment research directions. The right technique depends on the task, the model, and the hardware. Our experiment code and artifacts are open-sourced at https://github.com/Arnavvvkumar/deployment

Abstract

Efficient Neural Network Inference

Explore similar work

Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

AIPC: Agent-Based Automation for AI Model Deployment with Qualcomm AI Runtime

A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits