Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

Subhransu Das; Jiaming Cheng; Arnav Kumar; Sadia Afrose; Mingzhe Han; Michael Silagy; Shreya Palande; Brijesh Soni

cs.AIAug 16, 2026

Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

Authors: Subhransu Das, Jiaming Cheng, Arnav Kumar, Sadia Afrose, Mingzhe Han, Michael Silagy, Shreya Palande, Brijesh Soni, +1 more

Organizations: The Ohio State University, Department of Computer Science and Engineering, Columbus, OH, USA

Abstract

Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation. What compresses well, however, need not deploy well. We survey dozens of recent works that report compression results on real hardware and extract practical deployment guidelines from them. Following these guidelines, we deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation. No single technique wins across tasks. For question answering, Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 at a 1% ratio. For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU. Pruning can even inflate the deployed artifact by 21-49% by breaking k-quant super-block alignment; combined with longer, less format-compliant outputs, this raises Raspberry Pi latency up to 3.4x. Compression can also manufacture the appearance of competence rather than destroy it visibly: one LoRA-recovered variant stays fully parseable and holds 71% strict BoolQ accuracy while sending 97 of 100 predictions to a single class, at 52.6% balanced accuracy. We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition, and condense them into task-specific deployment research directions. The right technique depends on the task, the model, and the hardware. Our experiment code and artifacts are open-sourced at https://github.com/Arnavvvkumar/deployment

Figures & tables

Figure 1. Organization of the paper

Concept	Models	Edge Devices	Downstream Task	Performance Note
Pruning
On-device class-aware pruning (OCAP) ( Ma et al., 2023b )	VGG-16, ResNet-56, MobileNetV2	Jetson Nano, TX2, AGX Xavier	Classification	$>50\%$ latency reduction, with accuracy improvements reported up to 20%.
Latency-aware structured pruning ( Belhadi et al., 2025 )	MobileNetV2	Nvidia Jetson Nano	Classification	3 $\times$ model-size reduction and 2.2 $\times$ latency speed-up.
Morphological pruning (MHP/MEP) ( Crespí-Castañer et al., 2026 )	LeNet-300-100, LeNet-5	Raspberry Pi Pico, ESP32-E	Classification	Removes up to $\sim$ 98% of weights and achieves $>$ 46 $\times$ energy efficiency gain.
Mixed pruning with 8-bit quantization ( Chang et al., 2021 )	VGG16, FCN8s	Xilinx ZCU102 FPGA	Classification, segmentation	30 $\times$ compression for VGG16 and 24.5 $\times$ improvement in energy efficiency.
Channel pruning with Tucker decomposition ( Dai et al., 2022 )	VGG-16, AlexNet, ResNet-18	Samsung Edge device with Mali-T880 GPU	Classification	34.69 $\times$ parameter reduction and 8.31 $\times$ FLOPs reduction.

Table 1. Comparison of Model Compression Methods for Edge Deployment

Domain and Tasks	Experiments	Result References	Key Insights
LLMs • SQuAD • BoolQ • Natural Questions	Baseline benchmarking, DepGraph-MLP structured pruning, LoRA-based recovery, and GGUF quantization of Gemma, Qwen, and TinyLlama models. The models are evaluated on GPU, CPU, and Raspberry Pi platforms using task performance, model size, inference latency, and tokens per second.	• Fig. 2 • Fig. 3 • Fig. 4 • Fig. 5 • Table 3 • Table 4 • Table 7	• Comprehensive deployment results for LLMs under different compression techniques. • Pruning-induced forgetfulness and pruning-induced latency. • Prefill- and decode-level analysis of LLM inference. • BoolQ prior collapse under model compression.
Image Models • Semantic segmentation	Baseline benchmarking, structured pruning, quantization, and knowledge-distillation-assisted recovery of FPN, U-Net, and DeepLabV3+ models with ResNet-50 and EfficientNet-B3 backbones. The models are evaluated using parameters, MACs, model size, mIoU, accuracy, wall latency, CPU latency, and device temperature.	• Table 5 • Table 6	• Comprehensive deployment results for image models under different compression techniques. • Limitations of PyTorch’s default quantization workflow for convolution-dominated segmentation models. • Pruning-induced performance degradation and knowledge-distillation-assisted recovery. • Task performance is not monotonic with model size or parameter count.

Table 2. Summary of the empirical experiments, reported results, and key deployment-level observations.

Figure 2. Comparison of parameter count, Q4_K_M GGUF model size, and average fixed-prompt inference latency (the mean of per-sample latencies over the SQuAD, BoolQ, and NQ runs) for Gemma 3 1B IT, TinyLlama 1.1B, and Qwen3.5 0.8B on GPU, CPU, and Raspberry Pi platforms.

Figure 3. Fixed-prompt Q4_K_M baseline performance of small language models across GPU, CPU, and Raspberry Pi platforms using SQuAD F1 and BoolQ accuracy. GPU and CPU BoolQ values are strict accuracies (GPU parse rates: Gemma 78%, TinyLlama 92%, Qwen 100%; CPU: 41%, 93%, 100%); Raspberry Pi values are parsed accuracies (parse rates: 100%, 86%, and 99%, respectively).

Figure 4. Raspberry Pi predicted-token throughput for Q4_K_M baselines (0%) and surviving DepGraph-MLP structured-pruning runs. Dashes denote model–ratio cells without a surviving TPS value; Gemma has no baseline TPS because its coherent run did not capture a separate server TPS log.

Characteristic	Target Prune	LoRA Recovery	Quant. Level	GGUF Size (MB)	SQuAD F1	SQuAD EM	BoolQ Parsed	NQ F1	NQ EM	GPU Speed (s/sample)	Quality Mean
Qwen3.5 0.8B
Baseline	0	N/A	Q8_0	774.23	92.8	91	73	10.38	4	0.13	58.727
Baseline	0	N/A	Q6_K	600.57	93.35	91	72	11.08	5	0.1233	58.81
Baseline	0	N/A	Q5_K_M	551.22	93.85	92	75	10.18	5	0.1233	59.677
Baseline	0	N/A	Q4_K_M	507.85	84.08	82	73	7.69	2	0.1567	54.923
Pruned	1	No	Q6_K	696.59	76.51	65	69	8.04	4	0.1667	51.183

Table 4. Quantized Pruning and LoRA Recovery Results Across Models. BoolQ accuracy is computed over parseable responses; the collapsed pruned Gemma variants produce a parseable answer on only 1–4 of 100 prompts, so a BoolQ score of 100 reflects a single correct parseable response (strict accuracy 1%); the pruned TinyLlama variants parse on only 12–48 of 100 prompts, so their BoolQ scores overstate competence for the same reason and two of them exceed the TinyLlama baseline, which parses on 92; Table 7 illustrates the same prior-collapse pathology in the Qwen family. Quality Mean is the unweighted mean of SQuAD F1, BoolQ accuracy, and NQ F1; for the collapsed Gemma variants it is dominated by the degenerate BoolQ term and should not be read as a quality ranking. The pruned and LoRA-recovered rows were independently re-evaluated in the quantization-sweep harness rather than reused from Table 3 ; the screening and recovery campaigns used SQuAD/BoolQ generation ceilings of 12/3 tokens whereas this sweep used 48/16, so overlapping configurations are not expected to reproduce the earlier table exactly.

Model	Backbone	Full Size			Quantized			Structured Pruned (S=0.5)
Model	Backbone	Param / MAC / Size	mIoU	Mean Acc.(%)	Param / MAC / Size	mIoU	Mean Acc.(%)	Param / MAC / Size	mIoU	Mean Acc.(%)
FPN	ResNet-50	26.116 / 39.44 / 104.47	0.6162	69.48	26.116 / 39.44 / 104.27	0.6168	71.48	6.709 / 18.37 / 25.59	0.5784	64.61
FPN	EfficientNet-B3	12.476 / 16.33 / 47.19	0.6728	75.59	12.476 / 16.33 / 47.00	0.6710	75.77	6.411 / 10.40 / 24.46	0.5252	61.22
U-Net	ResNet-50	32.521 / 54.02 / 130.99	0.5612	61.91	32.521 / 54.02 / 130.88	0.5583	64.31	6.995 / 29.80 / 26.69	0.5517	62.86
U-Net	EfficientNet-B3	13.159 / 19.51 / 49.93	0.6636	74.80	13.159 / 19.51 / 49.82	0.6603	74.79	6.896 / 14.93 / 26.31	0.4467	54.76
DeepLabV3+	ResNet-50	26.678 / 46.29 / 106.71	0.6560	73.85	26.678 / 46.29 / 106.51	0.6490	73.15	7.943 / 20.78 / 30.30	0.6258	70.97

Table 5. Model comparison across Full, Quantized, and Structured Pruning (with Pruning Ratio=0.5) ( Das et al., 2026 ) . Structured pruning reduces average parameters by 61.55%, MACs by 43.14%, and storage by 61.98%, whereas default quantization leaves parameters and MACs unchanged and reduces storage by only 0.63% on average.

Model	Backbone	Pruning Ratio = 0.5			Pruning Ratio = 0.6			Pruning Ratio = 0.7			Pruning Ratio = 0.8			Pruning Ratio = 0.9
Model	Backbone	Latency (s)		Temp	Latency (s)		Temp	Latency (s)		Temp	Latency (s)		Temp	Latency (s)		Temp
		Wall	CPU	(°C)	Wall	CPU	(°C)	Wall	CPU	(°C)	Wall	CPU	(°C)	Wall	CPU	(°C)
DeepLabV3+	ResNet-50	25.9	12.4	52.1	24.2	11.9	51.6	21.5	11.14	51.6	19.5	10.6	51.6	16.7	9.9	51.1
DeepLabV3+	EfficientNet-B3	23.5	11.4	52.5	19.7	10.4	51.6	15.7	9.4	53.5	13.5	8.7	52.1	11.5	8.3	51.6
U-Net	ResNet-50	28.9	13.1	52.5	23.4	9.1	53	24.7	12.1	53	22.1	11.3	53	19.9	10.7	52.5
U-Net	EfficientNet-B3	20.3	8.1	51.6	17.5	7.5	51.6	14.7	6.3	51.1	15.8	8.6	50.1	11.8	5.9	49.1

Table 6. Deployment statistics of segmentation models across different sparsity levels ( Das et al., 2026 ) . Increasing the pruning ratio from 0.5 to 0.9 reduces mean wall latency from 23.93 s to 14.43 s, and EfficientNet-B3 backbones run faster on average than ResNet-50 backbones.

Model	Config	Parse (%)	Yes (%)	No (%)	Strict Acc.	Bal. Acc.
Qwen2.5 1.5B Instruct
	Baseline	100	75	25	75	67.9
	Pruned $3\%$	78	78	0	53	37.9
	+LoRA $1\%$	100	97	3	71	52.6
Qwen3.5 0.8B
	Baseline	100	47	53	73	78.8

Table 7. BoolQ prediction skew under compression. All runs use the Q4_K_M GGUF format on GPU over the same 100-sample subset (gold prior: 70% yes, 30% no). Strict accuracy is misled in opposite directions by the two model families, while balanced accuracy exposes that both collapse toward a constant prior.

Figure 5. A prefill/decode view of why structured pruning can slow edge inference. End-to-end latency splits into a one-off compute-bound prefill and a per-token memory-bandwidth-bound decode; because $k$ -quant fallback inflates the deployable weight bytes (508 to 612 MB), decode re-reads a larger weight set on each of the $N_{\text{gen}}$ output tokens. Measured phase changes appear in Table 9 .

Model	1%	3%	5%	7%	Spearman $\rho$
Gemma-1B	0.719	1.482	2.252	2.719	task floor
Qwen3.5	0.189	0.445	0.638	0.817	1.00 / 1.00
TinyLlama	0.107	0.261	0.383	0.491	0.80 / 0.80

Table 8. Fixed-activation pruning distortion

\mathcal{D}_{\mathrm{prune}}

computed on 100 SQuAD prompts with

\lambda_{k}=1

. Raw values are comparable only within a row; Spearman

\rho

relates each family’s four nested pruning configurations to its measured SQuAD F1 drop on GPU / CPU. Gemma is at the task floor at every ratio, so no coefficient is reported.

Model	Prompt eval	Per token	Tokens	Gen. tail	End-to-end
Gemma-1B (1%)	$+3.2$ – $3.3\%$	$+6.1$ – $6.2\%$	$+168.5\%$	$+202$ – $203\%$	$+147$ – $149\%$
Qwen3.5 (1%)	$+2.2$ – $4.2\%$	$+3.0$ – $7.0\%$	$+40.8\%$	$+54$ – $60\%$	$+29$ – $30\%$
TinyLlama (3%)	$+13.4$ – $16.0\%$	$+24.7$ – $25.7\%$	$+10.9\%$	$+40$ – $41\%$	$+36$ – $37\%$

Table 9. Measured phase changes after unadapted DG-MLP pruning, relative to the dense baseline in the quantization-sweep serving logs. Each entry spans the two recorded GPU-offloaded run sets, averaged over Q4_K_M , Q5_K_M , and Q6_K ; generated-token counts coincide across the run sets. In every family the generation tail grows several times faster than prompt evaluation.

Downstream Task	Representative Edge Use Cases	Primary Constraint	Evidence and Deployment Recommendation
SQuAD / Extractive QA	On-device WhatsApp or Telegram chatbot, voice-based question answering, educational tutor, field-manual assistant, offline maintenance guide, local document QA, disaster-response information assistant. ( Zhang et al., 2025b ) ( Luo et al., 2025 )	Latency is the common constraint for all LLM tasks; for SQuAD, answer precision and short-form output discipline are also critical.	Table 3 shows that Qwen3.5 0.8B provides the strongest unpruned SQuAD performance, achieving 84.08 F1 on GPU, 86.30 F1 on CPU, and 87.93 F1 on Raspberry Pi. However, DepGraph-MLP pruning severely reduces extractive QA quality: Qwen drops from 84.08 to 21.63 GPU SQuAD F1 at 7.03% pruning, while Raspberry Pi latency can increase up to 25.84 s/sample at 3.01% pruning, measured across campaigns whose prompt templates are not controlled. Therefore, for extractive QA, the preferred deployment choice is the unpruned or quantized Qwen model rather than an aggressively pruned variant. Table 4 further shows that Q5_K_M is the best Qwen quantization point, giving 93.85 SQuAD F1, 92 EM, and 551.22 MB GGUF size, while Q4_K_M reduces the size to 507.85 MB but lowers the quality mean.
Natural Questions / Open-domain QA	Offline search assistant, local RAG assistant, field-survey knowledge lookup, emergency-response query system, personal document assistant, agricultural advisory assistant.	Accuracy and answer reliability dominate; latency remains important, but the model must preserve factual recall and avoid answer drift.	Natural Questions is more fragile than SQuAD in our compressed LLM experiments. In Table 3 , even the best unpruned Qwen baseline reports only 7.69/2 NQ F1/EM on GPU, 8.18/2 on CPU, and 10.22/4 on Raspberry Pi. LoRA recovery is not uniformly restorative: for Qwen at 1% pruning, LoRA improves BoolQ but reduces SQuAD F1 and NQ F1; at 5% pruning, LoRA improves SQuAD and BoolQ but NQ remains degraded. Therefore, open-domain QA should use the strongest unpruned or lightly quantized baseline, preferably Qwen with Q5_K_M when memory is constrained. Pruning should be avoided unless the post-pruning model is re-evaluated directly on NQ-style queries.
BoolQ / Binary QA	Yes/no command validation, smart-home confirmation, robotic decision checks, clinical or agricultural checklist assistant, local policy-compliance assistant, FAQ triage. ( Hou et al., 2025 ) ( Raza et al., 2025 )	Latency and parseability are necessary, but balanced accuracy, prediction skew, and yes/no prior collapse must also be checked.	BoolQ accuracy alone is insufficient under compression. Table 3 shows that Qwen baseline achieves 73% BoolQ accuracy on GPU, 70% on CPU, and 66.67% parsed BoolQ accuracy on Raspberry Pi. However, compressed models may remain parseable while collapsing into degenerate yes/no priors. For example, Qwen3.5 at 5% pruning almost always predicts “no,” while another pruned model configuration can preserve a high strict score by predicting the majority “yes” class. Therefore, BoolQ deployments should not select models using strict accuracy alone. They should report parsed accuracy, parse rate, prediction skew, and balanced accuracy; LoRA recovery should also be validated per task because it can improve BoolQ while degrading SQuAD or NQ.
Chat / Free-form Generation	Local conversational agent, voice assistant, lightweight customer-support bot, tutoring assistant, text rewriting assistant, on-device command interface, privacy-preserving personal assistant ( Yang et al., 2024 ) ( Molina et al., 2024 ) ( Li et al., 2024 )	Latency, tokens per second, response length control, and output-format stability.	Figure 4 and Table 3 indicate that pruning does not guarantee faster generation. Although pruning removes MLP channels, it can make outputs longer and less format-compliant, increasing per-sample latency. For Qwen3.5, Raspberry Pi latency increases from 7.53 s/sample in the baseline to 25.84 s/sample at 3.01% pruning, measured across campaigns whose prompt templates are not controlled; the controlled CPU comparison moves in the same direction. Therefore, free-form chat deployments should prioritize quantization and decoding control over structural pruning. A practical policy is to use Q4_K_M when memory is the dominant constraint and Q5_K_M when answer quality is more important, while enforcing a maximum output length to prevent latency blow-up. This guidance extrapolates from the measured generation-latency behavior of the QA workloads; chat quality itself was not separately benchmarked.

Table 10. Use-case-specific deployment guidance for LLM downstream tasks.

Downstream Task	Representative Edge Use Cases	Primary Constraint	Evidence and Deployment Recommendation
Semantic Segmentation	Drone-based pest monitoring, crop-field surveillance, biological image segmentation, flood-scene parsing, robot navigation, field inspection, on-device ecological monitoring. ( Yang et al., 2023 )	mIoU, wall latency, CPU latency, memory footprint, and thermal stability.	Table 5 shows that the best full-size segmentation model is FPN–EfficientNet-B3, with 0.6728 mIoU, 75.59% mean accuracy, 12.476M parameters, 16.33G MACs, and 47.19 MB size. However, under structured pruning at $S=0.5$ , DeepLabV3+–ResNet-50 gives the best accuracy–compression trade-off, retaining 0.6258 mIoU and 70.97% accuracy while reducing size from 106.71 MB to 30.30 MB. Table 6 further shows that increasing pruning from 0.5 to 0.9 reduces average wall latency from 23.93 s to 14.43 s. Therefore, when accuracy dominates, the full FPN–EfficientNet-B3 model is preferred; when edge feasibility dominates, DeepLabV3+–ResNet-50 at moderate pruning is the safer choice; when latency dominates, high-pruning EfficientNet-backed models should be considered only after verifying mIoU degradation.
Image Classification / Stage Recognition	Spotted-lanternfly life-stage recognition, weed classification, leaf-disease screening, visual inspection, local camera-based monitoring, binary or multi-class field recognition. ( Zhang et al., 2026 )	Classification accuracy, memory footprint, and inference latency.	Table 5 reports mean accuracy jointly with segmentation mIoU. FPN–EfficientNet-B3 gives the highest full-size mean accuracy of 75.59%, while the quantized version gives 75.77% but with almost no real reduction in parameters or MACs. Structured pruning gives much stronger compression: DeepLabV3+–ResNet-50 reduces size from 106.71 MB to 30.30 MB while retaining 70.97% mean accuracy. For resource-constrained image-classification or post-segmentation stage-recognition tasks, structured pruning is therefore more useful than default quantization when the target is actual edge deployment. However, if the deployment device has sufficient memory and accuracy is the dominant criterion, the full FPN–EfficientNet-B3 model remains the preferred candidate. This guidance extrapolates from segmentation mean accuracy; image classification was not separately benchmarked.

Table 11. Use-case-specific deployment guidance for image-model downstream tasks.

Explore similar work

Apr 5, 2026

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse kernel overhead. Motivated by this gap between compression and acceleration, we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and VGG-16-BN). Across all settings, the ordered pipeline achieves a stronger accuracy-size-latency frontier than any single technique alone, reaching 0.99-1.42 ms CPU latency with competitive accuracy and compact checkpoints. Controlled ordering ablations with a fixed 20/40/40 epoch allocation further confirm that stage order is consequential, with the proposed ordering generally performing best among the tested permutations. Overall, our results provide a simple guideline for edge deployment: evaluate compression choices in the joint accuracy-size-latency space using measured runtime, rather than proxy metrics alone.

Longsheng Zhou, Yu Shen

Institute of Advanced Technology University of Science and Technology of China Hefei, China · Supercomputing Center University of Science and Technology of China Hefei, China

Neural Network Compression Efficient Neural Network Inference Neural Network Pruning

Ask PDF

May 2, 2026

Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI

Weight pruning is widely advocated for deploying Large Language Models on resource-constrained IoT and edge devices, yet its impact on model fairness remains poorly understood. We conduct a controlled empirical study of three instruction-tuned models (Gemma-2-9b-it, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct) across three pruning methods (Random, Magnitude, Wanda) at four sparsity levels (10-70%) on 12,148 BBQ bias benchmark items with 5 random seeds, totaling 2,368,860 inference records. Our results reveal a Smart Pruning Paradox: activation-aware pruning (Wanda) preserves perplexity nearly perfectly (just 3.5% increase at 50% sparsity for Mistral-7B), yet produces the highest bias amplification, with Stereotype Reliance Score increasing 83.7% and 47-59% of previously unbiased items developing new stereotypical behaviors at 70% sparsity. Random pruning destroys language capability entirely (perplexity exceeding

10^4

and reaching

10^8

) but produces only random-chance bias. We further show that unstructured pruning provides zero storage savings and zero inference latency reduction on real edge hardware, undermining the primary motivation for its use in IoT deployment. Of 180 dense-vs-pruned comparisons, 141 (78.3%) are significant (

p < 0.05

) with mean

|h| = 0.305

. Published quantization studies report up to 21% of responses flipping between biased and unbiased states; our pruning results show transition rates nearly three times higher (47-59%), suggesting pruning poses a categorically greater risk to alignment than quantization. These findings demonstrate that perplexity-based evaluation provides false assurance of behavioral equivalence, and that IoT deployment pipelines require bias-aware validation before deploying pruned models at the edge.

Plawan Kumar Rath, Rahul Maliakkal

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects

Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.

Hongyu Yu, Yifei Shen

Lenovo Research · University of Washington

LLM Quantization LLM Pruning LLM Compression KV-Cache Compression Efficient Language Model Inference

Ask PDF