MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

Anurita Das

cs.LGApr 22, 2026

MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

Authors: Anurita Das

Organizations: Genovation Technological Solutions Pvt Ltd

Abstract

Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.

Explore similar work

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8LLM Inference Efficiency Central Processing Units
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
May 7, 2026Shen Xu, Xiangwen Zhuge, Zhe Xu +3LLM Inference Efficiency Device
Profiling Lightweight Large Language Models
Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference Efficiency Profiling

Apr 7, 2025 · cs.DCJ/K move · Enter open · S save

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

Zonghang Li, Tao Li, Wenjiao Feng, Rongxing Xiao+7

MBZUAI · UESTC · City University of Hong Kong · La Trobe University

On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome this challenge, we present prima.cpp, a distributed on-device inference system that runs 30-70B LLMs on consumer home clusters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs. We introduce pipelined-ring parallelism (PRP) to overlap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading. We further propose Halda, a heterogeneity-aware scheduler that co-optimizes per-device CPU/GPU workloads and device selection under RAM/VRAM constraints. On four consumer home devices, a 70B model reaches 674 ms/token TPOT with <6% memory pressure, and a 32B model with speculative decoding achieves 26 tokens/s. Compared with llama.cpp, exo, and dllama, our proposed prima.cpp achieves 5-17 times lower TPOT, supports fine-grained model sizes from 8B to 70B, ensures broader cross-OS and quantization compatibility, and remains OOM-free, while also being Wi-Fi tolerant, privacy-preserving, and hardware-independent. The code is available at https://github.com/OpenCPIL/prima.cpp.

LLM Inference Efficiency Central Processing Units Offloading Parallel

Open paper Ask about it PDF

Abstract

Explore similar work

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices

Profiling Lightweight Large Language Models