CAMP: Context-Aware Mapping and Placement for LLMs on 3D-IMC Arrays
Authors: Orhun Görkem, Athanasios Vasilopoulos, Georgios Pagonis, Victoria Clerico, Hadjer Benmeziane, Irem Boybat, Hsinyu Tsai, Abu Sebastian, William Andrew Simon
Organizations: IBM Research, Rüschlikon, Switzerland · IBM Research, San Jose, CA, USA
Large Language Models (LLMs) incur severe memory-access overheads due to the cost of loading large weight matrices and key-value (KV) caches. Analog In-Memory Computing (AIMC) is an emerging solution that enables computation directly on static weights stored in memory, substantially reducing energy and latency. 3D-IMC further stacks memory arrays into vertical tiers, increasing capacity but revealing a new challenge: how weights are placed in memory determines which matrix-vector multiplications (MVMs) can execute concurrently, affecting latency and throughput. The concurrency aspect of 3D-IMC weight-mapping remains underexplored in the literature, leading to sub-optimal utilization of compute resources on 3D-IMC chips. In this context we propose CAMP (Context-Aware Mapping and Placement), an inference-context-aware weight mapping algorithm for 3D-IMC-based LLM accelerators. CAMP captures the potential resource contention resulting from the architecture, targeted network, and runtime context in a conflict matrix, enabling throughput-optimized weight mapping. This mapping is refined through a profiling-driven feedback loop that efficiently prunes the search space and returns optimized mappings within practical runtimes. CAMP is an offline, software-based solution requiring no hardware changes and improving 3D-IMC compute utilization from 42.4% to 60.3%, delivering up to 2.0x higher 3D-IMC throughput and 1.74x higher end-to-end throughput over state-of-the-art mapping algorithms.
Figures & tables
Figure 1. (a) Static weights may be mapped to improve the latency of an individual mvm ( tp ) or enable mvm to execute concurrently ( bp ). (b) Existing algorithms always maximize tp agnostically to inference context. (c) In contrast, camp maps in a context-aware manner by first encoding the context in a conflict matrix. (d) A refinement loop converges to a mapping with an optimized balance of tp , bp , and ep , (e) improving runtime efficiency via better utilization of 3dimc compute resources. Five-panel overview. Panel (a) contrasts a batch-parallel mapping, which places differently colored \acs*{qkv}, output, \acs*{g+u}, and down-projection weights on disjoint horizontal resources, with a tensor-parallel mapping that spreads weights horizontally but serializes work across vertical tiers. Panel (b) shows that a context-agnostic, tensor-parallel mapping underutilizes the \acs*{3dimc} macro. Panels (c) and (d) show \acs*{camp} converting architecture, network, and runtime context into a conflict matrix, then repeatedly refining and simulating a mapping. Panel (e) compares active compute resources over normalized time: \acs*{camp} averages 60 percent utilization versus 42 percent for Greedy-Util and completes the work in roughly half the time.
Figure 2. Each tier in an 3dimc memory array stores weights that are used to perform mvm in-situ. Figure illustrating a \acs*{3dimc} stack as five tiers of memory arrays. The middle array is blown up to highlight the internal crossbar which contains weight values. Arrows indicate how input activations flow into the array, and output activations flow out.
Figure 3. Hierarchical H-tree architecture (a-e) ( Shim and Yu, 2021 ; Shim et al., 2021 ; Jang et al., 2025 ) . Weight mapping options when leaf groups comprise 4 leaves (f-i). Nine-panel architecture and dataflow diagram. Panels (a)–(e) zoom from an attention unit connected to an \acs*{3dimc} macro, through the macro's H-tree of leaf groups, to leaves, accumulators, and a vertically stacked \acs*{3dimc} tile subject to the one-tier-at-a-time constraint. Panels (f) and (g) split a weight matrix by output columns and then input rows. Panel (h) compares one-, two-, and four-way tensor-parallel placements: increasing the degree shifts capacity use from vertical tiers to the planar dimension, broadcasts input chunks to more leaves, and accumulates more partial outputs. Panel (i) gathers the resulting output tensors into the global IO buffer.
Figure 4. Impact of 3dimc mapping strategies on decode-step throughput for Llama 3.1-70B across representative runtime contexts. Results shown for a sequence length of 128. Line chart of normalized throughput for throughput-friendly, latency-friendly, and balanced mappings under three serving contexts. At batch size 1, the latency-friendly mapping is best at 1.0 while the other mappings are near 0.85–0.9. At batch size 16 with on-chip streaming attention, the throughput-friendly mapping is best at about 1.0 and the latency-friendly mapping falls to 0.51. With off-chip attention at batch size 16, the balanced mapping is best at about 1.0, followed by the throughput-friendly mapping at 0.86 and the latency-friendly mapping at 0.60.
Figure 5. Overview of the camp mapping algorithm. Flowchart of \acs*{camp}'s three phases. The \acs*{llm} definition, runtime context, optional mixture-of-experts distribution, and system configuration feed a profiling phase in which graph tracing and a computational graph are evaluated by a system-level simulator to produce a conflict matrix. The initial-placement phase applies Place to produce mapping M. The refinement phase simulates candidate mapping M prime, measures performance l prime, applies another refinement action, and feeds the candidate back until the best mapping is selected.
Figure 6. Conflict matrices shown as heatmaps across workload configurations and inference contexts. An entry in row i and column j corresponds to the temporal proximity of mvm i and j among the set of mvm to be mapped. Five blue heatmaps whose intensity encodes pairwise \acs*{mvm} temporal proximity from zero to above 0.8. For four Llama 3.1 70B layers, panel (a), batch size 2 with on-chip streaming attention, contains small sparse diagonal clusters; panel (b), batch size 8 with streaming attention, becomes a broad connected diagonal band; and panel (c), batch size 8 with off-chip attention, separates into four isolated clusters. For four Mixtral 8-by-7B layers at batch size 8, panel (d) shows broad block-diagonal conflicts under a uniform expert distribution, whereas panel (e) shows nonuniform stripes and spots within the blocks under an imbalanced distribution.
Figure 7. mvm speedup across mapping algorithms, workloads, and batch sizes. An additional plot for ongoing compute utilization in terms of the number of active lg is also appended. Four grouped-bar charts report throughput normalized to Greedy-Util for Llama 3 8B, Llama 3 70B, Qwen 72B, and Mixtral 8-by-7B at batch sizes 1, 2, 4, 8, and 16. \acs*{camp} matches the baselines near batch size 1, then rises with batch size to approximately 1.95, 1.90, 2.00, and 1.71 times throughput, respectively, while 3D-aCortex remains near or below one; the gray full-compute bounds are higher. A linked time trace for Llama 70B at batch size 8 shows \acs*{camp} sustaining roughly 35–48 active leaf groups for about half the normalized duration, whereas Greedy-Util usually activates fewer than 25 leaf groups and runs for the full duration.
Figure 8. Normalized Llama 70B throughput vs. vertical tier count across mapping algorithms. Line chart of normalized throughput versus 280–380 vertical tiers for Greedy-Util, 3D-aCortex, and \acs*{camp} at batch sizes 1, 2, 4, 8, and 16. All methods coincide at 1.0 for batch size 1, and both baselines remain nearly flat at every batch size. \acs*{camp} is also flat at batch size 2 but improves as more tiers permit more flexible mappings at larger batches: it rises from about 3.1 to 3.3 at batch size 4, from 2.7 to 4.1 at batch size 8, and from 3.0 to 4.4 at batch size 16.
Figure 9. Llama 70B speedup with mappings tuned by camp for batch sizes 1 and 8, compared with Greedy-Util. The two mappings remain robust across batch sizes, avoiding remapping and weight programming for new batch sizes. Line chart of normalized throughput versus batch size from 1 to 32. The \acs*{camp} mapping tuned for batch size 1 nearly overlaps Greedy-Util, rising from roughly 1.1 at batch size 1 to about 2.5 at batch sizes 8–32. The \acs*{camp} mapping tuned for batch size 8 begins at 1.0, matches the other mappings near batch size 2, then rises sharply to 3.6 at batch size 4 and approximately 4.5–4.8 at batch sizes 8–32. Shaded regions recommend the batch-1 mapping below batch size 4 and the batch-8 mapping from batch size 4 onward.
Figure 10. Optimization curves illustrating latency improvement during the refinement phase with different beam search parameters W and k . The target workload is Llama 70B with a batch size of 8. Step curves of normalized latency versus mapping evaluations on a logarithmic horizontal axis for beam-search settings W equals k equals 1, 5, 10, and 15. All searches begin at the initial-placement latency of about 0.74 relative to the Greedy-Util reference at 1.0. The fully greedy search reaches about 0.52 in fewer than 50 evaluations, while wider searches continue longer and ultimately reach approximately 0.49–0.50; the W equals k equals 15 curve requires several thousand evaluations.
Figure 11. End-to-end decode phase speedup of Llama 70B with a 3dimc accelerator paired with various attention units, relative to the 2 × A100/H100 gpu baselines. Batch sizes of 16/16/64 are utilized for A100/H100/B200 gpu , respectively. In the 2- gpu systems, one 3dimc accelerator is shared between two gpu , each serving the above-mentioned batch sizes. Four grouped-bar panels show end-to-end throughput normalized to two A100 \acsp*{gpu} for sequence lengths 128, 1024, 2048, and 4096. Each panel compares \acs*{gpu}-only A100 and H100 baselines with \acs*{3dimc} systems using A100, H100, H100 with FP8 key-value cache, B200, or B200 with FP8, including configurations shared by two \acsp*{gpu}. \acs*{camp} exceeds Greedy-Util in every \acs*{3dimc} configuration, with larger absolute gains for faster attention units and shared systems. \acs*{camp}'s best speedups are about 15.7, 17.4, 19.0, and 16.3 at the four sequence lengths, versus Greedy-Util maxima of 9.0, 10.0, 11.0, and 12.7.
Figure 12. Throughput comparison for Llama 70B in a grid of 3dimc macro topology configurations and mapping algorithms. Topology configurations use the format < NLG×NL×NT >: number of lg , leaves per lg , and tiles per leaf. Grid of heatmaps comparing Greedy-Util, 3D-aCortex, and \acs*{camp} across eleven \acs*{3dimc} macro topologies and batch sizes 1, 2, 4, 8, and 16. Cell labels give speedup relative to the slowest configuration. Two topologies with 16 leaves per tile group are invalid because they lack output granularity. Differences are small at batch size 1, while \acs*{camp} is consistently darkest and fastest at larger batches, reaching up to 2.82 times at batch size 16. The highlighted 64-by-4-by-4 topology has a mean speedup of 1.56 across batches, near the best overall mean of 1.59.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13. Throughput speedup of camp over Greedy-Util ( Büchel et al., 2025 ) as the number of input/output channels and mvm latency vary, profiled on Llama 70B. A three-by-four grid of heatmaps reports \acs*{camp} speedup over Greedy-Util. Columns group \acs*{mvm} latencies of 100, 200, and 400 nanoseconds and one, two, or four H-tree output channels; row groups use batch sizes 1, 2, 4, and 8, each with one, two, or four H-tree input channels. Speedup is exactly one at batch size 1 and remains near one at batch size 2. At batches 4 and 8, additional input and output channels generally increase speedup, with a maximum of 1.91 for batch size 8, 100-nanosecond \acsp*{mvm}, four input channels, and four output channels; gains generally shrink for slower \acsp*{mvm}.
Figure 14. Expert co-activation frequencies with respect to uniform frequency in Mixtral-8 × 7B for 50 MMLU prompts ( Hendrycks et al., 2021 ) from the “global facts” (left, layer 31) and “business ethics” (right, layer 11) domains. Although moe -based networks are trained to balance expert utilization, co-activation frequencies vary across domains and layers. Two 8-by-8 expert co-activation matrices for Mixtral 8-by-7B, with accompanying marginal-frequency bars. The left matrix represents global-facts prompts at layer 31 and the right represents business-ethics prompts at layer 11. Diagonal entries are zero, while off-diagonal values vary substantially rather than forming a uniform pattern; the two domains emphasize different expert pairs and have different per-expert marginal frequencies.
Figure 15. Speedup across expert-activation distributions derived empirically from input domains in the MMLU dataset ( Hendrycks et al., 2021 ) . “Uniform” and “imbalanced” are synthetic distributions illustrating the extremes. Uniform is fully uniform on expert-co-activations. And for “imbalanced”, we increased coefficient of variation (CV = std/mean) for co-activation probabilities to 0.8 where the realistic domains from MMLU have it between 0.15 and 0.32. Results are averaged over 50 trials. Grouped bars compare normalized throughput for Greedy-Util, 3D-aCortex, and \acs*{camp} at batch sizes 1 and 8. Both baselines remain at approximately 1.0 for all expert distributions. \acs*{camp} is shown for uniform, formal-logic, anatomy, global-facts, machine-learning, and synthetic imbalanced distributions; its throughput ranges from about 1.10 to 1.13 at batch size 1 and increases monotonically from 1.63 for the uniform distribution to 1.82 for the imbalanced distribution at batch size 8.
Figure 16. Estimated number of options for mapping L Llama 70B transformer layers onto the modeled 3dimc macro. The red curve shows the Monte Carlo estimate; the other curves show conservative lower and upper bounds. Line chart of the base-2 logarithm of the number of mappings versus one through eight mapped transformer layers. The conservative upper bound grows from about 38 to 304, the Monte Carlo estimate grows from about 22 to 175 with a broad uncertainty band, and the lower bound grows from zero to about 61. A dashed horizontal line near 18 marks the evaluation budget for obtaining ground truth and is exceeded by the estimate even for one layer.
Figure 17. Ablation on parameters of beam width W and top k actions. Four four-by-four heatmaps sweep beam width W and action width k over 1, 5, 10, and 15. The two throughput heatmaps show that every setting reaches normalized throughput 1.0 at batch size 1, while batch-size-8 throughput ranges from 0.899 to 1.0 and generally improves with wider beams and actions. The two search-evaluation heatmaps show rapidly increasing cost: evaluations range from 56 at W equals k equals 1 to 8,360 at batch size 1 and 3,930 at batch size 8 when both widths equal 15.
Figure 18. Refinement iterations with beam search. Beam-search schematic. An initial state containing mapping M-init and latency l-init branches through refinement actions into W candidate mapping states in Beam 1. Each retained state branches into k new states in the next beam; only W states remain above the horizontal divider while the other open-parenthesis k minus 1 close-parenthesis times W states are pruned below it. Beams continue until no actions remain, and the mapping with the best simulated latency among all visited states is returned. Colored stacks identify \acs*{qkv}, output, \acs*{g+u}, and down-projection placements.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.
Analog in-memory computing (AIMC) speeds up neural-network inference by doing the arithmetic directly inside a memory array, instead of shuttling weights back and forth between memory and a processor. This saves energy, but the physical devices that store the weights are imperfect: programming errors, electrical noise, limited-resolution converters, and outright broken cells all distort the computation, and every physical chip is distorted in its own way. A designer with several such chips available faces an uncomfortable choice: run all of them and combine the answers (safe, but wasteful of energy), or trust a single chip blindly (cheap, but with no guarantee on how often it is wrong). This paper introduces RACE-AIMC (Risk-Aware Certified Ensemble for AIMC), a framework that resolves this choice with statistics rather than guesswork. Offline, RACE-AIMC studies a pool of physical accelerators, picks the single best one for a given energy budget, and computes a mathematically exact upper bound on how often that accelerator will be wrong when it chooses to answer. Online, only that one accelerator is switched on; a lightweight check decides whether to accept its answer or defer to a fallback. In our simulations using a noisy weight mapping and multiple independent test runs, every certified bound stayed under a 10% error target (mean bound 7.83% +- 0.89%, with 70.88% +- 0.98% of inputs answered directly). The resulting system matches the accuracy of a clean digital baseline while cutting modeled energy use by 69.02% relative to always running every accelerator in the pool.
Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.
Yuzhuang Xu, Xu Han, Yuxuan Li +1
Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China