TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving

Chenxu Niu

cs.ARAug 11, 2026

TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving

Authors: Chenxu Niu

Abstract

Energy-aware LLM serving requires comparing configurations under realistic request shapes, yet exhaustive target-GPU profiling is costly and a cheap predictor can be dangerously confident outside its measured scope. We present TokenPowerSandbox, an evidence-gated workflow that combines an interpretable CPU-resident projector, short target-GPU probes, full-workload verification, and tamper-evident freeze-before-measurement provenance. On one NVIDIA H100 80GB serving Qwen2.5-7B-Instruct with vLLM, three anchor repeats and six development workloads calibrate workload transfer. The same frozen model is evaluated on a blind holdout and a separately predeclared no-refit confirmation totaling 51 post-freeze runs. Energy MAPE is 6.23% and 7.35%, with Spearman rank correlations of 0.976 and 0.933. However, a predeclared TTFT gate passes at concurrency four (9.27% MAPE) and triggers abstention below four (64.80%), showing why energy accuracy cannot certify latency.

Figures & tables

Panel of Figure 1 — Figure 1. Evidence contract and acquisition paths. Sandbox records are simulated. Predictions, identities, and the balanced schedule are hash-frozen before the interleaved Probe/Verify measurements; the configuration corpus compares fidelities retrospectively. The gate releases only identity-, chronology-, and metric-scope-consistent claims or abstains. A flow diagram starts from an experiment contract, produces CPU Sandbox estimates, and freezes predictions and the run plan before branching to measured Probe and full-workload Verify acquisition. Both measured paths feed an evidence gate that releases a bounded claim or abstains.

	Holdout v2 MAPE		Corrected rank
Metric	Base	Corrected	$\rho$	Pairs
Energy/1k output	63.01%	6.23%	0.976	27/28
Output throughput	23.93%	10.53%	0.976	27/28
P95 TTFT	73.44%	32.67%	0.952	26/28
P95 TPOT	34.29%	7.99%	0.929	25/28
	No-refit scope confirmation v3

Table 1. Frozen workload-transfer results. Base values are the uncorrected analytical projector; corrected values use development-only residuals. Pair counts exclude observed ties.

Panel of Figure 2 — Figure 2. Signed error of frozen CPU energy predictions on the blind holdout and no-refit scope confirmation. Positive values denote overprediction. All 17 workloads remain inside a $\pm 15\%$ band; for the no-refit confirmation, 15% was the predeclared primary endpoint. Rank correlations are 0.976 and 0.933. A scatter plot compares signed CPU energy-prediction errors for the blind holdout and no-refit confirmation across observed device-energy levels. All 17 points lie inside the plus-or-minus 15 percent band.

Panel of Figure 3 — Figure 3. No-refit scope confirmation by concurrency $k$ . The campaign passes its predeclared aggregate energy-MAPE endpoint; observed energy MAPE is also below 15% in each stratum. TTFT passes its 20% boundary test at $k=4$ but fails below it, triggering explicit abstention. Horizontal bars compare energy and TTFT mean absolute percentage errors at concurrency one, two, and four. Energy remains below the 15 percent reference in all strata, while TTFT exceeds its gate at concurrency one and two.

Figure 4. Why full verification remains necessary. Left: Sandbox P95-TTFT rank is nearly reversed relative to Verify (

\rho=-0.968{}

); a short measured Probe repairs most ordering. Right: Verify medians for all 12 frozen candidates, with an inset magnifying the five SLO-feasible candidates; hollow points are the seven candidates whose TTFT exceeds the 1,600-ms SLO. Rings denote the three-candidate Probe frontier (one of three lies on the verified Pareto set), the square marks the expert baseline on the Sandbox stage-local frontier—the choice if Sandbox were used alone—and the star marks the unique verified Pareto configuration within the frozen 12-candidate set. Two panels show configuration fidelity. The left panel traces TTFT ranks across Sandbox, Probe, and Verify stages. The right panel plots verified median energy against median output-token throughput for all 12 frozen candidates and magnifies the five SLO-feasible points, marking the Probe frontier, expert baseline, and unique verified Pareto point within that candidate set.

Metric	Expert	Pareto	Change
Energy (J/1k out.)	371.13	367.27	$-1.04\%{}$
Output tok/s	1732.60	1785.63	$+3.06\%{}$
P95 TTFT (ms)	1343.60	1053.56	$-21.59\%{}$
P95 TPOT (ms)	17.25	16.76	$-2.84\%{}$

Table 2. Discovery-corpus median Verify outcomes for the predeclared expert baseline and the unique SLO-feasible Pareto configuration within the frozen 12-candidate set.

Metric	Expert median	Winner median	Paired effect
Energy (J/1k out.)	372.21	367.26	$-1.39\%$ mean
Output tok/s	1727.44	1783.71	$+3.46\%$ mean
P95 TTFT (ms)	1340.92	1057.11	$-21.43\%$ median
P95 TPOT (ms)	17.29	16.77	$-3.30\%$ mean

Table 3. Independent five-pair Verify confirmation after candidate lock.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 211 papers

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
Aug 28, 2026Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton +1Energy-Aware LLM Inference LLM Inference
Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA
EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
May 14, 2026Zhiye Song, Kyungmi Lee, Eun Kyung Lee +3LLM Inference Efficiency Energy-Efficient ML
Massachusetts Institute of Technology Cambridge, MA 02139 · IBM Research Yorktown Heights, NY 10598
Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
Apr 10, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezEnergy-Aware LLM Inference LLM Inference Efficiency
Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland

Aug 28, 2026J/K move · Enter open · S save

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

Prabhu Vellaisamy, Vanessa Lam, Shawn Blanton, John Paul Shen

Electrical and Computer Engineering Department Carnegie Mellon University Pittsburgh, PA, USA

Energy-Aware LLM Inference LLM Inference LLM Inference Efficiency LLM Serving Efficient Language Model Inference

Open paper AskPDF arXiv

Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can decrease even when total request energy increases. We characterize this behavior with a decomposed energy model: a fixed one-time prefill with a fixed generation setup cost, while each output-token generation step adds marginal step energy. We evaluate this LLM inference energy model on NVIDIA H100 and H200 GPUs across dense and mixture-of-experts (MoE) models, reporting both request energy and token energy as functions of model type (M), phase (P), batch size (B), context length (C), and output length (N). For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token while total batched inference-window energy increases from 1.19 to 5.93 kJ. Batching also reduces token energy, but the gain is context-bounded: at 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. MoE models amplify this effect: sparse routing and fragmented expert execution increase fixed energy at low concurrency, while batching spreads that energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. These results show that energy-aware serving should jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

Abstract

LLM Serving

Explore similar work

Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures