Test-time scaling (TTS) improves large language model (LLM) reasoning by sampling multiple answers, but fixed sampling budgets ignore both query difficulty and the changing energy cost of inference under GPU contention. We measure up to a 4.1-fold increase in energy for the same four-sample query as GPU load rises. We present ATEST, a resource-aware TTS framework that jointly considers the expected value of additional samples and their current GPU cost. ATEST-Online, its default mode, requires no offline calibration: it incrementally generates samples, estimates the stability of the plurality answer, monitors live GPU conditions, and learns step energy cost online to decide when to stop. ATEST-Calibrated instead selects a per-query budget using offline accuracy and cost models, favoring deployments with stable loads. We evaluate both modes on four reasoning and question-answering datasets using Phi-3-mini on an NVIDIA A100 under controlled varying load, and test transfer to an NVIDIA L40S and Qwen2.5-3B. Across the four A100 datasets, ATEST-Online answers every query without exceeding the energy budget, while achieving the highest or statistically tied accuracy and reducing sampling as load increases. On GSM8K at high load, it achieves 60.0% accuracy with no deferrals, versus 44.0% accuracy and 42.4% deferrals for ATEST-Calibrated. Ablations show that online cost learning drives adaptation to load. Including offline profiling, ATEST-Online uses 42 times less GPU time and 48 times less energy than ATEST-Calibrated for the evaluated workload. Online control trades higher tail latency and lower throughput on short tasks for robustness under varying load; offline calibration remains competitive under stable conditions.
Figures & tables
Fig. 1: The idea of ATEST . Standard test-time scaling (TTS) generates the same number of samples for a query whether the GPU is idle or busy. ATEST looks at the live GPU load. When the GPU is busy and each sample costs more energy, it skips samples that are not worth their cost.
Fig. 2: Overview of ATEST . (a) Standard TTS generates a fixed number of samples B and does not use the live GPU state. (b) ATEST -Calibrated, the offline mode of ATEST , builds accuracy and cost models offline for each dataset and picks B once per query. (c) ATEST -Online starts with one sample and adds samples step by step ( 1→3→4→6→8 ). (d) An illustration of one decision: \scriptsize1⃝ a Monte Carlo look-ahead to Bmax=8 estimates how likely the current winner is to stay the winner, and ΔU is how much the next step raises this; \scriptsize2⃝ the online cost model learns from the measured energy of every step, follows the load, and predicts the next step’s energy with a safety margin; \scriptsize3⃝ with the same votes, ATEST -Online continues on an idle GPU but stops on a busy one.
Method
Cost unit
Live GPU state
Self-consistency family [ 2 , 3 , 4 , 6 ]
samples
✗
↪ Fixed-4 (baseline)
samples
✗
Adaptive allocation family [ 7 , 5 , 8 ]
tokens, compute
✗
↪ Adaptive-Cal (baseline)
energy (model)
✗
ATEST (this work)
↪ ATEST -Calibrated (offline mode)
energy (model)
✓
TABLE I: Families of TTS methods. Indented rows are the methods we run.
Fig. 3: Motivation. (a) Energy per answered query for a fixed budget of B=4 at four GPU load levels, relative to idle. The value under each level is the measured GPU utilization. (b) Error of the offline cost model, averaged over four datasets. Without live GPU state (Adaptive-Cal), the error grows from 26% at idle to 283% at high load. Adding live GPU state ( ATEST -Calibrated) lowers the high-load error to 55%, but the error is still large at medium load (130%).
Property
ATEST -Calibrated
ATEST -Online
Value of more samples
Offline accuracy model
Live vote stability
Cost model
Built offline, fixed
Learned online (RLS)
Uses live GPU state
Yes
Yes
Uses current samples
No
Yes
When B is chosen
Once, before sampling
Step by step
Dataset profiling
Required
Not required
TABLE II: The two ATEST modes.
Method
Acc. ↑
Defer ↓
Mean
J/corr. ↓
Budget
(%)
B
used (%)
GSM8K
Fixed-4
0.364
51.8
1.928
312.4
93.1
Adaptive-Cal
0.368
48.8
1.726
330.5
99.5
ATEST -Calibrated
0.600
11.2
1.796
196.5
96.5
ATEST -Online
0.626
0.0
1.580
191.9
98.3
MATH
Fixed-4
0.246
49.6
2.016
600.5
86.1
TABLE III: Overall results under the idle → low → medium → high load schedule (500 queries per dataset). Our default mode, ATEST -Online, is shaded. Accuracy counts deferred queries as wrong. Mean B counts deferred queries as B=0 . Budget used above 100% (red) means the energy budget was exceeded. Bold: best value per dataset.
Fig. 4: Behavior at each load level (125 queries per level). Top: accuracy over all queries (deferred = wrong). Middle: deferred queries (%). Bottom: mean B over answered queries. ATEST -Online (blue, solid) never defers and lowers its budget as load rises. ATEST -Calibrated (orange, dashed) also lowers its budget, but it starts deferring at medium or high load on all four datasets, which pulls its accuracy down.
ATEST -Calibrated
ATEST -Online
Offline profiling (before serving)
GPU time (h)
5.73
0
Energy (MJ)
7.16
0
Generated samples
176,000
0
Serving 2,000 queries
Run time (h)
0.14
0.14
TABLE IV: End-to-end cost of The two ATEST modes on four datasets (2,000 queries under the load schedule). ATEST -Calibrated must profile every dataset before serving. ATEST -Online starts serving at once and learns its cost model during serving, so its learning cost is included in the serving numbers.
Variant
GPU state
Online learning
Load response
Full-HW
Yes
Yes
Yes
noHW-State
No
Yes
Yes
ReasoningOnly-Strict
No
No
No ( B=3.12 )
TABLE V: Controlled ablation of ATEST -Online with a non-binding budget and a fixed remaining-energy signal (§ VI-H ). “Load response” means that mean B changes across load levels. ReasoningOnly-Strict uses the same mean budget, B=3.12 samples per query, at every load level.
ATEST -Calibrated
ATEST -Online
Dataset
p95
QPS
p95
QPS
MAPE
GSM8K
0.509
2.62
1.081
2.62
23%
MATH
1.019
1.78
1.608
1.99
22%
ARC-C
0.130
19.72
0.111
17.19
134%
CSQA
0.086
29.69
0.180
16.60
112%
TABLE VI: Runtime behavior of the two ATEST modes under the load schedule: p95 latency per query (s), throughput (QPS), and the energy error of the online cost model of ATEST -Online (MAPE).
Phi-3-mini on A100
Phi-3-mini on L40S
Qwen2.5-3B on L40S
Dataset
Method
Acc. ↑
Defer (%) ↓
J/corr. ↓
Acc. ↑
Defer (%) ↓
J/corr. ↓
Acc. ↑
Defer (%) ↓
J/corr. ↓
GSM8K
Fixed-4
0.364
51.8
312.4
0.506
34.2
386.2
0.138
48.8
1019
ATEST -Online
0.626
0.0
191.9
0.638
0.0
303.0
0.280
0.0
536
MATH
Fixed-4
0.246
49.6
600.5
0.380
24.4
730.3
0.242
35.4
1067
ATEST -Online
0.426
0.0
371.2
0.448
0.0
614.3
0.348
0.0
700
ARC-C
Fixed-4
0.568
30.8
25.1
0.672
15.4
31.4
0.380
26.0
155
TABLE VII: Results on a second GPU and a second model (500 queries per dataset, same load schedule, budget rule, and frozen controller settings). A100 values are from Table III . Emax is re-measured for each GPU and model, so joules can be compared within a column group but not across groups. Bold: better of the two methods in each group.
Accuracy ↑
J/correct ↓
Dataset
Fixed-4
Cal.
Online
Fixed-4
Cal.
Online
GSM8K
0.668
0.686
0.632
176.5
173.3
191.1
MATH
0.446
0.402
0.402
357.0
421.9
414.8
ARC-C
0.732
0.792
0.794
12.6
18.4
17.3
CSQA
0.712
0.726
0.716
10.5
13.4
11.2
TABLE VIII: Results with a stable GPU (no injected load, 500 test queries per dataset). Accuracy counts deferred queries as wrong: Fixed-4 defers 10–50 queries, ATEST -Calibrated 0–11 queries, and ATEST -Online none.
Test-time scaling (TTS) has become an effective approach for improving large language model performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework, AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width--depth TTS as controller synthesis over pre-collected reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the agent diagnose why a TTS program fails. Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy--cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes. Our data, and code will be open-source at https://github.com/zhengkid/AutoTTS.
Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.
Mobina Kashaniyan, Ali Jannesari
Department of Computer Science Iowa State University, USA
Test-time scaling improves LLM reasoning by generating and aggregating multiple candidate answers, yet many pipelines use fixed per-query budgets that spend the same compute on easy and difficult prompts. These fixed budgets are also difficult to inspect because they do not explain why a given prompt receives a particular number of samples. We propose adaptive} test-time scaling with a lightweight fuzzy controller that maps interpretable signals, including estimated prompt complexity and model confidence, to a per-query sampling budget. The controller assigns fewer samples to easier or more confident prompts and more samples to harder or less certain prompts, making inference-time compute inspectable rather than fixed or opaque. We evaluate under a fair-alignment protocol with matched decoding settings and controlled answer selection, and compare against best-of-N, compute-aware scaling, and self-certainty-based baselines on question-answering and mathematical reasoning tasks. Across models and datasets, adaptive fuzzy control improves over several standard baselines and remains close to a selector-matched full-budget control while reducing the average number of samples. These findings suggest that interpretable adaptive sampling is a practical direction for more efficient test-time reasoning in large language models.