Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints

Shai Feldman; Yaniv Romano

cs.LGOct 5, 2026

Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints

Authors: Shai Feldman, Yaniv Romano

Organizations: Department of Computer Science, Technion IIT, Israel · Department of Electrical and Computer Engineering, Technion IIT, Israel

Abstract

We evaluate large language models (LLMs) in multi-turn interactions through their time-to-event: the number of interaction steps required to produce an event of interest, such as a successful jailbreak or agentic task completion. Under limited compute, interactions may be terminated before the event occurs, so that event times are only partially observed (censored). Existing allocation methods for calibrating time-to-event bounds satisfy the budget only in expectation and can exceed the available budget on a particular evaluation run. Enforcing a hard constraint is particularly challenging as the cost of a trajectory is initially unknown. We introduce Hard-budget Allocation with Reflow for Predictive calibration (HARP), a budget allocation that satisfies hard resource constraints and adaptively reallocates unused budget. We show how to use HARP to construct lower predictive bounds (LPBs) on the time-to-event and to estimate evaluation metrics such as the jailbreak rate on a fixed benchmark. Although HARP induces dependence in acquisition decisions across different trajectories, we prove that HARP never exceeds the target budget, that its LPBs have finite-sample coverage guarantees, and that its metric estimates are unbiased. Experiments on agentic task success, LLM jailbreaks, toxic content generation, and RAG hallucinations show that HARP achieves coverage close to the nominal level with low variance, while never exceeding the given budget.

Figures & tables

Figure 1: Toxicity LPBs. Coverage comparison: absolute coverage deviation and paid turns per parent prompt, all four target models, 90% coverage,

\bar{B}=20

turns per sample, twenty runs. HARP -D (+depth); chosen once for the LPB task by the task-level criterion in Appendix F.4 . Uncalibrated uses raw quantiles and has no cost box. The dashed red line marks the budget.

Figure 2: AutoIF LPBs and UPBs, Qwen 2.5 14B. Coverage deviation (left) and budget used per sample (right), over twenty runs. Target coverage: 90% for LPB and 80% for UPB; target budgets: 10 and 20 turns per sample, respectively (red dashed marks). HARP uses D with depth for both bounds, selected by the criterion in Appendix F.4 . Uncalibrated has no acquisition-cost box.

Figure 3: Red-team event rates, Llama Guard judge. Estimates and across-seed variances, all four targets,

\bar{B}=20

turns per sample, twenty runs. HARP -D (no depth); chosen once for metric estimation by the task-level criterion in Appendix F.4 . Red marks denote benchmark-cohort means. Acquired methods use the same estimator.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset/judge	LPB	UPB	Metrics
Toxicity	20	20	20
Red team, Qwen	10	20	20
Red team, Llama Guard	10	20	20
Hallucination	10	20	20
AutoIF	10	20	20

Appendix

Table 1: Target budgets in paid interaction turns per parent calibration sample. Each setting uses the same budget for all four target models; both acquired methods include their planning and control costs.

Variant	LPB	UPB	Event rate	Restricted mean
HARP -C (no depth)	1.935	0.622	0.632	0.466
HARP -C (+depth)	1.017	0.678	1.081	0.720
HARP -D (no depth)	1.410	0.615	0.671	0.457
HARP -D (+depth)	0.948	0.585	1.200	0.819

Appendix

Table 2: Four-variant comparison over twenty runs. Entries are median within-configuration variance ratios to DAPRO : LPB/UPB coverage and the two population metrics. All four variants are retained to assess C/D and depth.

Figure 4: Toxicity LPBs. 90% coverage,

\bar{B}=20

turns per sample, twenty runs per target. HARP -D (+depth); chosen once for the LPB task by the task-level criterion in Appendix F.4 . Black/red dashed lines denote coverage/budget. All methods use matched splits.

Figure 5: Red team, Qwen judge LPBs. 90% coverage,

\bar{B}=10

Figure 6: Red team, Llama Guard LPBs. 90% coverage,

\bar{B}=10

Figure 7: Hallucination LPBs. 90% coverage,

\bar{B}=10

Figure 8: AutoIF LPBs. 90% coverage,

\bar{B}=10

Figure 9: Toxicity population metrics. Budget

\bar{B}=20

turns per sample, twenty runs per target. HARP -D (no depth); chosen once for metric estimation by the task-level criterion in Appendix F.4 . The event is harmful/ungrounded in safety settings and successful completion for AutoIF; restricted mean is

\mathbb{E}[\min(T,200)]