Authors: Ruicheng Ao, Hongyu Chen, Siyang Gao, Hanwei Li, David Simchi-Levi
Organizations: Institute for Data, Systems, and Society, Massachusetts Institute of Technology · School of Data Science, City University of Hong Kong · Department of Decision Analytics and Operations, City University of Hong Kong · Department of Civil and Environmental Engineering, Massachusetts Institute of Technology · Operations Research Center, Massachusetts Institute of Technology
In service system design, it is often the case that the main evidence of system performance appears in textual form, such as customer support transcripts, complaint narratives, and compliance review reports. With the increasing use of large language models (LLMs), these textual trajectories can be converted into low-cost proxy scores that support comparison and optimization of candidate service configurations. However, LLM-based evaluations can exhibit systematic bias and require human audits for correction. To this end, this research studies how to identify the best service configuration with high confidence while minimizing the use of expensive human audits. We formulate the problem as fixed-confidence best arm identification with an always-observed, potentially biased proxy and selectively acquired human-verified outcomes. Moreover, several configurations can run on one case. A human audit can provide a case label that scores all their outputs. We show that sharing this reference and deciding whether to audit after observing proxy scores can each strictly reduce asymptotic identification cost. The selected configurations determine the available proxy scores and verified outcomes, so the proposed algorithm allocates configuration evaluations and subsequent audits jointly. For a specified finite family of data-generating models containing the true law and distinguishing different best configurations, its expected cost asymptotically attains the information-theoretic lower bound. For audits of individual outputs, we develop prediction-powered confidence sequences valid under adaptive auditing and delayed feedback. Numerical experiments on synthetic examples and service datasets show that the proposed algorithms require substantially less human review while maintaining reliable selection.
Figures & tables
Figure 1: Anytime-valid coverage validation. Empirical coverage rates exceed theoretical 1−δ targets across all confidence levels and sample sizes. Error bars show 95% bootstrap confidence intervals over 1000 trials. The polynomial stitched boundary confidence sequence maintains guaranteed coverage under adaptive sampling.
Figure 2: Allocator performance across gap settings ( Δ∈{0.10,0.15,0.20} ). Neyman allocation achieves 48–50% cost reduction over Uniform baseline while lying within 20% of Oracle upper bound. Error bars show standard deviations over 20 trials. The relative ordering of strategies is consistent across all gap values.
Figure 3: Failure mode validation. Adaptive auditing (UncertaintyWeighted allocator) achieves 10% cost reduction over fixed auditing while maintaining 100% accuracy. No-Judge baseline requires full auditing (cost 10,500), while No-Audit fails completely (0% accuracy). The cost bars use log scale to accommodate the large range.
Designing service systems requires selecting among alternative configurations -- choosing the best chatbot variant, the optimal routing policy, or the most effective quality control procedure. In many service systems, the primary evidence of performance quality is textual -- customer support transcripts, complaint narratives, compliance review reports -- rather than the scalar measurements assumed by classical optimization methods. Large language models (LLMs) can read such textual evidence and produce standardized quality scores, but these automated judges exhibit systematic biases that vary across alternatives and evaluation instances. Human expert review remains accurate but costly. We study how to identify the best service configuration with high confidence while minimizing expensive human audits, given that automated evaluation is cheap but biased. We formalize this as a sequential decision problem where a biased proxy score is observed for every evaluation, and a verified outcome can be acquired selectively at additional cost. We prove that LLM-only selection fails under arm-dependent bias, and that naive selective-audit estimators can be asymptotically biased. We develop an estimator combining proxy scores with inverse-propensity-weighted residuals and construct anytime-valid confidence sequences. Our algorithm, PP-LUCB, jointly decides which alternatives to evaluate and whether to request human audits, concentrating reviews where the LLM judge is least reliable. We prove correctness and establish instance-dependent cost bounds showing near-optimal efficiency. On a customer support ticket classification task, our algorithm correctly identifies the best model in 40/40 trials while achieving 90% audit cost reduction.
Ruicheng Ao, Hongyu Chen, Siyang Gao +2
Institute for Data, Systems, and Society, Massachusetts Institute of Technology. · School of Data Science, City University of Hong Kong. · Department of Decision Analytics and Operations, City University of Hong Kong.
LLM service selection requires performance evidence that reflects the demands of the current task. Aggregate scores obscure differences in capability and difficulty, while semantically related examples can involve different combinations of skills. We present RouteAlign, which uses task requirements to structure both the measurement of model performance and the selection of services. Task requirements describe capability demand and difficulty, enabling alignment with corresponding model performance measurements. Historical requirements determine how outcomes contribute to global profiles and local capability estimates; incoming requirements determine how those measurements inform the current decision. A shared scorer integrates the evidence, and a generation-cost preference controls selection among eligible services. Evaluation across diverse tasks and heterogeneous LLM services shows that RouteAlign extends the empirical quality-cost frontier toward higher task quality. Comparisons with open Jev-style decision backends show consistent quality gains, with lower generation cost than their strongest tested configuration. Ablations support the contribution of local performance evidence, while catalog-update experiments demonstrate effective reuse of the fitted scorer with refreshed model measurements.
Jihao Zhao, Ding Chen, Yanfang Chen +4
MemTensor (Shanghai) Technology Co., Ltd., Shanghai, China · Renmin University of China, Beijing, China · Shanghai Jiao Tong University, Shanghai, China
Inspired by the problem of identifying the best model from a collection of large language models (LLMs) with heterogeneous querying costs, we formulate and analyse a variant of the multi-armed bandit (MAB) with (i) dueling feedback, where pairwise comparisons between model responses provide robust preference signals, and (ii) heterogeneous sampling costs, reflecting the differing costs of querying different LLMs. Assuming the existence of a Condorcet winner, a condition we empirically validate across multiple real-world datasets, we propose a Track-and-Stop style algorithm for best-arm identification with prescribed confidence. We prove that the algorithm almost surely achieves the asymptotically optimal cost as the error tends to zero. Finally, we extensively evaluate our approach on both synthetic and real-world instances, demonstrating consistent improvements over classical cost-unaware algorithms and their cost-aware extensions.
Centre for Machine Intelligence and Data Science (C-MInDS) Indian Institute of Technology Bombay · Department of Electrical Engineering Indian Institute of Technology Bombay