Authors: Hadi Khalaf, Sajani Vithana, Kunal Talreja, Haewon Jeong, Viveck R. Cadambe, Flavio Calmon
Organizations: Harvard School of Engineering and Applied Science · Kempner Institute for the Study of Natural & Artificial Intelligence · School of Electrical and Computer Engineering, Georgia Institute of Technology · Department of Electrical and Computer Engineering, University of California, Santa Barbara · Center for Computational Mathematics, Flatiron Institute
Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost-accuracy frontier over routing and cascading baselines.
Figures & tables
Figure 1: Classification circuits and their cost–accuracy tradeoffs. Majority voting, routing, and cascades are examples of circuits, differing in how they select and combine LLM classifiers. For any prompt, our method searches over a broad class of circuits and uses a dynamic program to determine which classifier to query next and when to stop. The right panel shows such circuits for the prompt z : starting from its embedding, the circuit may predict immediately (as in A ) or sequentially acquire additional LLM measurements as needed (as in B and C ). Across prompts, these decisions produce the cost–accuracy frontier for the BeaverTails safety dataset.
Figure 2: Illustration of a few known circuits within the framework of Definition 2 .
Figure 3: Generating circuits. We first fit a probabilistic model from labeled prompts and instrument measurements ( Section 3.2 ). At inference time, a prompt embedding is passed through this model, and a dynamic program generates a circuit that adaptively queries instruments and decides when to stop ( Equation 7 ).
Figure 4: LMCircuit improves the cost-accuracy frontier over routing and cascading baselines. As λ decreases and additional queries become less costly, the circuits make more calls and also change which instruments they use (see lower panel). The allocations differ substantially across tasks, indicating that the value of an instrument depends on the classification problem and on other measurements available to the circuit.
Figure 5: LMCircuit is substantially faster to construct than most routing and cascading baselines. Across all seven tasks, LMCircuit is trained in 1–3 minutes, while some baselines require minutes to hours.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The circuit’s beliefs are calibrated at every budget. For each task, left: Brier score on the test set ( lower is better ) against expected cost per item; right: reliability diagrams of the circuits at different budgets and of majority voting. The dashed line is perfect calibration. The circuit’s expected calibration error is at most 2.1% at every budget on six tasks, and at most 5.2% on BIG-Bench Hard, whereas majority voting’s is 4.8% to 24.6% and its miscalibration differs in direction by task.
Figure 7: Instrument answers are independent within a prompt but correlated across prompts. (a) ϕ coefficient of two instruments’ answers within a prompt, averaged over prompts on which both instruments’ answers vary. (b) The same coefficient across prompts, pooling all calls. (c) Total correlation of the eight answers on each prompt ( Equation 43 ) against the 95th percentile of its permutation null. Red marks a rejection at the 5% level. Within a prompt the coefficients lie between −0.10 and 0.15 , while across prompts they reach 0.69 . Only one of the 20 prompts rejects joint independence, the rate expected by chance.
Figure 8: The fitted model predicts each instrument’s accuracy on unseen data without labels and recovers nearly all of the dependence between instruments. Correlations and accuracies implied by the fitted model on unlabeled test prompts ( Equation 44 ) against those measured from recorded calls and labels. Left: correlation of two instruments’ answers across prompts (one point per pair). Middle: correlation of their errors. Right: accuracy of each instrument. The dashed line is equality.
Figure 9: Additional queries improve accuracy when instrument errors vary across calls. We stratify unseen prompts by r(z) and report accuracy for four predictors: the embedding prediction ηθ(ϕ(z),s0) , which makes no instrument calls; our circuit at normalized budgets of 0.03 and 1 ; and majority voting over the instruments the circuit can query. Numbers above the bars give the mean number of calls per prompt, and percentages below each group give its share of the test set. Relative to the embedding prediction, the circuit at budget 1 improves accuracy on noisy prompts by 5–44 percentage points. On all right prompts, it comes within 1.3 percentage points of majority voting on four tasks while using 38–86% fewer calls.
Figure 10: The circuit allocates more queries when the prior prediction and instrument measurements disagree. We present the expected number of calls per test prompt for circuits at normalized budgets 0.03 , 0.1 , and 0.2 on Arena, BeaverTails and PolitiCAUSE, grouped by prompt type (left), instrument disagreement d(z) (middle: 0 , (0,1/3] , (1/3,2/3] , and (2/3,1] ), and whether the embedding prediction agrees with the majority of the recorded calls (right). Percentages below each group give its share of the test set. Mean calls increase from all right to mostly wrong prompts on every task. The clearest separation occurs when conditioning on agreement with the embedding prediction: at budget 0.2 , the circuit makes 1.7 – 1.9× as many calls when the embedding and instrument majority disagree on Arena ( 1.84 vs. 1.08 ), BeaverTails ( 1.77 vs. 0.95 ) and PolitiCAUSE ( 1.85 vs. 1.11 ).
Figure 11: Instruments with higher information per unit cost enter the circuit earlier. Expected calls per unseen prompt to each instrument as a function of the circuit’s expected cost per item. Instruments are colored by their rank under the estimated information rate r^i in Equation 45 , with the corresponding value in bits per unit cost shown in parentheses. Across the three tasks, instruments with larger r^i tend to begin receiving calls at lower expected cost. The highest-rate instrument enters first on every task, including Arena, where Gemma 26B is not the cheapest instrument.
Figure 12: Building the instrument pool. We build the instrument pool one instrument at a time according to different strategies: by estimated conditional information given the embedding and the pool (greedy algorithm), by accuracy on the training calls (most accurate first), cheapest first, or in random order (mean of three runs). Every point is our circuit restricted to that pool, using a single fitted model (an earlier version of the model without latent types). The circled point is the circuit with all instruments. Top: frontier area of test accuracy. Middle: test accuracy when every instrument in the pool is queried ( λ=0 ). Bottom: the upper bound I/2 of Proposition 4 on the drop in Bayes error from adding each instrument ( I its estimated conditional information, in nats), and the drop in the model’s Bayes error E[min{η,1−η}] that the instrument’s answers actually produce, both on test.
Figure 13: Example of generated circuit and instrument’s error correlations on BeaverTails
Figure 14: Example of generated circuit and instrument’s error correlations on LMArena
Deploying multiple large language models (LLMs) in parallel to classify an unknown ground-truth label is a common practice, yet the problem of optimally allocating queries across heterogeneous models remains poorly understood. In this paper, we formulate a robust, offline query-planning problem that minimizes total query cost subject to statewise error constraints which guarantee reliability for every possible ground-truth label. We first establish that this problem is NP-hard via a reduction from the minimum-weight set cover problem. To overcome this intractability, we develop a surrogate by combining a union bound decomposition of the multi-class error into pairwise comparisons with Chernoff-type concentration bounds. The resulting surrogate admits a closed-form, multiplicatively separable expression in the query counts and is guaranteed to be feasibility-preserving. We further show that the surrogate is asymptotically tight at the optimization level: the ratio of surrogate-optimal cost to true optimal cost converges to one as error tolerances shrink, with an explicit rate of O(loglog(1/αmin)/log(1/αmin)). Finally, we design an asymptotic fully polynomial-time approximation scheme (AFPTAS) that returns a surrogate-feasible query plan within a (1+ε) factor of the surrogate optimum.
Arlen Dean, Zijin Zhang, Stefanus Jasin +1
Washington University in St. Louis, Olin Business School · Boston College, Carroll School of Management · University of Michigan, Stephen M. Ross School of Business +1
Every call to an LLM classification endpoint produces a labeled input-output pair already retained in production logs. These pairs constitute a free, growing training set: a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost. The open questions are when the surrogate is reliable enough to deploy, what it handles versus defers, and how that boundary evolves as data accumulates. We introduce TRACER (Trace-based Adaptive Cost-Efficient Routing), an open-source system that trains ML surrogates on an LLM's own production traces and governs deployment through a parity gate: the surrogate is activated only when its agreement with the LLM exceeds a user-specified threshold α. To make the routing boundary transparent, TRACER generates interpretability artifacts describing which input regions the surrogate handles, where it plateaus, and why it defers. On a 77-class intent benchmark with a Sonnet 4.6 teacher, TRACER achieves 83-100% surrogate coverage depending on the quality target α; on a 150-class benchmark, the surrogate fully replaces the teacher. On a natural language inference task, the parity gate correctly refuses deployment because the embedding representation cannot support reliable separation. The system is available as open-source software.
Task-fine-tuned large language model (LLM) classifiers acquire task-specific decision knowledge, but this knowledge remains implicit in distributed internal computations, making their decision logic difficult to interpret. We introduce the Executable Decision Compression (EDC) framework and propose J-Miner, which mines vocabulary-named variables from internal readouts and learns rules shared across inputs to produce executable explanations. Analysis reveals that a small set of these variables captures much of the classifier's decision behavior, holding for both varying parameter scales within a family and distinct families. Across six binary tasks, a rule using just one variable reproduces 76.7% of source-classifier decisions on average, rising to 88.8% with 16 variables. Most of the decision information retained by these variables comes from internal activations beyond literal surface matching. A lightweight text reader predicts the variable states, allowing the same fixed rules to execute independently of the source classifier.
Yunfan Gao, Xinyi Huang, Tao Sheng +3
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · Meituan +1