Maximizing LLM Utility While Managing LLM Fallibility
Organizations: E. U. Samani and C. G. Atkeson are with the Robotics Institute at Carnegie Mellon University
Abstract
Large language models (LLMs) can supply autonomous systems with plans, predictions, and judgments, but these contributions are sometimes wrong. This review asks how a system can maximize an LLM's utility while managing its fallibility. Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates. Systems built from imperfect components, such as noisy sensors and faulty software, have long worked in this way, and the classical methods they use can be applied to LLM-based systems as well. A system can change its reliance on an LLM in several ways, depending on the LLM's role and on the other components in the system. Each change needs an estimate of the LLM's reliability, and the LLM's role and the kind of contribution it makes determine what to estimate. Classical methods from decision theory, estimation, and online learning then turn these estimates into reliance decisions. These decisions can reduce how often LLM errors affect the system. However, some errors matter even when they are rare. An independently verified mechanism can prevent such errors from being executed. Without one, adapting reliance using reliability estimates can still keep their risk within a bound. Studies on adapting reliance on LLMs are emerging but limited, and the reviewed comparisons suggest that adapting reliance can improve system performance.
Figures & tables
| Role | What the LLM contributes | Example |
|---|---|---|
| Formulator / architect | An objective, reward, specification, heuristic, or code | An LLM writes reward functions for learning [ 30 ] , planning specifications [ 31 ] , and policy and planning code [ 32 , 33 ] . |
| Candidate proposer | A candidate design, configuration, hypothesis, or solution | An LLM proposes batches of monomer sequences [ 34 ] . |
| Search shaper | A prior, constraint, ranking, region, or other guidance for a search | An LLM guides Bayesian optimization of 3D print settings [ 35 ] . |
| Consequence modeler | A prediction of an outcome, property, feasibility, or uncertainty before it is measured | An LLM predicts the electrical conductivity of polymer composite films [ 18 ] . |
| Evaluator | A score, judgment, critique, or check of a candidate, output, plan, or state | An LLM judges whether plans for rearranging blocks are valid [ 12 ] . |
| Selector / planner | A choice of candidate, action, skill, tool, plan, or branch | An LLM generates plans for rearranging blocks [ 12 ] or scores candidate actions for robot manipulation [ 36 ] . |
| Dimension | Decision a system makes about using the LLM | What the system must have to change reliance along this dimension |
|---|---|---|
| Assignment | Whether the LLM or another source supplies the contribution | Another source for the needed contribution |
| Influence | How much weight the LLM’s contribution gets | Other information to combine with the LLM contribution |
| Invocation | When to ask the LLM for a new contribution | A way to continue without a new LLM contribution |
| Oversight | Whether to check the LLM’s contribution before using it | A way to check the LLM’s contribution |
| Persistence | Whether an LLM contribution already in use stays in use | An LLM contribution in use and a way to continue without it |
| Latitude | Which options the LLM is allowed to choose from | A way to enforce permitted choices independently of the LLM |
| Functional role | Assignment | Influence | Invocation | Oversight | Persistence | Latitude |
|---|---|---|---|---|---|---|
| Formulator / architect | Have a person or a library supply the formulation, or part of it | Weight LLM-written terms against hand-written ones, such as reward terms | Request a revised formulation after testing the current one | Test the formulation before adopting it, such as by running the code or training a policy [ 30 ] | Keep or replace an adopted formulation as test evidence comes in | Restrict changes to the formulation, such as by allowing edits only to the weights of fixed terms |
| Candidate proposer | Have another generator supply candidates, such as through mutation or random sampling | Combine LLM-proposed candidates with others, such as by recombination, when their representations allow it | Request new candidates when other proposal operators stop improving the search | Screen candidates before use, such as in simulation or with a realizability checker [ 40 ] | Keep candidates in the active search or discard them as test results arrive | Restrict the candidate space, such as through parameter bounds |
| Search shaper | Obtain guidance from a heuristic or a person instead | Weight guidance against search evidence, such as measured objective values [ 19 ] | Request guidance when the search stalls, sooner after useful suggestions [ 17 ] | Test guidance before committing search budget, such as by sampling a suggested region | Keep or retire guidance as measured search gains change | Restrict guidance, such as by limiting the search space it may exclude |
| Consequence modeler | Replace the prediction with other outcome information, such as a measurement | Combine predictions with other outcome information, such as surrogate estimates or measurements [ 18 ] | Request a prediction when outcome uncertainty could change the next decision | Check predictions against observed outcomes, such as experimental measurements [ 18 ] | Keep or retire predictions still guiding decisions as measurements arrive | Limit predictions to a permitted range, such as physically plausible values |
| Evaluator | Have another judge, a validator, or a person supply the evaluation | Combine evaluations from several judges [ 41 ] | Request an evaluation when cheaper checks leave a candidate unresolved | Spot-check evaluations against a reference, such as human labels | Keep or retire evaluations still guiding decisions as reference checks arrive | Restrict evaluation outputs, such as the permitted score range |
| Selector / planner | Have another planner or a person make the choice or plan [ 36 ] | Weight choices against task feasibility, such as skill affordance estimates [ 10 ] | Request a new choice or plan when execution feedback warrants replanning | Check choices or plans before execution, such as with a symbolic validator [ 12 ] | Keep executing a plan or stop partway, such as by committing to only its first steps | Restrict choices or plans, such as by limiting the available actions or skills |
| Functional role, contribution interface, and example property | Reliability question and quantity | How to estimate the quantity |
|---|---|---|
| Formulator / architect Specification Example property: fit to the intended requirement | How well does the specification express the intended requirement? Quantities: counts of omissions and detected contradictions, share of specification statements traced to requirement sources | Trace each specification statement to a requirement source. Then check which requirements the specification, under the domain assumptions, omits or contradicts, as in requirements engineering [ 43 ] . |
| Candidate proposer Candidate set Example property: quality and variety of the set | How many good candidates does the set contain, and how novel and diverse are they? Quantities: yield, quality, novelty, diversity, coverage under a common evaluator and budget | Score all candidates, from the LLM and other sources, with the same evaluator and budget. Measure novelty and coverage against a reference set, using ideation or quality-diversity metrics [ 44 , 45 ] . |
| Search shaper Suggested points or regions Example property: improvement on the best value found so far | How often do suggested points, or points sampled from a suggested region, improve on the best value found so far? Quantity: fraction of these points that improve on that value | Evaluate the objective at each suggested point, or at points sampled uniformly from each suggested region. Count those that improve on the best value. |
| Consequence modeler Dynamic model Example property: prediction accuracy in the operating region | How accurately does the model predict new input–output data in the operating region? Quantities: one-step and multistep prediction error | Compare the model’s predictions with observed outcomes on new input–output data, at the horizon at which the model will be used, as in system identification [ 46 ] . |
| Consequence modeler Simulator or surrogate Example property: agreement with observations | How does the model compare with observations? Quantities: prediction error, model discrepancy | Pair the model’s predictions with observations of the same inputs, and model the discrepancy between them [ 47 ] . Account for observation error, and state the assumptions used to separate it from the discrepancy. |
| Consequence modeler Probability forecast Example property: accuracy and concentration of forecasts | How well calibrated are the probabilities, and how concentrated and accurate are the forecasts? Quantities: calibration, sharpness, proper scores, which answer different questions [ 48 , 49 ] | Record each forecast before its outcome resolves. Measure sharpness from the forecasts alone. Once outcomes resolve, check calibration and score each forecast with a proper scoring rule [ 50 ] . |
| Dimension | Classical method | How to apply it to an LLM | Related LLM system |
| Assignment | Algorithm selection assigns each input to the source predicted to perform best, given the input’s features [ 56 ] . | Test every source on inputs that resemble those met in use. Estimate each source’s reliability on each type of input with the same criterion, such as its valid-plan rate per problem type. Send each new input to the source with the highest estimate for its type. | RouteLLM learns a router from offline preferences [ 57 ] . |
| A contextual bandit learns which source works best in each context [ 8 , 58 ] . | Record each input’s context, the chosen source, and its reward, such as task success or plan validity. Update each source’s expected reward for that context, including rewards that arrive only after a costly check, such as a physical experiment. Choose the source for each new input with the bandit, which keeps trying alternatives. | CABS-D routes each query to one LLM, using selected rewards and surrogate information [ 20 ] . | |
| A discounted bandit learns which source works best now by giving older outcomes less weight [ 9 ] . | Record each chosen source’s reward. If an LLM’s reliability may shift, such as after a model update or a move to a new machine, give older rewards less weight. Keep sending some inputs to every source. Route each input by the current estimates. | ParetoBandit routes each query to one LLM and gives older rewards less weight, so it can track shifts in LLM quality and price. It was tested offline, on precomputed rewards with injected shifts [ 59 ] . | |
| Adjustable autonomy assigns each decision to the automation or to a person, counting the cost of transferring control [ 60 , 61 ] . | Calibrate the LLM’s confidence in its choice on inputs that resemble those met in use. Also estimate the person’s decision quality. Hand the decision to the person when their expected improvement over the LLM’s choice outweighs the cost and delay of transferring control. | KnowNo asks the user whenever more than one action qualifies [ 36 ] . | |
| Influence | Weighted Majority and Hedge weight each source’s vote by its accumulated loss [ 62 , 63 ] . | Record each source’s error on the same inputs, such as each judge’s error rates. Give less weight to sources with higher accumulated loss. Check these inputs for errors shared by every source. If the system executes the combined choice, act on a single source’s choice for some inputs to observe each source’s own loss. | CALIPER sets each LLM expert’s weight from its discounted errors on measured objectives [ 19 ] . |
| Kalman or Bayesian fusion weights the sources by the estimated uncertainty and correlation of their errors [ 64 ] . | Estimate the bias and uncertainty of the LLM’s predictions against measurements. Model the correlation between its errors and those of the other sources. Weight each source by the resulting error model. Update the error model as new measurements arrive, and reweight the sources. If the LLM sometimes errs by much more than usual, give such errors less weight with a robust filter [ 65 ] . | LABO models a measured outcome as a scaled LLM prediction plus a discrepancy [ 18 ] . |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 866 papers