Confidence Estimation in Language Models

The field studies whether language models can tell when their answers are likely to be wrong, which matters for deciding when to answer, abstain, retrieve evidence, or spend more computation. Methods include verbalized confidence, token probabilities and sampling-based agreement, as well as hidden-state probes, activation signals, post-hoc calibration, and conformal prediction. These approaches are tested across settings such as factual question answering, mathematical reasoning, code generation, multilingual tasks, and tool-using agents. A central challenge is that confidence signals can disagree or shift with prompts, training, and distribution changes; good calibration or error ranking alone does not guarantee useful decisions or reliable risk guarantees.