Language Model Calibration

Language model calibration asks whether a model’s stated or computed confidence corresponds to how often its answers are correct, so systems can decide when to answer, retrieve evidence, defer, or abstain. Researchers estimate confidence from verbal reports, token probabilities, self-consistency, or hidden representations, then improve it with post-hoc methods such as temperature scaling and isotonic regression, conformal prediction, or training methods including supervised fine-tuning and reinforcement learning. Evaluations commonly use question answering and mathematical reasoning benchmarks, and increasingly test retrieval routing, model judges, medical tasks, and performance under domain or prompt shifts. A central tension is that calibration can change with the elicitation protocol and deployment context, and well-calibrated probabilities do not necessarily rank errors well or guarantee safe decisions.