Language Model Calibration

Follow this topic Ask about this topic

Momentum

43 papers in the last four weeks, up 330% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 247

All topics

Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier
Sep 24, 2026Adam Górski, Mateusz Jąkalak, Rafał JakubowskiMulti-Label Classification Language Model Safety Evaluation
Billennium S.A.
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Sep 23, 2026Makar Ulesov, Vladislav Smirnov, Omar Ibrahim +1Benchmark Auditing LLM Auditing
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · School of Computing and Information, University of Pittsburgh
Calibration as a First-Class Criterion in LLM Evaluation
Sep 22, 2026Mario Sanz-Guerrero, Katharina von der WenseLLM Evaluation Confidence Estimation in Language Models
Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA
How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification
Sep 22, 2026Sophie Henning, Georg Hofmann, Alexander Schulte +2Multi-Label Classification Language Model Calibration
TU Munich · Munich Center for Machine Learning (MCML) · LMU Munich +3
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Sep 21, 2026Kevin David Hayes, Arka Pal, Haosong Zhang +2Language Model Error Detection Confidence Estimation in Language Models
Department of Computer Science, University of Maryland · Ritual AI · Fudan University +1
Evaluating Decision Models for Text Annotation in Computational Social Science
Sep 21, 2026Hazem Ibrahim, Yasir ZakiLLM Evaluation LLM-Assisted Annotation
Computer Science, New York University Abu Dhabi, Abu Dhabi, UAE
SupportCal: Label-Free Calibration of Post-Trained LLMs via Reference Support and Corroboration
Sep 21, 2026Linhan Luo, Lequan Lin, Dai Shi +3Post-Hoc Calibration Language Model Calibration
The University of Sydney · University of Cambridge · The University of Adelaide
OSCAR: Order-aware Scoring and Calibration for AI Rankings
Sep 21, 2026You Liu, Yue Liu, Quanchao Lu +1LLM Evaluation LLM-as-a-Judge
Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)
Sep 21, 2026Amir Rafe, Subasish DasLLM Auditing Language Model Calibration
Ingram School of Engineering, Texas State University, San Marcos, TX, USA
Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model
Sep 21, 2026Simiao Ren, Kidus Zewde, Xingyu Shen +6LLM Inference Efficiency LLM Evaluation
Scam.ai (Reality Inc.)
The Missing "I Don't Know": Why Three Reasoning-Reliability Findings Converge on Calibrated Abstention
Sep 15, 2026Srijith RavikumarLanguage Model Safety Evaluation Language Model Calibration
Amazon.com LLC
Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
Sep 15, 2026Haichen Hu, Yuheng Zhang, David Simchi-LeviLanguage Model Calibration Language Model Distillation
MIT · UIUC
Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML
Sep 14, 2026Sidi Chang, Peiying ZhuLLM-as-a-Judge LLM Auditing
Blossom AI San Francisco, CA, USA
R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration
Sep 14, 2026Dhruv Dixit, Paritosh PandeyRetrieval-Augmented Generation Automated Fact-Checking
Department of Electrical and Computer Engineering Stevens Institute of Technology Hoboken, New Jersey, United States · Department of Computer Science University of North Carolina Chapel Hill, North Carolina, United States
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization
Sep 13, 2026Sarah Wilson, Gail Kaiser, Patrick MusauSelective Prediction Large Language Model-Guided Optimization
Google · Columbia University · Vanderbilt University
TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps
Sep 13, 2026Rohit Patel, Susil Kumar Mohanty, Jeenal ChaudharyLLM Inference Efficiency Retrieval-Augmented Generation
Department of Computer Science and Engineering Indian Institute of Technology Jodhpur, Jodhpur, India
Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
Sep 11, 2026Yu-Chung HsiaoLLM Evaluation LLM-as-a-Judge
Cisco Systems
ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification
Sep 9, 2026Jianzong Wang, Chuhang Liu, Botao Zhao +6Confidence Estimation in Language Models Language Model Calibration
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · Tsinghua Shenzhen International Graduate School, Shenzhen, China · Hubei University of Technology, Hubei, China
Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format
Sep 9, 2026Touchapon Kraisingkorn, Krittin Pachtrachai, Wachiravit ModecruaLLM Evaluation Language Model Calibration
Amity Research and Application Center (ARAC), Amity Group
Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
Sep 1, 2026Kangjia Zhao, Jiajun Li, Haozhan Shen +8Tool-Use Evaluation LLM Agent Evaluation
Zhejiang University · Shanghai Jiao Tong University · National University of Singapore +3
DualStake: Dual-Path Confidence Calibration in Deep Research Agents
Sep 1, 2026Yinuo Xu, Yuwei Liang, Jianjie Cheng +4Deep Research Agents Language Model Calibration
NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences · Meituan Inc.
Self-Reports Are Not Verification: Environment-Grounded Auditing of LLM Operators in Evolutionary Search
Sep 1, 2026Enrong Pan, Ryan Zhou, Ting HuLLM Auditing Language Model Self-Assessment
Queen’s University
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Aug 31, 2026Qiyao Yan, Chenpeng Wang, Liangming PanLLM Evaluation Language Model Calibration
State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · YiXin-AILab, YIXIN, Beijing, China +1
Calibrating Small Language Models for Claim Check-Worthiness Detection
Aug 31, 2026Pratuat Amatya, V Venktesh, Vinay SettyClaim Verification Small Language Models
1Factiverse AS · University of Stavanger · 3Stockholm University
CoMPASS: Collaborative Molecular Property Prediction via Adaptive Small-Large Model Synergy
Aug 31, 2026Wentao Li, Jiangjie Qiu, Yijun Li +2Graph Neural Networks Language Model Calibration
Beijing Key Laboratory of Artificial Intelligence for Advanced Chemical Engineering Materials · State Key Laboratory of Chemical Engineering and Low-Carbon Technology · Department of Chemical Engineering, Tsinghua University
Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection
Aug 18, 2026Bin Li, Dongdong Wang, Siyang LuPost-Hoc Calibration Online Anomaly Detection
School of Cyberspace Science and Technology, Beijing Jiaotong University, Beijing, China · University of Florida, Gainesville, FL, USA · School of Computer and Technology, Beijing Jiaotong University, Beijing, China
Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity
Aug 13, 2026Irina Proskurina, Mayank Kumar, Oyindolapo O. KomolafeConfidence Estimation in Language Models Question Answering
Cohere Labs Community · Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · School of Computer Science Engineering and Technology (SCSET), Bennett University, Greater Noida, India +1
ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral
Aug 11, 2026Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin +1Selective Prediction Clinical Decision Support
Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
Aug 11, 2026Xinzhe Huang, Biwu Yao, Kedong Xiu +4LLM Guardrails Language Model Calibration
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China +3

Sep 24, 2026, cs.AIJ/K move · Enter open · S save

Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

Adam Górski, Mateusz Jąkalak, Rafał Jakubowski

Billennium S.A.

Multi-Label Classification Language Model Safety Evaluation OOD Generalization Content Moderation Language Model Calibration Safety Filtering

Open paper AskPDF arXiv

We develop a multi-label Polish content-safety classifier by fine-tuning allegro/herbert-base-cased (124M) across five categories (hate, vulgarity, sexual content, crime, self-harm) using a Focal + R-Drop objective, and evaluate the resulting model against Bielik Guard (Sójka) on the shared out-of-distribution Gadzi Język benchmark. Both systems are given per-category threshold tuning on the same calibration split. Under that matched protocol our model holds a small but statistically significant lead in micro F1, while an apparent macro-F1 lead does not survive: it was an artifact of comparing a tuned model against an untuned one. We also report what that micro figure is worth. Because Gadzi Język is 97% crime-positive, a classifier that flags crime on every input and nothing else already scores 0.910 micro F1 on the same test split, so micro separates neither system from a degenerate strategy and macro is the column that does. Per-category and per-protocol figures are reported in Section 4. The residual out-of-distribution gap is one of calibration rather than discrimination. Ranking quality stays high while positive probabilities collapse, and per-category temperature scaling recovers the loss where Platt scaling and isotonic regression do not. That recovery turns out to be conditional on the calibration set containing safe text. Gadzi Język contains almost none, so thresholds fitted on it flag crime on every safe input, and a balanced refit buys a deployable operating point at the cost of adversarial recall. We report both operating points rather than only the flattering one. Two changes that are standard practice, per-class cost-sensitive weighting and mean pooling, each raise in-distribution macro F1 while lowering the out-of-distribution figure, which indicates that robustness has to be selected for directly rather than inherited from in-distribution accuracy.

Momentum

Related topics

Latest papers 247