Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

Bin Li; Dongdong Wang; Siyang Lu

cs.LGAug 18, 2026

Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

Authors: Bin Li, Dongdong Wang, Siyang Lu

Organizations: School of Cyberspace Science and Technology, Beijing Jiaotong University, Beijing, China · University of Florida, Gainesville, FL, USA · School of Computer and Technology, Beijing Jiaotong University, Beijing, China

Abstract

Online log anomaly detection is critical for maintaining the reliability of large-scale computing systems. Although recent language model-based log anomaly detectors achieve strong detection performance, their confidence estimates remain poorly calibrated. We show that these detectors frequently assign excessive confidence to incorrect predictions, particularly for anomalous logs under severe class imbalance. Moreover, confidence on erroneous predictions remains persistently high even when conventional calibration metrics indicate good calibration, creating a critical reliability gap for operational monitoring systems. To address this issue, we propose Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework for reliable log anomaly detection. LoRD learns prediction-route-specific reliability models from latent representations of correctly classified validation samples and estimates prediction reliability through route-wise reconstruction distances. Based on the estimated reliability, LoRD selectively recalibrates high-risk predictions to suppress overconfident errors while preserving reliable predictions. Extensive experiments on four large-scale log benchmark datasets and multiple language model-based detectors demonstrate that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing anomaly detection performance.

Figures & tables

Fig. 1: Overview of existing log anomaly detection frameworks. Logs are preprocessed, embedded into semantic representations, encoded using sequential models such as Bi-LSTM or Transformer, and finally used for anomaly prediction.

Fig. 4: Overview of LoRD with AE. Correctly classified validation samples are separated by prediction route to train route-specific AE. Reliability distances are then used to partition predictions into route-aware regions, which are subsequently mapped to confidence calibration policies.

Route	$\mathcal{R}_{r}^{\mathrm{low}}$	$\mathcal{R}_{r}^{\mathrm{mid}}$	$\mathcal{R}_{r}^{\mathrm{high}}$
Normal $(r=0)$	HardAssign: $p=1$	Identity	SoftPull: $p\rightarrow 0.5+\epsilon$
Abnormal $(r=1)$	SoftPull: $p\rightarrow 1$	Identity	HardAssign: $p=0.5+\epsilon$

TABLE I: Route-region calibration policy map

\pi

Dataset	Model	Uncal	TempS	LogS	BetaS	SeleS	Ens.	LoRD
BGL	TextCNN	$0.977_{\scriptstyle\pm 0.003}$	$0.976_{\scriptstyle\pm 0.004}$	$0.988_{\scriptstyle\pm 0.002}$	$0.989_{\scriptstyle\pm 0.003}$	$0.968_{\scriptstyle\pm 0.005}$	$0.966_{\scriptstyle\pm 0.005}$	$0.540_{\scriptstyle\pm 0.006}$
	LogRobust	$0.987_{\scriptstyle\pm 0.009}$	$0.973_{\scriptstyle\pm 0.016}$	$0.988_{\scriptstyle\pm 0.004}$	$0.988_{\scriptstyle\pm 0.003}$	$0.907_{\scriptstyle\pm 0.098}$	$0.748_{\scriptstyle\pm 0.163}$	$0.577_{\scriptstyle\pm 0.092}$
	LightLog	$0.829_{\scriptstyle\pm 0.022}$	$0.983_{\scriptstyle\pm 0.007}$	$0.967_{\scriptstyle\pm 0.009}$	$0.979_{\scriptstyle\pm 0.004}$	$0.831_{\scriptstyle\pm 0.015}$	$0.998_{\scriptstyle\pm 0.000}$	$0.702_{\scriptstyle\pm 0.036}$
	NeuralLog	$0.990_{\scriptstyle\pm 0.002}$	$0.975_{\scriptstyle\pm 0.007}$	$0.974_{\scriptstyle\pm 0.007}$	$0.988_{\scriptstyle\pm 0.001}$	$0.847_{\scriptstyle\pm 0.016}$	$0.985_{\scriptstyle\pm 0.001}$	$0.509_{\scriptstyle\pm 0.002}$
	GPT-2	$0.998_{\scriptstyle\pm 0.000}$	$0.992_{\scriptstyle\pm 0.000}$	$0.993_{\scriptstyle\pm 0.000}$	$0.991_{\scriptstyle\pm 0.000}$	$1.000_{\scriptstyle\pm 0.000}$	$0.998_{\scriptstyle\pm 0.000}$	$0.566_{\scriptstyle\pm 0.002}$
Spirit	TextCNN	$0.890_{\scriptstyle\pm 0.055}$	$0.862_{\scriptstyle\pm 0.051}$	$0.776_{\scriptstyle\pm 0.091}$	$0.833_{\scriptstyle\pm 0.056}$	$0.858_{\scriptstyle\pm 0.039}$	$0.926_{\scriptstyle\pm 0.009}$	$0.556_{\scriptstyle\pm 0.022}$

TABLE II: Calibration performance measured by CoE on anomalous log detection across benchmark datasets and representative detectors. Reported values denote mean CoE with standard deviation over multiple runs, where lower values indicate better performance. The best and second-best results in each row are highlighted in bold and underlined italics.

Detector	Architecture	BGL		Spirit		Liberty
		Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$
LogRobust	SingleAE	$0.562_{\scriptstyle\pm 0.066}$	$0.582_{\scriptstyle\pm 0.038}$	$0.556_{\scriptstyle\pm 0.041}$	$0.516_{\scriptstyle\pm 0.011}$	$0.740_{\scriptstyle\pm 0.049}$	$0.506_{\scriptstyle\pm 0.002}$
	Route-0-Only	$0.993_{\scriptstyle\pm 0.005}$	$0.542_{\scriptstyle\pm 0.002}$	$0.999_{\scriptstyle\pm 0.001}$	$0.583_{\scriptstyle\pm 0.018}$	$0.999_{\scriptstyle\pm 0.001}$	$0.617_{\scriptstyle\pm 0.050}$
	LoRD	$0.990_{\scriptstyle\pm 0.003}$	$0.541_{\scriptstyle\pm 0.002}$	$0.999_{\scriptstyle\pm 0.000}$	$0.508_{\scriptstyle\pm 0.003}$	$0.986_{\scriptstyle\pm 0.008}$	$0.502_{\scriptstyle\pm 0.001}$
NeuralLog	SingleAE	$0.549_{\scriptstyle\pm 0.052}$	$0.610_{\scriptstyle\pm 0.023}$	$0.647_{\scriptstyle\pm 0.111}$	$0.543_{\scriptstyle\pm 0.059}$	$0.783_{\scriptstyle\pm 0.033}$	$0.788_{\scriptstyle\pm 0.200}$
	Route-0-Only	$0.998_{\scriptstyle\pm 0.001}$	$0.506_{\scriptstyle\pm 0.002}$	$0.998_{\scriptstyle\pm 0.001}$	$0.575_{\scriptstyle\pm 0.059}$	$1.000_{\scriptstyle\pm 0.000}$	$0.716_{\scriptstyle\pm 0.044}$

TABLE III: Ablation study of LoRD on BGL, Spirit, and Liberty with LogRobust and NeuralLog as base detectors. Abn. CoC and Abn. CoE denote the average confidence on correctly and incorrectly classified abnormal samples, respectively.

	Spirit						Liberty
Method	Nor. CoC $\uparrow$	Nor. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	$D\downarrow$	$C\uparrow$	Nor. CoC $\uparrow$	Nor. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	$D\downarrow$	$C\uparrow$
w/o Reject	0.899	0.957	1.000	0.504	0.468	0.754	0.856	0.922	0.994	0.501	0.446	0.759
w/o Soft	0.908	0.905	1.000	0.503	0.415	0.776	0.879	0.843	0.988	0.501	0.364	0.796
LoRD	0.942	0.901	1.000	0.509	0.405	0.783	0.916	0.812	0.986	0.502	0.323	0.817

TABLE IV: Ablation study of LoRD on Spirit and Liberty with LogRobust as the base detector. CoC and CoE denote the average confidence on correctly and incorrectly classified samples, respectively. Lower

D

and higher

C

indicate better overall calibration quality.

	Spirit				Liberty
Method	CoC $\uparrow$	CoE $\downarrow$	$D\downarrow$	$C\uparrow$	CoC $\uparrow$	CoE $\downarrow$	$D\downarrow$	$C\uparrow$
w/o Reject	1.000	0.507	0.508	0.736	1.000	0.572	0.450	0.757
w/o Soft	1.000	0.507	0.506	0.741	1.000	0.602	0.395	0.790
LoRD	1.000	0.513	0.495	0.747	1.000	0.680	0.354	0.814

TABLE V: Component ablation of LoRD using NeuralLog as the base detector. CoC and CoE denote the confidence on correctly and incorrectly classified abnormal samples, respectively. “w/o Reject” and “w/o Soft” remove the reject region and SoftPull calibration, respectively. Lower

D

and higher

C

indicate better overall confidence reliability.

Explore similar work

LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection
Jun 6, 2026David Eje, Tanmay Sharma, Khush Patel +2Interpretable Anomaly Detection Openai
Department of Computer Science, Innopolis University, Innopolis, Russia
FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection
May 21, 2026Huanchi Wang, Zihang Huang, Yifang Tian +3Interpretable Anomaly Detection Early Failure Prediction
Department of Electrical and Computer Engineering University of Toronto, Toronto, Ontario
Seeing the Needle in the Haystack: Towards Weakly-Supervised Log Instance Anomaly Localization via Counterfactual Perturbation
May 9, 2026Yutszyuk Wong, Wentai Wu, Yuen-Ying Yeung +1Interpretable Anomaly Detection
Jinan University · Guangzhou, China · South China University of Technology

Jun 6, 2026, cs.LGJ/K move · Enter open · S save

LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection

David Eje, Tanmay Sharma, Khush Patel, Manuel Mazzara+1

Department of Computer Science, Innopolis University, Innopolis, Russia

Interpretable Anomaly Detection Openai Offline Reinforcement Learning

Open paper AskPDF arXiv

Detecting anomalies in large-scale system logs is critical for the reliability and security of modern computing infrastructure. We present LogNEO, a log anomaly detector built on EleutherAI's GPT-Neo (1.3B parameters) and fine-tuned with a novel partial-credit, exponentially decaying position-aware reward scheme combined with cross-entropy regularisation via Proximal Policy Optimisation (PPO). The position-aware reward explicitly models prediction difficulty: early positions receive higher rewards for correct predictions, while later positions incur stronger penalties for errors. LogNEO attains F1-scores of 0.927, 0.913, and 0.984 on the HDFS, BGL, and Thunderbird benchmarks, improving recall by up to 6 percentage points over the prior state-of-the-art LogGPT while maintaining comparable precision. A production microservice deployment over Apache Kafka, Redis, and TensorRT-accelerated inference demonstrates 45 ms end-to-end latency at 15,000 events per second.

Abstract

Explore similar work

LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection

FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection

Seeing the Needle in the Haystack: Towards Weakly-Supervised Log Instance Anomaly Localization via Counterfactual Perturbation