Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

Bin Li; Dongdong Wang; Siyang Lu

cs.LGAug 18, 2026

Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

Authors: Bin Li, Dongdong Wang, Siyang Lu

Organizations: School of Cyberspace Science and Technology, Beijing Jiaotong University, Beijing, China · University of Florida, Gainesville, FL, USA · School of Computer and Technology, Beijing Jiaotong University, Beijing, China

Abstract

Online log anomaly detection is critical for maintaining the reliability of large-scale computing systems. Although recent language model-based log anomaly detectors achieve strong detection performance, their confidence estimates remain poorly calibrated. We show that these detectors frequently assign excessive confidence to incorrect predictions, particularly for anomalous logs under severe class imbalance. Moreover, confidence on erroneous predictions remains persistently high even when conventional calibration metrics indicate good calibration, creating a critical reliability gap for operational monitoring systems. To address this issue, we propose Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework for reliable log anomaly detection. LoRD learns prediction-route-specific reliability models from latent representations of correctly classified validation samples and estimates prediction reliability through route-wise reconstruction distances. Based on the estimated reliability, LoRD selectively recalibrates high-risk predictions to suppress overconfident errors while preserving reliable predictions. Extensive experiments on four large-scale log benchmark datasets and multiple language model-based detectors demonstrate that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing anomaly detection performance.

Figures & tables

Fig. 1: Overview of existing log anomaly detection frameworks. Logs are preprocessed, embedded into semantic representations, encoded using sequential models such as Bi-LSTM or Transformer, and finally used for anomaly prediction.

Fig. 4: Overview of LoRD with AE. Correctly classified validation samples are separated by prediction route to train route-specific AE. Reliability distances are then used to partition predictions into route-aware regions, which are subsequently mapped to confidence calibration policies.

Route	$\mathcal{R}_{r}^{\mathrm{low}}$	$\mathcal{R}_{r}^{\mathrm{mid}}$	$\mathcal{R}_{r}^{\mathrm{high}}$
Normal $(r=0)$	HardAssign: $p=1$	Identity	SoftPull: $p\rightarrow 0.5+\epsilon$
Abnormal $(r=1)$	SoftPull: $p\rightarrow 1$	Identity	HardAssign: $p=0.5+\epsilon$

TABLE I: Route-region calibration policy map

\pi

Dataset	Model	Uncal	TempS	LogS	BetaS	SeleS	Ens.	LoRD
BGL	TextCNN	$0.977_{\scriptstyle\pm 0.003}$	$0.976_{\scriptstyle\pm 0.004}$	$0.988_{\scriptstyle\pm 0.002}$	$0.989_{\scriptstyle\pm 0.003}$	$0.968_{\scriptstyle\pm 0.005}$	$0.966_{\scriptstyle\pm 0.005}$	$0.540_{\scriptstyle\pm 0.006}$
	LogRobust	$0.987_{\scriptstyle\pm 0.009}$	$0.973_{\scriptstyle\pm 0.016}$	$0.988_{\scriptstyle\pm 0.004}$	$0.988_{\scriptstyle\pm 0.003}$	$0.907_{\scriptstyle\pm 0.098}$	$0.748_{\scriptstyle\pm 0.163}$	$0.577_{\scriptstyle\pm 0.092}$
	LightLog	$0.829_{\scriptstyle\pm 0.022}$	$0.983_{\scriptstyle\pm 0.007}$	$0.967_{\scriptstyle\pm 0.009}$	$0.979_{\scriptstyle\pm 0.004}$	$0.831_{\scriptstyle\pm 0.015}$	$0.998_{\scriptstyle\pm 0.000}$	$0.702_{\scriptstyle\pm 0.036}$
	NeuralLog	$0.990_{\scriptstyle\pm 0.002}$	$0.975_{\scriptstyle\pm 0.007}$	$0.974_{\scriptstyle\pm 0.007}$	$0.988_{\scriptstyle\pm 0.001}$	$0.847_{\scriptstyle\pm 0.016}$	$0.985_{\scriptstyle\pm 0.001}$	$0.509_{\scriptstyle\pm 0.002}$
	GPT-2	$0.998_{\scriptstyle\pm 0.000}$	$0.992_{\scriptstyle\pm 0.000}$	$0.993_{\scriptstyle\pm 0.000}$	$0.991_{\scriptstyle\pm 0.000}$	$1.000_{\scriptstyle\pm 0.000}$	$0.998_{\scriptstyle\pm 0.000}$	$0.566_{\scriptstyle\pm 0.002}$
Spirit	TextCNN	$0.890_{\scriptstyle\pm 0.055}$	$0.862_{\scriptstyle\pm 0.051}$	$0.776_{\scriptstyle\pm 0.091}$	$0.833_{\scriptstyle\pm 0.056}$	$0.858_{\scriptstyle\pm 0.039}$	$0.926_{\scriptstyle\pm 0.009}$	$0.556_{\scriptstyle\pm 0.022}$

TABLE II: Calibration performance measured by CoE on anomalous log detection across benchmark datasets and representative detectors. Reported values denote mean CoE with standard deviation over multiple runs, where lower values indicate better performance. The best and second-best results in each row are highlighted in bold and underlined italics.

Detector	Architecture	BGL		Spirit		Liberty
		Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$
LogRobust	SingleAE	$0.562_{\scriptstyle\pm 0.066}$	$0.582_{\scriptstyle\pm 0.038}$	$0.556_{\scriptstyle\pm 0.041}$	$0.516_{\scriptstyle\pm 0.011}$	$0.740_{\scriptstyle\pm 0.049}$	$0.506_{\scriptstyle\pm 0.002}$
	Route-0-Only	$0.993_{\scriptstyle\pm 0.005}$	$0.542_{\scriptstyle\pm 0.002}$	$0.999_{\scriptstyle\pm 0.001}$	$0.583_{\scriptstyle\pm 0.018}$	$0.999_{\scriptstyle\pm 0.001}$	$0.617_{\scriptstyle\pm 0.050}$
	LoRD	$0.990_{\scriptstyle\pm 0.003}$	$0.541_{\scriptstyle\pm 0.002}$	$0.999_{\scriptstyle\pm 0.000}$	$0.508_{\scriptstyle\pm 0.003}$	$0.986_{\scriptstyle\pm 0.008}$	$0.502_{\scriptstyle\pm 0.001}$
NeuralLog	SingleAE	$0.549_{\scriptstyle\pm 0.052}$	$0.610_{\scriptstyle\pm 0.023}$	$0.647_{\scriptstyle\pm 0.111}$	$0.543_{\scriptstyle\pm 0.059}$	$0.783_{\scriptstyle\pm 0.033}$	$0.788_{\scriptstyle\pm 0.200}$
	Route-0-Only	$0.998_{\scriptstyle\pm 0.001}$	$0.506_{\scriptstyle\pm 0.002}$	$0.998_{\scriptstyle\pm 0.001}$	$0.575_{\scriptstyle\pm 0.059}$	$1.000_{\scriptstyle\pm 0.000}$	$0.716_{\scriptstyle\pm 0.044}$

TABLE III: Ablation study of LoRD on BGL, Spirit, and Liberty with LogRobust and NeuralLog as base detectors. Abn. CoC and Abn. CoE denote the average confidence on correctly and incorrectly classified abnormal samples, respectively.

	Spirit						Liberty
Method	Nor. CoC $\uparrow$	Nor. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	$D\downarrow$	$C\uparrow$	Nor. CoC $\uparrow$	Nor. CoE $\downarrow$	Abn. CoC $\uparrow$	Abn. CoE $\downarrow$	$D\downarrow$	$C\uparrow$
w/o Reject	0.899	0.957	1.000	0.504	0.468	0.754	0.856	0.922	0.994	0.501	0.446	0.759
w/o Soft	0.908	0.905	1.000	0.503	0.415	0.776	0.879	0.843	0.988	0.501	0.364	0.796
LoRD	0.942	0.901	1.000	0.509	0.405	0.783	0.916	0.812	0.986	0.502	0.323	0.817

TABLE IV: Ablation study of LoRD on Spirit and Liberty with LogRobust as the base detector. CoC and CoE denote the average confidence on correctly and incorrectly classified samples, respectively. Lower

D

and higher

C

indicate better overall calibration quality.

	Spirit				Liberty
Method	CoC $\uparrow$	CoE $\downarrow$	$D\downarrow$	$C\uparrow$	CoC $\uparrow$	CoE $\downarrow$	$D\downarrow$	$C\uparrow$
w/o Reject	1.000	0.507	0.508	0.736	1.000	0.572	0.450	0.757
w/o Soft	1.000	0.507	0.506	0.741	1.000	0.602	0.395	0.790
LoRD	1.000	0.513	0.495	0.747	1.000	0.680	0.354	0.814

TABLE V: Component ablation of LoRD using NeuralLog as the base detector. CoC and CoE denote the confidence on correctly and incorrectly classified abnormal samples, respectively. “w/o Reject” and “w/o Soft” remove the reject region and SoftPull calibration, respectively. Lower

D

and higher

C

indicate better overall confidence reliability.

Explore similar work

Jun 6, 2026cs.LG

LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection

Detecting anomalies in large-scale system logs is critical for the reliability and security of modern computing infrastructure. We present LogNEO, a log anomaly detector built on EleutherAI's GPT-Neo (1.3B parameters) and fine-tuned with a novel partial-credit, exponentially decaying position-aware reward scheme combined with cross-entropy regularisation via Proximal Policy Optimisation (PPO). The position-aware reward explicitly models prediction difficulty: early positions receive higher rewards for correct predictions, while later positions incur stronger penalties for errors. LogNEO attains F1-scores of 0.927, 0.913, and 0.984 on the HDFS, BGL, and Thunderbird benchmarks, improving recall by up to 6 percentage points over the prior state-of-the-art LogGPT while maintaining comparable precision. A production microservice deployment over Apache Kafka, Redis, and TensorRT-accelerated inference demonstrates 45 ms end-to-end latency at 15,000 events per second.

David Eje, Tanmay Sharma, Khush Patel +2

Department of Computer Science, Innopolis University, Innopolis, Russia

Interpretable Anomaly Detection Openai Offline Reinforcement Learning

Ask PDF

May 21, 2026cs.SE

FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection

Production systems generate millions of log lines daily, yet most anomaly detectors operate at the session or window-level, flagging groups of lines rather than identifying the specific message responsible. This coarse granularity forces operators to inspect many routine lines per alert. Message-level detection offers finer granularity, but remains challenging. A single event template may correspond to both normal and anomalous messages, failures arise from heterogeneous subsystems, and line-level labeling at scale is impractical. Although large language models (LLMs) can reason over log semantics, applying them to every line is too costly for continuous monitoring. We present FAME (Failure-Aware Mixture-of-Experts), a label-efficient message-level mixture-of-experts framework that uses an LLM only once offline. We annotate at most K labeled lines per template to derive binary normal/anomaly indicators and representative examples. The LLM proposes a partition of templates into failure domains, and a certification step validates the proposal before training. FAME trains a lightweight router and domain experts that run on-premise and output anomaly predictions and failure-domain labels. On BGL, FAME achieves F1 = 98.16 at K = 100 reducing annotation effort by 76x and detects 86.3% of anomalies from unseen EventIDs. On Thunderbird, FAME reaches F1 = 99.95 with perfect recall.

Huanchi Wang, Zihang Huang, Yifang Tian +3

Department of Electrical and Computer Engineering University of Toronto, Toronto, Ontario

Interpretable Anomaly Detection Early Failure Prediction Mixture-Of-Experts

Ask PDF

May 9, 2026cs.LG

Seeing the Needle in the Haystack: Towards Weakly-Supervised Log Instance Anomaly Localization via Counterfactual Perturbation

Log anomaly detection is a critical task for system operations and security assurance. However, in networked systems at scale, log data are generated at massive scale while instance-level annotations are prohibitively expensive, posing great difficulties to fine-grained anomaly localization. To address this challenge, we propose LogMILP (Log anomaly localization based on Multi-Instance Learning enhanced by prototypes and Perturbation), a weakly supervised framework that enables both bag-level anomaly detection and instance-level anomaly localization using only bag-level labels. Our method guides the model to pinpoint the critical log entries using prototype-guided structural modeling with counterfactual perturbation consistency regularization, thereby improving localization reliability and interpretability under coarse-grained supervision. Experimental results on three public datasets demonstrate that LogMILP achieves competitive detection performance while yielding significantly more reliable instance-level localization. Our code is open-sourced at https://github.com/YUK1207/LogMILP.

Yutszyuk Wong, Wentai Wu, Yuen-Ying Yeung +1

Jinan University · Guangzhou, China · South China University of Technology

Interpretable Anomaly Detection

Ask PDF