Lev-3B: A Data Recipe for Compact Probabilistic Log Decision Models

Leo Betthauser; Yingqi Liu; Uday Shah; Jishang Wei; Sainyam Galhotra; Liang Gou; Vedant Dharnidharka; Hao Yang

cs.LGOct 28, 2026

Lev-3B: A Data Recipe for Compact Probabilistic Log Decision Models

Authors: Leo Betthauser, Yingqi Liu, Uday Shah, Jishang Wei, Sainyam Galhotra, Liang Gou, Vedant Dharnidharka, Hao Yang

Organizations: Cisco

Abstract

Many log-analysis tasks involve choosing among known options, such as detecting anomalous sessions or identifying attack techniques, and benefit from calibrated probabilities. We present Lev-3B, an open 3B-parameter decision model built on Granite-4.1-3B-Base with a shared pointer readout, and study its training recipe. The recipe has two key parts. First, training follows a curriculum: the model learns bounded natural-language decisions, using no log data, before it sees any logs. Second, the log stage mixes the target tasks with general log data spanning 35 auxiliary tasks unrelated to either evaluation task. We evaluate one shared checkpoint on sources seen in training and on sources excluded from training, calibration and model selection. The two-stage recipe transfers substantially better than training on log data alone. Adding general log data improves performance on excluded sources. At a fraction of their serving cost, Lev-3B outperforms prompted frontier models on anomaly detection, including on an excluded logging system, but trails them on attack-technique classification. Its probabilities are well calibrated on familiar sources, but on the excluded logging system it confidently misses anomalies. We release the checkpoint, recipe, data manifests, and prediction-level artifacts.

Figures & tables

Source and task	Unit	Train	Dev.	Cal.	Test
BGL anomaly	Window/session	55,775	9,295	9,297	18,592
Spirit anomaly	Window/session	34,624	3,808	3,977	7,858
Thunderbird anomaly	Window/session	61,758	10,294	10,293	20,587
HDFS anomaly (excluded)	Block session	0	0	0	115,013
ATT&CK, represented	Grouped record	544 real	51	38	134
ATT&CK, withheld (excluded)	Grouped record	0	0	0	178

Table 1: Data and source access. Zeros are required exclusions. Training counts show unique records (real records only for ATT&CK).

	Represented (F1)			HDFS (excluded)
System	BGL	Spirit	TB	AUROC	AUPRC	F1
Granite-4.1-3B-Base	0.19	0.00	0.44	0.6063	0.1579	12.21
Granite-3B instruct, prompted	63.59	16.36	2.41	–	–	14.93
Jev (0-shot)	57.67	18.12	3.12	0.6168	0.0537	5.65
Jev (5-shot)	57.68	21.16	6.55	0.7074	0.0958	6.54
Microsoft Decision-1 (0-shot)	60.03	21.98	3.10	0.7162	0.2376	14.29

Table 2: Anomaly detection: comparison with prompted and hosted systems. TB denotes Thunderbird. F1 is anomaly-class F1 in percent; AUROC and AUPRC are on

[0,1]

. HDFS is excluded from adaptation. Bold : best in column. Dashes denote unavailable continuous scores. ∗ denote metrics computed on a 10% sample of the data.

	Represented sourcetypes		Withheld sourcetypes
System	Accuracy	Macro-F1	Accuracy	Macro-F1
Granite-4.1-3B-Base	0.00	0.00	0.00	0.00
Granite-3B instruct, prompted	3.0	0.1	5.1	1.6
Jev (0-shot)	24.6	20.6	32.6	28.1
Jev (5-shot)	29.9	23.2	34.3	25.6
Microsoft Decision-1 (0-shot)	22.4	17.3	28.1	23.9

Table 3: ATT&CK parent-technique classification with 130 offered alternatives: comparison with prompted and hosted systems. All metrics are percentages. Every row uses identical inputs and options. Withheld sourcetypes contribute no training data, and some withheld-split techniques have no real training records. Shaded : our model. Bold : best in column. Underline : best among non-frontier systems.

Model	BGL+Lib $\to$ TB	Lib+BGL $\to$ TB	TB+Lib $\to$ BGL	BGL+TB $\to$ Lib	TB+Lib $\to$ HDFS
LogRobust [ 52 ]	57.36/91.79/70.60	18.07/4.59/7.32	66.48/31.69/42.92	79.15/5.29/9.91	2.43/82.50/4.73
LogPrompt [ 29 ]	51.10/70.65/59.30	—	39.49/76.72/52.15	66.54/47.30/55.29	8.73/84.07/15.81
MetaLog [ 51 ]	42.10/86.23/56.58	56.70/73.50/64.01	29.16/23.35/25.93	90.72/10.02/18.04	3.50/0.05/0.10
LogICL [ 49 ]	70.16/79.83/74.68	65.96/66.06/66.01	40.21/34.71/37.25	82.39/69.70/75.52	72.87/22.75/ 34.67
Lev (two-source)	83.21/97.88/ 89.95	86.15/94.52/ 90.14	85.21/47.40/ 60.91	96.97/93.22/ 95.06	100.00/19.86/33.14

Table 4: Comparison with prior cross-system log anomaly detectors under the multi-source protocol of LogICL. Each cell is precision / recall / F1 (percent) for training on the two listed sources (primary first) and testing on the target. Lev rows are variants with anomaly-only stage-two training, not the release checkpoint. LogPrompt is prompted zero-shot and uses no training data. Baseline numbers are from [ 49 ] . Lib: Liberty; TB: Thunderbird. Bold: best F1 per column.

	Training data					Represented (F1)			HDFS (excluded)
Variant	D	A	L	T	Steps	BGL	Spirit	TB	AUROC	AUPRC	F1
Granite-4.1-3B-Base (no adaptation)					0	0.19	0.00	0.44	0.6063	0.1579	12.21
Log data only (A+L+T)		✓	✓	✓	1	99.33	85.96	92.71	0.7154	0.1039	11.80
Granite-4.1-3B-Base $\rightarrow$ D	✓				1	65.93	17.97	2.82	0.6981	0.2724	9.05
D $\rightarrow$ A	✓	✓			2	99.64	91.82	94.63	0.7787	0.3117	31.23
D $\rightarrow$ A+ T	✓	✓		✓	2	99.30	77.47	91.36	0.7509	0.2485	25.31

Table 5: Anomaly detection: ablation of training data and stages. D: general decision data; A: anomaly data; L: general log data; T: ATT&CK data. A check mark means the component is in the training data. Steps is the number of training stages: in the two-step recipe, D comes first and the remaining components follow; in log-data-only, the log data are trained in a single stage without D; the base model has no adaptation (0). Shaded : our models. Bold : best in column among these rows. HDFS is excluded from adaptation.

	Training data					Represented		Withheld
Variant	D	A	L	T	Steps	Acc.	Macro-F1	Acc.	Macro-F1
Granite-4.1-3B-Base (no adaptation)					0	0.00	0.00	0.00	0.00
Log data only (A+L+T)		✓	✓	✓	1	29.10	24.27	28.09	24.34
Granite-4.1-3B-Base $\rightarrow$ D	✓				1	4.48	4.02	8.43	3.89
D $\rightarrow$ A	✓	✓			2	4.48	4.02	5.06	4.89
D $\rightarrow$ A+T	✓	✓		✓	2	35.8	31.7	34.3	19.8

Table 6: ATT&CK classification: ablation of training data and stages. D, A, L, T and Steps as in Table 5 . All metrics are percentages. Shaded : our models. Bold : best in column among these rows.

	NLL		ECE
Source / system	Raw	Fit	Raw	Fit
Represented
BGL anomaly	0.0189	0.0173	0.0046	0.0070
Spirit anomaly	0.0791	0.0578	0.0094	0.0025
Thunderbird anomaly	0.0259	0.0286	0.0083	0.0166
ATT&CK represented	3.8512	3.1198	0.3099	0.1272

Table 7: Probability quality before (raw) and after a single checkpoint-level temperature fit (fit). Lower is better. Uniform choice over 130 techniques has NLL

=\ln 130\approx 4.87

. Shaded rows : our model. Bold : best per source and metric among rows shown. Jev is native only; prior baselines use no excluded-source labels. ∗ marks metrics computed on a 10% sample of the corresponding test set.

Figure 1: Anomaly-probability reliability on (a) Spirit, a represented source, and (b) HDFS, which is excluded from adaptation. The dashed diagonal is perfect calibration. Curves show raw Lev-3B, fitted-temperature Lev-3B and native Jev probabilities in ten equal-width bins.

Source	Version	Conf. error rate (errors / accepted)	Coverage	FA / miss
HDFS	Lev-3B raw	1.94% (2189 / 113021)	0.9827	84 / 2105
HDFS	Lev-3B fitted	1.73% (1913 / 110757)	0.9630	1 / 1912
HDFS	Jev (0-shot)	86.1% (6593 / 7657)	0.0666	6509 / 84
HDFS	Jev (5-shot)	72.4% (5819 / 8036)	0.0699	5789 / 30
HDFS	Microsoft Decision-1 (0-shot)	8.18% (3672 / 44905)	0.3904	3068 / 604
HDFS	Microsoft Decision-1 (5-shot)	5.59% (4157 / 74330)	0.6463	3337 / 820

Table 8: Confidence failures at confidence

\geq 0.9

. Confident error rate is the fraction of accepted predictions (confidence

\geq 0.9

) that are wrong, with errors / accepted in parentheses; coverage is the fraction of all examples that are accepted. For anomaly rows, FA / miss are confident false alarms and confident misses (counts, not rates). n/a: ATT&CK is not binary. Shaded rows : our model. ∗ marks metrics computed on a 10% sample of the corresponding test set and approximated to the full population.

	Attack classification		Anomaly detection
System	Latency	Cost/1,000	Latency	Cost/1,000
Lev-3B, A10G on-demand	1,429 ms	$0.5147	924 ms	$0.2583
Lev-3B, A10G 3yr-reserved	1,429 ms	$0.2226	924 ms	$0.1117
Jev (0-shot)	167 ms	$0.2527	143 ms	$0.2497
Microsoft Decision-1 (0-shot)	675 ms	$0.3596	497 ms	$0.2279
OpenAI Decisions (0-shot)	453 ms	$0.8055	434 ms	$0.3849

Table 9: Inference latency and cost per 1,000 requests. Shaded rows : our model. Bold : lowest latency and lowest cost per column.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Fixed $0.5$	Source	Mean OT	Median OT	Shrink $.25$	Shrink $.50$	Shrink $.75$	Med./IQR
Lev-3B	43.32	28.37	18.20	34.58	43.08	41.02	34.42	43.02
Jev	5.65	11.85	8.97	11.87	11.24	11.77	11.87	8.63

Appendix

Table 10: Anomaly-class F1 (percent) on the 115,013-session HDFS population under source-derived and unlabeled-target-adaptive thresholds. Mean OT and median OT transport the mean and median of the three source fractions above the source threshold. Shrink

s

interpolates in logit space between the source threshold (

s=0

) and mean OT (

s=1

). Med./IQR applies robust location-scale transport in logit space. Bold: best per row.

	Represented sourcetypes		Withheld sourcetypes
Checkpoint	Accuracy	Macro-F1	Accuracy	Macro-F1
D $\rightarrow$ A+T, real only	31.3	24.7	34.8	19.7
D $\rightarrow$ A+T, real + synthetic	35.8	31.7	34.3	19.8

Appendix

Table 11: Effect of synthetic ATT&CK augmentation (percent). Both Lev rows are D

\rightarrow

A+T, without general log data.

			Accuracy		Macro-F1
Split	Real support	$n$	Jev	Lev-3B	Jev	Lev-3B
Test	High ( $\geq 10$ )		28.1	40.4	36.6	43.6
Test	Medium (2–9)		19.7	31.1	18.2	31.8
Test	Low (1)		37.5	37.5	33.3	35.4
Withheld	High ( $\geq 10$ )		24.4	43.9	33.2	35.1
Withheld	Medium (2–9)		35.3	27.9	33.8	23.5

Appendix

Table 12: Accuracy and macro-F1 (percent) by real training support per technique, for Jev and the synthetic-augmented D

\rightarrow

A+T checkpoint. Buckets count real training records only.

Real-support floor	$n$ (of 178)	Lev-3B	Jev (0-shot)	GPT-5.6 Luna	GPT-5.6 Sol
None	178	42.1 / 29.5	32.6 / 28.1	51.7 / 42.7	51.7 / 45.4
$\geq 5$ (19 techniques)	117	41.9 / 25.3	22.2 / 22.6	45.3 / 31.9	44.4 / 37.0
$\geq 10$ (11 techniques)	82	48.8 / 33.3	26.8 / 34.0	56.1 / 37.9	53.7 / 45.9
$\geq 20$ (4 techniques)	25	52.0 / 50.0	28.0 / 38.6	60.0 / 54.9	64.0 / 56.4

Appendix

Table 13: Withheld-sourcetype accuracy / macro-F1 (percent) for the release checkpoint, restricted to techniques with at least the given number of real (non-synthetic) training records. Jev uses the same zero-shot predictions as Table 3 .

Source	Decision content	Requests	Questions	Share (%)
AG News	News topic and binary topic propositions	1,000	3,000	8.0
Amazon reviews	Five-level product-review rating	1,000	1,000	8.0
Banking77	Banking intent, 77 alternatives	1,000	1,000	8.0
BoolQ	Yes/no question answering	1,000	1,000	8.0
DBpedia14	Encyclopedia category, 14 alternatives	1,000	1,000	8.0
IMDB	Binary movie-review sentiment	1,000	1,000	8.0

Appendix

Table 14: Decision-v7 training composition. Share is the percentage of the 12,576 training requests, rounded to one decimal. Policy and compositional data are generated by Kev; the other rows are converted from public NLP datasets.

Task family	Question content	Tasks	Requests	Share (%)
LogGLUE-derived	Summaries, semantic relations, tokens, and contrasts	6	7,568	26.3
LogQA-style	Answer selection and log-field or template aspects	5	4,769	16.6
Trace LogQA	Trace, event, timing, and evidence questions	1	4,000	13.9
Template classification	Original and derived log-template choices	2	2,000	7.0
Java build results	Build success and failure category	2	2,000	7.0
LO2v2	Fault-injection status and affected service	2	2,000	7.0

Appendix

Table 15: General log training data distribution across different task families.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 386 papers

Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection
Aug 18, 2026Bin Li, Dongdong Wang, Siyang LuPost-Hoc Calibration Online Anomaly Detection
School of Cyberspace Science and Technology, Beijing Jiaotong University, Beijing, China · University of Florida, Gainesville, FL, USA · School of Computer and Technology, Beijing Jiaotong University, Beijing, China
LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection
Jun 6, 2026David Eje, Tanmay Sharma, Khush Patel +2RL for Language Models Online Anomaly Detection
Department of Computer Science, Innopolis University, Innopolis, Russia
FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection
May 21, 2026Huanchi Wang, Zihang Huang, Yifang Tian +3Fault Detection Mixture-of-Experts Models
Department of Electrical and Computer Engineering University of Toronto, Toronto, Ontario

Aug 18, 2026J/K move · Enter open · S save

Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

Bin Li, Dongdong Wang, Siyang Lu

School of Cyberspace Science and Technology, Beijing Jiaotong University, Beijing, China · University of Florida, Gainesville, FL, USA · School of Computer and Technology, Beijing Jiaotong University, Beijing, China

Post-Hoc Calibration Online Anomaly Detection Language Model Calibration LLM Reliability

Open paper AskPDF arXiv

Online log anomaly detection is critical for maintaining the reliability of large-scale computing systems. Although recent language model-based log anomaly detectors achieve strong detection performance, their confidence estimates remain poorly calibrated. We show that these detectors frequently assign excessive confidence to incorrect predictions, particularly for anomalous logs under severe class imbalance. Moreover, confidence on erroneous predictions remains persistently high even when conventional calibration metrics indicate good calibration, creating a critical reliability gap for operational monitoring systems. To address this issue, we propose Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework for reliable log anomaly detection. LoRD learns prediction-route-specific reliability models from latent representations of correctly classified validation samples and estimates prediction reliability through route-wise reconstruction distances. Based on the estimated reliability, LoRD selectively recalibrates high-risk predictions to suppress overconfident errors while preserving reliable predictions. Extensive experiments on four large-scale log benchmark datasets and multiple language model-based detectors demonstrate that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing anomaly detection performance.

Abstract

Language Model Calibration

Explore similar work

Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection

LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection

FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection