RankShift: In-Database Detection and Explanation of Categorical Shifts

Omair Shafi Ahmed

cs.LGAug 28, 2026

RankShift: In-Database Detection and Explanation of Categorical Shifts

Authors: Omair Shafi Ahmed

Organizations: Microsoft

Abstract

A login service can receive its usual number of failed sign-ins while one source grows from 2% to 30% of them. The same pattern appears in system logs when a rare event template becomes common while the message rate stays stable. These events change which categories are active without changing how many events occur. RankShift detects such changes inside the analytical database that stores the data. It compares each window's category shares with a benign reference using a Pearson score whose terms identify the categories responsible for the change. The same query returns the score, calibrated alert, and largest increasing contributions. We evaluate RankShift on HDFS, BGL, and Thunderbird. It matches the count-vector autoencoder within 0.001 AUROC on HDFS (0.999 versus 1.000) and leads on Thunderbird (0.983 versus 0.949). In a controlled fixed-volume experiment, RankShift detects rare-category shifts that are invisible to event-count monitoring, reaching 0.787 AUROC compared with 0.771 for the autoencoder. Across all three corpora, observed false-alarm rates track the requested operating levels. RankShift requires no model training or inference service, and the autoencoders deployed state is 137x larger.

Figures & tables

Fig. 1: Illustrative fixed-volume shift in failed sign-ins. (a) Each minute contains 100 failed sign-ins, while source x rises from 2% to 30% of the total. A monitor of total volume remains flat. (b) At minute 8, source x accounts for 98% of the RankShift score (3.92 of 4.00). The signs show whether each source gained or lost share.

dataset	window / categories	vocabulary construction	detection split	alert split	reported result
HDFS	block / 29	fixed event identifiers	benign half $\rightarrow$ held-out benign + anomalies	random benign reference/calibration/test split	detection, alert
BGL	hour / 644	first-half Drain, frozen + OTHER	first-half benign $\rightarrow$ complete second half	second-half benign reference/calibration/test split	detection, alert, redistribution
Thunderbird	minute / 1,025	first-half benign top-1,024 + OTHER	first-half benign $\rightarrow$ complete second half	second-half benign reference/calibration/test split	detection, alert
CTU-13	minute / 2,049	first-half benign top-2,048 + OTHER	first-half benign $\rightarrow$ second-half attacks	–	raw-source recovery

TABLE I: Datasets and evaluation protocols. Category vocabularies are fixed before testing. Alert evaluation uses disjoint benign reference, calibration, and test cohorts.

dataset	test period	RankShift	autoencoder
HDFS	held-out	0.999 ( $<$ .001)	1.000 ( $<$ .001)
BGL	second half	0.699	0.864 (0.001)
Thunderbird	second half	0.983	0.949 (0.001)

TABLE II: Detection AUROC on shared test samples. HDFS reports mean (standard deviation) over five paired splits. BGL and Thunderbird use first-half benign windows for setup and the complete second half for testing; autoencoder values report mean (standard deviation) over five training seeds.

dataset	$\alpha$	false alarm	detection
HDFS	.01	0.010	1.000
	.05	0.041	1.000
	.10	0.098	1.000
BGL	.01	0.006	0.490
	.05	0.038	0.760
	.10	0.080	0.767

TABLE III: Calibrated alert rates averaged over five random partitions. Within each partition, the benign reference, calibration, and test cohorts are disjoint. False-alarm rate is measured on benign test windows; detection rate is measured on anomalous windows.

Fig. 2: Detection under controlled fixed-total redistribution on BGL. The horizontal axis gives the fraction of transferred events assigned to common recipient categories; the remainder enters rare recipient categories. The upper panel reports mean AUROC over five paired seeds. The lower panel reports RankShift minus autoencoder AUROC, with positive values favoring RankShift. Error bars show one standard deviation. Every synthetic anomaly has the same event count as its original benign window, giving the volume baseline 0.500 AUROC.

Fig. 3: Complete RankShift query time as event volume increases. Lines report median client wall time over five warm runs; upper bars report p95. DuckDB and ADX complete workloads through 64 million events.

events (M)	DuckDB client	DuckDB GiB	ADX client	ADX GiB/node
0.64	0.428 (0.448)	0.12	0.900 (0.950)	0.12
3.2	0.813 (0.855)	0.42	1.300 (1.500)	0.50
16	2.972 (3.288)	1.27	3.300 (3.800)	2.50
32	5.820 (5.993)	2.27	6.000 (7.000)	5.00
64	12.2 (12.881)	5.09	11.000 (13.000)	9.00

TABLE IV: Native-query scaling under the same logical workload. Client entries are median (p95) seconds over five warm runs. Peak memory is the maximum observed across those runs; ADX memory is per node.

measure	RankShift	autoencoder
online ( $\mu$ s/window)	19.5 (60.4)	43.5 (65.7)
setup (s)	0.009 (0.011)	2.279 (20.104)
deployed state (bytes)	5,160	706,264

TABLE V: Matched same-process cost on reference-frozen BGL. Times are median (p95) over five complete runs.

Distribution Shift Detection

Distribution shift detection identifies when data or model behavior changes between reference and deployment settings.

See all 62 papers

Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting
Oct 12, 2026Siyu Zhou, Jialin Ouyang, Fan Xia +8Distribution Shift Detection Change-Point Detection
University of California, San Francisco · University of Maryland · Johns Hopkins University
PairAudit: Guiding Human Review with Graph Tokens under Distribution Shift
Oct 7, 2026Jiran Tao, Binyan JiangGraph-Based Anomaly Detection Network Intrusion Detection
The Hong Kong Polytechnic University
NLLog: Lightweight, Explainable SOC Anomaly Detection via Log-to-Language Rewriting
Jun 3, 2026Samuel Ndichu, Tao Ban, Seiichi Ozawa +2Online Anomaly Detection Cybersecurity
National Institute of Information and Communications Technology, Tokyo, Japan · Kobe University, Kobe, Japan

Oct 12, 2026J/K move · Enter open · S save

Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting

Siyu Zhou, Jialin Ouyang, Fan Xia, Adarsh Subbaswamy+7

University of California, San Francisco · University of Maryland · Johns Hopkins University

Distribution Shift Detection Change-Point Detection

Open paper AskPDF arXiv

Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically control false alarms by requiring the discrepancy measure used to detect shifts to be pre-specified. However, for unstructured data, pre-specification creates a sensitivity--breadth trade-off: a narrowly defined measure is sensitive to the anticipated shift but may miss unexpected changes, whereas a broadly defined measure may detect a wider range of shifts but with reduced sensitivity. Although adaptively hypothesizing the shift direction can overcome this limitation, such data-dependent adaptation generally invalidates conventional false-alarm guarantees. We reconcile adaptivity with statistical validity through testing-by-betting, reframing sequential changepoint detection as a betting game in which the bettor continually updates its hypothesized shift direction, or witness function, as new observations arrive. We introduce asymptotic e-detectors, which approximate the pre-change distribution using a calibration set of size

m

, and prove that they asymptotically control the average run length to a false alarm as

m

increases. Empirically, calibration sets as small as

m=30

provide effective control. We then develop adaptive monitoring strategies ranging from black-box bettors that learn from embeddings to interpretable bettors based on LLM-proposed concepts. Across three real-world datasets, including data from a clinical AI scribe deployment, adaptive monitors detect shifts faster than fixed monitors at the same false-alarm level. Further, when the true shifts are sparse and interpretable, LLM-concept monitors provide the fastest and most interpretable detection, while also guiding root-cause diagnosis for complex shifts.

Abstract

Distribution Shift Detection

Explore similar work

Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting

PairAudit: Guiding Human Review with Graph Tokens under Distribution Shift

NLLog: Lightweight, Explainable SOC Anomaly Detection via Log-to-Language Rewriting