Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting

Siyu Zhou; Jialin Ouyang; Fan Xia; Adarsh Subbaswamy; Drew Prinster; Suchi Saria; Augusto Garcia-Agundez; Sarah Pollet

cs.LGOct 12, 2026

Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting

Authors: Siyu Zhou, Jialin Ouyang, Fan Xia, Adarsh Subbaswamy, Drew Prinster, Suchi Saria, Augusto Garcia-Agundez, Sarah Pollet,

Organizations: University of California, San Francisco · University of Maryland · Johns Hopkins University

Abstract

Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically control false alarms by requiring the discrepancy measure used to detect shifts to be pre-specified. However, for unstructured data, pre-specification creates a sensitivity--breadth trade-off: a narrowly defined measure is sensitive to the anticipated shift but may miss unexpected changes, whereas a broadly defined measure may detect a wider range of shifts but with reduced sensitivity. Although adaptively hypothesizing the shift direction can overcome this limitation, such data-dependent adaptation generally invalidates conventional false-alarm guarantees. We reconcile adaptivity with statistical validity through testing-by-betting, reframing sequential changepoint detection as a betting game in which the bettor continually updates its hypothesized shift direction, or witness function, as new observations arrive. We introduce asymptotic e-detectors, which approximate the pre-change distribution using a calibration set of size $m$ , and prove that they asymptotically control the average run length to a false alarm as $m$ increases. Empirically, calibration sets as small as $m=30$ provide effective control. We then develop adaptive monitoring strategies ranging from black-box bettors that learn from embeddings to interpretable bettors based on LLM-proposed concepts. Across three real-world datasets, including data from a clinical AI scribe deployment, adaptive monitors detect shifts faster than fixed monitors at the same false-alarm level. Further, when the true shifts are sparse and interpretable, LLM-concept monitors provide the fastest and most interpretable detection, while also guiding root-cause diagnosis for complex shifts.

Figures & tables

Figure 1: Adaptive monitoring procedure overview: At each time step, an adaptive bettor uses past observations to bet on how the data may have shifted. When the next observation arrives, the bet’s return is computed by comparing it with a calibration set, and an alarm is raised once the accumulated wealth crosses a threshold.

Figure 2: LLM-concept bettor for estimating when and what shifted, using LLM reasoning, unsupervised learning, and statistical changepoint estimation.

Figure 3: Experimental results. Top: EDD against nominal

\mathrm{ARL}_{0}

, where lower is better. Bottom: Concepts learned by the LLM-concept bettor, plotted by prevalence before vs after the changepoint. Concepts in (e-g) are colored by their agreement with the true shifted attribute (stars). Concepts in (h) are colored by themes (Table C.3 ), as true shifts are unknown for real-world AI deployments. The circled concept in (h) asks whether the draft includes unneeded non-clinical information, such as a review of systems or introductory sentences.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure B.1: Keyphrase extraction prompt (abbreviated) for CEBaB experiment.

Panel of Figure 5 — Figure B.2: Concept proposal prompt (abbreviated) for CEBaB experiment. Top keyphrase clusters (topics) are shown from an example replicate.

Panel of Figure 6 — Figure B.3: Concept annotation prompt for the CEBaB experiment.

	CEBaB	CUB: yellow bill	CUB: white crown	AI scribe
Data	Restaurant reviews from CEBaB	Bird images from CUB-200-2011	Bird images from CUB-200-2011	HPI section of Rheumatology clinical notes
Shifted attribute	Whether review mentions ambiance	Whether bird has yellow bill in image	Whether bird has white crown in image	Clinical notes from 60 days before vs after model update on 2026-02-25
Prevalence pre $\to$ post	0.52 $\to$ 0.88	0.17 $\to$ 0.64	0.30 $\to$ 0.95	N/A
Embedding model	MiniLM	CLIP ViT-B/32	CLIP ViT-B/32	MiniLM
Embedding dim $d$	384	512	512	384
$n_{\text{pre}}$	50	50	50	100

Appendix

Table C.1: Summary of experimental settings across the three datasets.

Dataset	Direction	IC text	OC text
CEBaB	Informed	a typical restaurant review describing the customer’s experience of the visit	a restaurant review that comments on the ambiance or decor of the restaurant
CEBaB	Unrelated	a typical restaurant review describing the customer’s experience of the visit	a restaurant review that mentions parking the car near the restaurant
CUB (yellow bill)	Informed	a photo of a bird	a photo of a bird with a yellow bill
CUB (yellow bill)	Unrelated	a photo of a bird	a photo of a bird perched on a wire or fence post
CUB (white crown)	Informed	a photo of a bird	a photo of a bird with a white crown
CUB (white crown)	Unrelated	a photo of a bird	a photo of a bird perched on a wire or fence post

Appendix

Table C.2: Text exemplars used to define pre-specified drift directions for fixed monitors. Each direction is defined by an in-class (IC) and out-of-class (OC) text description, embedded to form a unit drift vector.

Figure C.1: Empirical $\mathrm{ARL}_{0}$ against nominal $\mathrm{ARL}_{0}$ on the $100$ CEBaB ambiance null streams. Shaded regions are simultaneous $95\%$ bands over streams, and the dotted line marks empirical $\mathrm{ARL}_{0}$ equal to nominal $\mathrm{ARL}_{0}$ .

Panel of Figure 10 — Figure C.2: Empirical $\mathrm{ARL}_{0}$ against nominal $\mathrm{ARL}_{0}$ on the CUB yellow bill (left) and white crown (right) null streams, over $100$ null streams per setting. Shaded regions are simultaneous $95\%$ bands over streams, and the dotted line marks empirical $\mathrm{ARL}_{0}$ equal to nominal $\mathrm{ARL}_{0}$ .

Theme	Codebook definition	# Concepts	Before $\to$ after	Example concepts (before $\to$ after)
Unneeded content	Draft content that does not belong in the note because it is unnecessary, redundant, verbose, non-clinical, introductory or identifying, or the removal, condensing or restructuring of such content, whether clinical or not	46	0.79 $\to$ 0.60	Does the initial draft include unnecessary or irrelevant non-clinical information such as review of systems, introductory sentences, or personal details? ( $0.93\to 0.60$ ) Does the edit remove redundant or unnecessary note sections (including review of systems or initial summary)? ( $0.86\to 0.54$ )
Clinical correction	Clinical content (diagnoses, symptoms, medications, treatment) that the clinician adds, corrects or clarifies, or that the draft is missing	21	0.56 $\to$ 0.61	Does the edit correct or improve diagnostic information? ( $0.38\to 0.59$ ) Does the initial draft miss or have incorrect medication, treatment, or procedure details? ( $0.59\to 0.43$ )
Non-clinical correction	Non-clinical information (patient or social history, follow-up, administrative or demographic details) that the clinician adds, corrects or clarifies, or that the draft is missing	5	0.51 $\to$ 0.67	Does the edit improve the accuracy and completeness of patient, social history, or provider demographic information? ( $0.50\to 0.68$ )

Appendix

Table C.3: Themes of the AI scribe concepts in Figure 3 (h), over the four concepts active at the end of each of

18

streams. An LLM judge ( gpt-4.1-mini ) assigns each concept to a theme from its text alone, using a fixed codebook based on the vendor’s stated purpose for the update and the clinicians’ reports of editing less. The first two themes come from these reports; the third theme is named after reading concepts that fit neither. Before

\to

after shows the mean fraction of notes for which a concept is annotated as present, before and after the changepoint.

Stream	Concept	Theme	Before $\to$ after
1	Does the edit add or correct procedural or diagnostic details based on inaccuracies or omissions in the initial draft?	Clinical correction	$0.56\to 0.67$
	Does the edit clarify or correct medication, management, or demographic information based on inaccuracies or omissions in the initial draft?	Clinical correction	$0.61\to 0.72$
	Does the edit add or correct missing or inaccurate patient history and administrative details (such as functional status, anatomical location, provider identification, or lab data) in the initial draft?	Non-clinical correction	$0.63\to 0.76$
	Does the edit add or correct missing treatment and clinical history details based on inaccuracies or omissions in the initial draft?	Clinical correction	$0.53\to 0.64$
2	Does the edit remove unnecessary or irrelevant patient-related information (such as identification, introduction, background, history, or personal details) from the initial draft?	Unneeded content	$0.82\to 0.66$
	Does the edit improve procedural and clinical detail documentation by clarifying, correcting, or adding missing information based on the initial draft?	Clinical correction	$0.53\to 0.72$

Appendix

Table C.4: Concepts active at the end of the first four AI scribe streams in Figure 3 (h), with the theme each is assigned in Table C.3 and the fraction of notes for which it is annotated as present before and after the update.

Figure D.1: Empirical

\mathrm{ARL}_{0}

as a multiple of the nominal

\mathrm{ARL}_{0}=1/\alpha

with a fixed score (left,

\mathrm{Beta}(2,2)

scores) and with a learned score from a classifier refit on the stream history (right). Dashed colored lines reuse a calibration dataset of size

m\in\{10,30,100,1000\}

at every step, the solid black line uses the oracle expectation

\mathbb{E}_{P_{0}}[\hat{f}_{t}(X)]

, and the solid grey line redraws

10

reference observations at every step. The dotted line marks the nominal

\mathrm{ARL}_{0}

. Each point summarizes

2{,}000

null streams, and streams without an alarm after

100{,}000

steps are counted at

100{,}000

, so each value is a lower bound.

Figure D.2: Distribution of the null alarm time

\tau_{m}

1/\alpha=300

with a fixed score (left) and a learned score (right). Every null stream draws its own calibration dataset, so each curve estimates

P(\tau_{m}\leq t)

averaged over calibration datasets from

2{,}000

null streams. The black curve is the oracle detector, which uses the exact pre-change expectation.

Distribution Shift Detection

Distribution shift detection identifies when data or model behavior changes between reference and deployment settings.

See all 62 papers

Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift
Jun 23, 2026Kevin Ren, Manish Raghavan, Nikhil GargAI-Generated Text Detection Distribution Shift
Cornell Tech · MIT
Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains
Jul 4, 2026Zhuoer Shen, Mingyi Wang, Shaofeng Zou +1Distribution Shift Robustness AI-Generated Text Detection
Department of Computer Science, University of California, Santa Barbara · School of ECEE, Arizona State University
Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers
Jun 10, 2026Jun Wen LeongDistribution Shift Conformal Prediction

Jun 23, 2026J/K move · Enter open · S save

Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift

Kevin Ren, Manish Raghavan, Nikhil Garg

Cornell Tech · MIT

AI-Generated Text Detection Distribution Shift Continual Test-Time Adaptation Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

Deployed AI text detection approaches are supervised, relying on training-time access to labeled datasets of both human-written and AI-generated text. This approach is vulnerable to three types of distribution shifts that occur continually post-deployment, and for which labeled data is often unavailable: adversarial humanization, new LLMs being released, and unidentifiable temporal drift in human writing after widespread LLM usage. Simultaneously, existing approaches do not leverage a key signal of LLM usage: inference-time homogeneity. We propose a test-time adaptation (TTA) approach that adapts to distribution shifts by leveraging homogeneity among unlabeled samples observed at inference time. Empirically, we find that state-of-the-art supervised detectors systematically fail when they encounter distribution shifts in AI-generated and human writing, both adversarial and natural, while TTA with semi-supervised learning is largely robust; e.g., the commercial model Pangram detects just 27.0% of our adversarial AI-generated text, compared to 93.3% for our test-time approach. We establish that TTA is a promising framework for AI text detection in the wild.

Abstract

Distribution Shift Detection

Explore similar work

Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift

Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains

Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers