From Training Exposure to Membership Evidence: A Controlled Study of OLMo 2

Ali Banaei; Muhammad Shahab Shahzad; Gerald Q. Maguire; Marco Chiesa; Dejan Kostić

cs.CROct 30, 2026

From Training Exposure to Membership Evidence: A Controlled Study of OLMo 2

Authors: Ali Banaei, Muhammad Shahab Shahzad, Gerald Q. Maguire, Marco Chiesa, Dejan Kostić

Organizations: KTH Royal Institute of Technology, Stockholm, Sweden

Abstract

Whether a language model was trained on a document matters for copyright, privacy, and benchmark contamination. Yet most membership-inference (MI) evaluations on language models collect members and non-members after the fact, so success may reflect distribution shift and failure is inconclusive. We re-ran OLMo 2 1B's official 50-billion-token mid-training stage, replacing selected training chunks with documents from 7,814 pairs matched on domain and difficulty. For each pair, a coin flip determined which document was inserted and which was held out. The run released by the Allen Institute for AI (AI2) from the same seed saw identical data everywhere else and serves as the counterfactual. One exposure raises a previously unseen document's likelihood on average but rarely identifies it. Using the counterfactual as reference, the reference ratio has an area under the ROC curve (AUC) of 0.767 but, with one threshold across domains, catches 3.8% of members at a 1% false-positive rate (FPR). In contrast, 13 matched pairs give a paired test 80% power at the 5% level to detect training on a collection, given same-domain non-members and a close reference. A reference trained with another seed recovers 92% of the counterfactual's AUC advantage over chance, whereas reference-free attacks stay within 0.07 of chance when pooled over domains. The full official post-training recipe, re-run on both models, leaves 89% of the average effect. Even with a closely matched reference, substantial document-level variability limits individual membership inference. We will release the checkpoints, per-token log-probabilities, and the evaluation documents; the code is available at https://github.com/ali-bana/OLMo-Twin.

Figures & tables

Fig. 1: One document cannot be decided, but a dozen can. Verified-unseen documents are injected once. (a) TPRat

1\text{\,}\mathrm{\%}

FPRwith one threshold across domains, for the four reference regimes of Section II (no adversary holds the exact counterfactual); even the exact counterfactual catches

3.8\text{\,}\mathrm{\%}

with the reference ratio and

8.8\text{\,}\mathrm{\%}

with the standardized ratio (see § V-A ). (b) AUCwhen

m

documents are pooled and compared by their mean score: a retrained reference is nearly as good as the counterfactual; at

m{=}20

the counterfactual reaches

0.990

and the retrained reference

0.987

Fig. 2: The testbed. AI2’s seed-42 run of OLMo 2 1B’s stage-2 mid-training consumes a fixed sequence of

12\,212\,224

chunks. The treated run consumes the same sequence with

24\,081

slots (

0.197\text{\,}\mathrm{\%}

) replaced by copies of injected documents. Each injected document (a blue slot) has a partner matched in length and pre-treatment difficulty that was held out by a coin flip. Both runs save checkpoints every

1\,000

updates (

24

per run) and both are carried through the official post-training recipe, so every document can be scored under step-matched pairs of models at every stage.

Fig. 3: One exposure moves a document’s likelihood by a small, consistent amount. (a) Per-document effect

\tau(d)

for the three arms over all documents; the never-injected arms are not centered at zero (§ V-F ). (b) The within-pair contrast for verified-unseen single-exposure pairs removes that offset: the mean is

+0.0221

and

85.4\%

of pairs are ordered correctly.

Fig. 4: No reference regime yields a usable per-document operating point for most documents. Receiver operating characteristic (ROC)curves for the four reference regimes, and for the standardized ratio against the exact counterfactual (blue dotted), on verified-unseen single-exposure pairs, (a) linear and (b) log–log with the

1\text{\,}\mathrm{\%}

FPRline. Standardizing lifts the low-FPRend; the other curves converge on the diagonal below about

0.1\text{\,}\mathrm{\%}

FPR.

Fig. 5: Difficulty matching collapses the reference-free statistics we evaluate. AUCof every attack statistic on verified-unseen single-exposure pairs with bootstrap

95\text{\,}\mathrm{\%}

intervals; the dashed line is chance. Pooled over domains, reference-free statistics lie within

0.07

of chance; what a reference buys depends on how close it is to the counterfactual. The standardized ratio changes AUClittle; its gain is at low FPR( Table IV ).

Fig. 6: Set membership is easy; document membership is not. (a) AUCwhen

m

documents are pooled and compared by their mean score, for three reference regimes. (b) Precision at

1\text{\,}\mathrm{\%}

FPRas a function of the prevalence of members in the candidate pool, at the exact counterfactual’s TPR; at a prevalence of

0.2\text{\,}\mathrm{\%}

, about the share of training slots injected here, it is below

1\text{\,}\mathrm{\%}

Fig. 7: A retrained reference is almost as good as the exact counterfactual, because seed noise is small. Three distributions over verified-unseen documents: the difference between two seeds of the same recipe on never-injected text (seed noise), the difference the injection made to the held-out partners (spillover, § V-F ), and the within-pair membership effect; about

1\text{\,}\mathrm{\%}

of the spillover and membership values lie beyond the axis and are not drawn.

Fig. 8: Forgetting is large; the annealed learning rate hides it. All panels use verified-unseen single-exposure pairs scored under

24

step-matched checkpoints. (a) The within-pair effect as a function of the checkpoint it is read at, grouped by when the copy was placed: each curve is flat at zero until its own copy lands, then spikes and decays. (b) Aligned to each document’s own placement: the peak falls with placement time because the learning rate anneals, the endpoint barely moves, and the last checkpoint before placement is zero. (c) The fraction of the peak surviving to the end of the stage against how much training followed, by placement window and pooled against time since the copy.

Fig. 9: Detection lags the effect. True-positive rate at

1\text{\,}\mathrm{\%}

FPRand

d_{z}

against the number of copies

r

on the matched arXiv ladder (cohort M at one copy, D at 2–16, D32 at 32); the effect itself grows linearly in

r

. At

r{\geq}8

the

1\text{\,}\mathrm{\%}

FPRthreshold is the single largest held-out score, so those points sit at the resolution floor of

n{\approx}100

negatives (

40

r{=}32

Fig. 10: Documents in neither model still gain. Mean

\tau(d)

for never-injected documents by domain — the held-out partners and, except for StackExchange, an independent null pool — with the number of tokens injected in that domain. The domain that received

90\text{\,}\mathrm{\%}

of the injected tokens moves most; the ordering among the small domains does not follow token share.

Fig. 11: Detectability is governed by the variance of the score, not only by the size of the effect. Single-exposure cohorts: within-pair effect with

95\text{\,}\mathrm{\%}

intervals and, on the right, the AUCagainst the exact counterfactual, by which the rows are ordered. The two orderings disagree. Re-exposure cohorts (E) are near-duplicates of trained text; F, compared across surface forms, fails its placebo test; X is synthetic.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Cohort	Source	Pairs	$r$	Tokens
M	arXiv, 11 months, 5 categories	$1\,986$	1	17.2M
L	arXiv prefixes, $5128\,192\text{\,}\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{s}$	$250$	1	0.8M
W new	Wikipedia, created after cutoff	$1\,000$	1	0.8M
W gap	Wikipedia, absent from snapshot	$600$	1	0.8M
S se	StackExchange, 5 sites	$1\,000$	1	0.8M
D	arXiv, $r\in\{2,4,8,16\}$	$400$	2–16	25.8M

Appendix

TABLE I: Cohorts as built. Pairs, copies per document

r

and tokens injected (rows rounded to

0.1

M; the total is exact). The pooled analyzed set has

7\,326

pairs after exclusions.

Stage	$\hat{\tau}$	SD	$d_{z}$	auc	$\textsc{tpr}@1\%\textsc{fpr}$ (%)
Mid-training	$+0.0221$	$0.0309$	$0.71$	$0.767$	$3.8$
+ SFT	$+0.0155$	$0.0239$	$0.65$	$0.731$	$3.7$
+ SFT + DPO	$+0.0199$	$0.0343$	$0.58$	$0.708$	$4.8$
+ RLVR1	$+0.0196$	$0.0350$	$0.56$	$0.700$	$5.1$
+ RLVR2 (2k)	$+0.0196$	$0.0355$	$0.55$	$0.697$	$4.7$
+ RLVR2 (end)	$+0.0197$	$0.0355$	$0.55$	$0.699$	$4.9$

Appendix

TABLE II: The membership signal across post-training on verified-unseen single-exposure pairs. Stages are cumulative. RLVR1 is the end of the first RLVRstage; RLVR2 (2k) is step 2000 of the second stage, the checkpoint corresponding to the released OLMo 2 1B Instruct, and RLVR2 (end) is that stage’s final step (2604).

Exact counterfactual
Attack	auc	$\textsc{tpr}@1\%\textsc{fpr}$ (%)
Ratio + Min-K%	$0.816$	$5.0$
Standardized ratio	$0.778$	$8.8$
Ratio	$0.767$	$3.8$
Ratio + Min-K%++	$0.665$	$1.6$
Retrained reference

Appendix

TABLE III: Attack AUCand true-positive rate at

1\text{\,}\mathrm{\%}

FPR , grouped by the reference the adversary holds; verified-unseen single-exposure pairs against their matched partners.

Attack	auc [95% CI]		$\textsc{tpr}@1\%\textsc{fpr}$ [95% CI]		per cohort
Exact counterfactual
Ratio	$0.767$	[.759,.774]	$3.8$	[2.9,4.8]	$16.5$
Standardized ratio	$0.778$	[.771,.783]	$8.8$	[7.3,10.9]	$31.4$
Retrained reference
Ratio	$0.745$	[.737,.753]	$3.7$	[2.8,4.8]	$8.2$
Standardized ratio	$0.725$	[.718,.731]	$8.1$	[6.0,9.5]	$20.2$

Appendix

TABLE IV: The reference ratio and the standardized ratio. Every row is evaluated on the same

4\,832

verified-unseen single-exposure pairs, with pair-bootstrap

95\text{\,}\mathrm{\%}

intervals; “per cohort” is the TPRat

1\text{\,}\mathrm{\%}

FPRwith one threshold per cohort.

Cohort	$n$	$r$	$\hat{\tau}$	auc	tpr (%)
D32	$40$	32	$+0.7258$	$1.000$	$100.0$
S se	$1\,000$	1	$+0.0251$	$0.962$	$52.0$
D	$400$	2–16	$+0.1508$	$0.943$	$56.2$
DB	$60$	4	$+0.0757$	$0.921$	$21.7$
W gap	$600$	1	$+0.0178$	$0.900$	$12.7$
W new	$1\,000$	1	$+0.0182$	$0.874$	$8.5$

Appendix

TABLE V: Per-cohort effect and detectability against the exact counterfactual, with the

1\text{\,}\mathrm{\%}

FPRthreshold set within each cohort;

r

is copies per document and

n

is pairs after exclusions.

	Mid-training	SFT	DPO	RLVR(each stage)
Start	stage-1 endpoint, with optimizer state	mid-training endpoint	final SFTmodel	final DPOmodel; final RLVR1 model
Data	$576$ stage-2 files, seed-42 order	allenai/tulu-3-sft-olmo-2-mixture-0225 @ d91a0785	allenai/olmo-2-0425-1b-preference-mix @ c6454f3d	allenai/RLVR-GSM-MATH-IF-Mixed-Constraints @ 7dbd180f ; allenai/RLVR-MATH @ bd2a9355
Size	$12\,212\,224$ of $12\,570\,760$ chunks	$861\,812$ examples	$371\,767$ pairs	$29\,689$ ; $7\,499$ prompts
Max. length	$4\,096$	$4\,096$	$2\,048$	$2\,048$ prompt, $2\,048$ response
Objective	cross-entropy, z-loss $10^{-4}$ (configured $10^{-5}$ )	cross-entropy on assistant tokens, mean per micro-batch	length-normalized DPO, $\beta=5$	policy gradient with group-normalized advantages, Kullback–Leibler coefficient $0.01$ ; reward $10\times$ verifier score
Learning rate	$7.4487\text{\times}{10}^{-5}$ , linear to 0, no warm-up	$3\text{\times}{10}^{-5}$ , linear to 0, $3\text{\,}\mathrm{\%}$ warm-up	$2.5\text{\times}{10}^{-6}$ , linear to 0, $10\text{\,}\mathrm{\%}$ warm-up	$5\text{\times}{10}^{-7}$ , constant

Appendix

TABLE VI: Training settings. Our mid-training run follows AI2’s published stage-2 configuration; post-training settings are the same for the treated model and the control. Dataset revisions are the first eight hexadecimal digits of the Hugging Face commit.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 296 papers

Oct 2, 2026

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. Despite recent progress, existing benchmarks suffer from three limitations: limited coverage of training stages, insufficient distributional alignment between members and non-members, and lack of rigorous filtering of non-members against the training corpus. To address these limitations, we propose OLMo-Detect, a multi-stage, confounder-controlled benchmark built upon the fully open OLMo 2 pipeline. OLMo-Detect spans pre-training, mid-training, and post-training, explicitly aligns members and non-members on three key axes, and rigorously filters non-members via infini-gram. To assess robustness to distribution shifts, we further introduce OLMo-Detect (Shifted), a variant where members are misaligned with non-members. We evaluate 15 unsupervised and 3 supervised membership inference attacks (MIAs) across the OLMo 2 family, finding that: (i) overall performance is limited: the best unsupervised and supervised MIAs both reach an AUC of only 0.68, and supervised MIAs degrade under cross-domain evaluation; (ii) MIA performance peaks at mid-training and is lower at pre-training and post-training, a pattern driven by data type rather than a stage effect: curated math data is far more detectable than other types; (iii) overall scores improve from 1B to 13B but plateau at 32B; and (iv) no unsupervised MIA is robust to distribution shifts, with AUCs shifting by up to 0.42. Finally, we find that our findings on OLMo 2 generalize to OLMo 3 and non-OLMo models.

Tao Shi, Chaoyi Xiang, Qiongkai Xu +1

The University of Melbourne · Macquarie University

Membership Inference Attacks Privacy Leakage in Language Models LLM Auditing

Ask PDF

Sep 9, 2026

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models

When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each. Those counts make three questions answerable directly. The answers form a pincer, closing from two sides. At the duplication levels ordinary text actually has, five models from 1B to 13B parameters carry at most a faint trace of their own exposure. We measure that trace with a design that reads the same sentence through two models, which cancels fluency and quality by construction, and it comes to a rank correlation near -0.08, where -1 would be a perfect relation and 0 none. Where the trace does become strong, above roughly a thousand copies, the two corpora agree on which sentences those are, because they are the famous ones, so exposure can no longer be told apart from fame. Two further measurements show how apparent membership signal gets manufactured. A common way to build a non-member is to change one word of a member. The model does prefer the original, but the gap is the same, within noise, whether the original appeared once or a hundred times, so what the model is rewarding is the author's word choice, not memory. Above a thousand copies the gap grows with model size on the twelve sentences we can test there, at the same boundary where the pincer closes. And swapping the controls for sentences that differ from the members in register moves a detector from 0.83 to 0.94 AUC, on a scale where 0.5 is a coin flip and 1.0 is perfect separation. We release the sentence banks, counts, and code.

Arman Nik Khah

The University of Texas at Dallas

Memorization in Language Models LLM Auditing Data Repetition in Language Model Training

Ask PDF

Jun 16, 2026

CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models

Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques. In particular, subtle distribution shifts between member and non-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. Therefore, all open-source models with intermediate checkpoints and public training data can be converted into MIA testbeds. We apply our framework to a half-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters. To facilitate further privacy research, we open-source a modular library for designing and implementing attacks in this setting: https://github.com/safr-ai-lab/pandora_llm.

Jeffrey G. Wang, Jason Wang, Marvin Li +1

Harvard University · Harvard Business School

LLM Evaluation Membership Inference Attacks LLM Auditing

Ask PDF

Abstract

Privacy Leakage in Language Models

Explore similar work

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models

CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models