Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. Despite recent progress, existing benchmarks suffer from three limitations: limited coverage of training stages, insufficient distributional alignment between members and non-members, and lack of rigorous filtering of non-members against the training corpus. To address these limitations, we propose OLMo-Detect, a multi-stage, confounder-controlled benchmark built upon the fully open OLMo 2 pipeline. OLMo-Detect spans pre-training, mid-training, and post-training, explicitly aligns members and non-members on three key axes, and rigorously filters non-members via infini-gram. To assess robustness to distribution shifts, we further introduce OLMo-Detect (Shifted), a variant where members are misaligned with non-members. We evaluate 15 unsupervised and 3 supervised membership inference attacks (MIAs) across the OLMo 2 family, finding that: (i) overall performance is limited: the best unsupervised and supervised MIAs both reach an AUC of only 0.68, and supervised MIAs degrade under cross-domain evaluation; (ii) MIA performance peaks at mid-training and is lower at pre-training and post-training, a pattern driven by data type rather than a stage effect: curated math data is far more detectable than other types; (iii) overall scores improve from 1B to 13B but plateau at 32B; and (iv) no unsupervised MIA is robust to distribution shifts, with AUCs shifting by up to 0.42. Finally, we find that our findings on OLMo 2 generalize to OLMo 3 and non-OLMo models.
Figures & tables
Benchmark
Stage(s)
Domain(s)
Member
Non-Member
Assessments
C
A
F
WikiMIA ( Shi et al., 2023 )
Pre-training
Wikipedia
Pre-2017 data
Post-2023 data
✗
✗
✗
StackMIA ( Ye et al., 2024 )
Pre-training
Stack Exchange
Pre-2017 data
Post-2023 data
✗
✗
✗
DETCON ( Dong et al., 2024 )
Post-training
HumanEval, GSM8K
Outputs of post-SFT LLMs
Outputs of base LLMs
✗
✗
✗
MIMIR ( Duan et al., 2024 )
Pre-training
The Pile: 7 domains
Train splits
Held-out data
✗
✗
✓ †
PatentMIA ( Zhang et al., 2024b )
Pre-training
Google Patents
Pre-2023 data
Post-2024 data
✗
✗
✗
Table 1: Comparison of existing benchmarks for membership inference on LLMs. We assess on three criteria: (i) Coverage (C) of training stages (pre-training, mid-training, and post-training); (ii) Explicit Alignment (A) of distributions between members and non-members; (iii) Rigorous Filtering (F) of non-members against the training corpus. ✓ † indicates that MIMIR uses a filter with a permissive threshold of 80%, leaving potentially high residual overlap. ✓ ‡ means that OLMoMIA aligns on semantic similarity, but most MIAs target exact wording rather than meaning.
Stage
Domain
Subsets
Split
Samples
Tokens
Quality
Temporal
Lexical
Pre-training
DCLM-Baseline
NM
150
217,117
FastText: 0.01809 ± 0.00007
2018-01-19 ± 930d
0.07
M
150
217,016
FastText: 0.01811 ± 0.00000
2018-01-18 ± 940d
peS2o
NM
268
216,981
—
2022-12-14 ± 11d
0.10
M
268
217,230
—
2022-12-15 ± 13d
OpenWebMath
NM
151
217,183
—
2019-10-13 ± 853d
0.11
M
151
217,252
—
2019-10-17 ± 864d
Table 2: Overall per-domain statistics of OLMo-Detect . Within each domain, members and non-members are aligned on up to three axes: Text Quality , Temporal Range, and Lexical Similarity. Text Quality uses OLMo 2’s metric and is enforced for quality-thresholded domains; Temporal Range is enforced when timestamps are available; and Lexical Similarity, measured by Jensen–Shannon divergence (lower is better), is enforced for all domains. “—” denotes uncontrolled axes. NM : non-member; M : member. Subsets lists constituent subsets; blanks indicate single-subset domains whose subset shares the domain’s name. † SFT and DPO are constructed per model size due to differing training data; values shown are for 1B. Each domain is split into dev/test sets at approximately 1:9 sample and token ratios; only totals are shown. Full statistics are in Appendix D .
Pre-training
Mid-training
Post-training
DCLM-Baseline
OpenWebMath
peS2o
StarCoder
Math Mix
Stack Exchange
SFT
DPO
RLVR
Avg
Overlap
4.07%
8.33%
3.11%
9.31%
6.65%
5.71%
6.32%
2.20%
2.06%
5.31%
BoW AUC
0.49
0.52
0.48
0.50
0.52
0.48
0.51
0.49
0.51
0.50
Table 3: Per-domain verification results, averaged across model sizes. Overlap : average 13-gram overlap of each domain’s non-members against OLMo 2’s training corpus. BoW AUC : blind detectability, measured by the AUC of a BoW classifier. Per-model-size results are in Tables 22 and 24 .
Method
1B
7B
13B
32B
Model-Avg
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
PPL L
0.54
0.64
0.61
0.60
0.55
0.72
0.68
0.65
0.56
0.79
0.66
0.67
0.58
0.80
0.68
0.69
0.56
0.74
0.66
0.65
Zlib L
0.57
0.67
0.66
0.63
0.59
0.76
0.66
0.67
0.59
0.83
0.67
0.70
0.61
0.85
0.69
0.72
0.59
0.78
0.67
0.68
Lower L
0.57
0.61
0.59
0.59
0.57
0.69
0.62
0.63
0.58
0.75
0.65
0.66
0.58
0.73
0.67
0.66
0.58
0.69
0.63
0.63
MinK L
0.55
0.65
0.62
0.61
0.56
0.73
0.67
0.65
0.57
0.80
0.66
0.68
0.59
0.81
0.69
0.70
0.57
0.75
0.66
0.66
MinK++ L
0.48
0.53
0.59
0.54
0.50
0.54
0.62
0.56
0.51
0.59
0.59
0.57
0.54
0.56
0.63
0.57
0.51
0.56
0.61
0.56
Table 4: AUCs of unsupervised MIAs on OLMo-Detect , grouped by family: L likelihood-based, P perturbation-based, Pr prompt-based, and O output-distribution-based. Pre / Mid / Post macro-average AUC over each stage’s domains, and All macro-averages all stages. Model-Avg averages across model sizes, and Method-Avg averages across methods. The best method per column is in bold. † SelfCrit is evaluated on post-training’s DPO and RLVR. Per-domain results are in Appendix I .
Method
Setting
1B
7B
13B
32B
Model-Avg
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Sup-CAMIA
In-domain
0.56
0.72
0.61
0.63
0.54
0.78
0.72
0.68
0.55
0.86
0.72
0.71
0.55
0.84
0.67
0.69
0.55
0.80
0.68
0.68
LODO
0.56
0.63
0.52
0.57
0.60
0.73
0.62
0.65
0.57
0.66
0.55
0.59
0.59
0.70
0.62
0.64
0.58
0.68
0.58
0.61
MIA-Tuner
In-domain
0.54
0.60
0.53
0.56
0.53
0.61
0.55
0.56
0.54
0.60
0.53
0.56
0.54
0.53
0.51
0.52
0.54
0.59
0.53
0.55
LODO
0.52
0.50
0.47
0.50
0.53
0.48
0.46
0.49
0.47
0.48
0.46
0.47
0.51
0.53
0.50
0.52
0.51
0.50
0.47
0.49
Table 5: Supervised MIAs on OLMo-Detect under In-domain and LODO settings. Top: AUCs of Sup-CAMIA and MIA-Tuner; the best method per column is in bold. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members; we therefore report its AUC gain ( Δ AUC) over the base MIA (in parentheses), averaged across stages. Base AUC : base MIA’s AUC averaged across model sizes; positive gains are bolded. Details are in Appendix K .
Pre-training
Mid-training
Post-training
DCLM
peS2o
OWM
StarCoder
Math Mix
SE
SFT
DPO
RLVR
Python
Java
GSM8K
Math SE
SO
Aya
WildChat
UF-WildChat
CoT-GSM8K
CoT-MATH
AUC (all MIAs)
0.56
0.52
0.54
0.56
0.59
0.71
0.59
0.60
0.57
0.60
0.50
0.70
0.63
AUC (eff. MIAs)
0.62
0.50
0.57
0.59
0.64
0.88
0.61
0.61
0.59
0.62
0.50
0.90
0.74
Dup. count
9.00
1.12
1.00
1.00
1.00
6.00
6.00
6.00
2.09
2.00
1.13
1.25
1.50
Table 6: Cross-stage analysis on OLMo-Detect . Top: Per-subset AUCs averaged across model sizes and over all MIAs (excluding FSD, which reports gain, and SelfCrit, which is evaluated on DPO/RLVR) or effective (eff.) MIAs (overall AUC ≥0.60 ); bold means AUC >0.70 . Dup. count : training-corpus occurrences per member in each subset. DCLM: DCLM-Baseline; OWM: OpenWebMath; SE: Stack Exchange; SO: Stack Overflow. Bottom: Stage-level AUCs with and without the three curated math subsets (GSM8K, CoT-GSM8K, CoT-MATH), and AUCs on the curated math subsets versus the others. Δ : average per-MIA AUC gain on curated math subsets over the others; parentheses report the number of MIAs with positive gains and the one-sided Wilcoxon p -value.
PPL
Zlib
Lower
MinK
MinK++
DCPDD
ReCaLL
CAMIA
PAC
Neigh.
EMMIA
DCQ
CDD
Guided
I / D
4 / 2
2 / 2
5 / 1
2 / 1
1 / 3
2 / 3
4 / 0
4 / 1
5 / 1
5 / 1
3 / 2
2 / 1
3 / 0
1 / 2
Largest Δ
+ 0.29
− 0.31
− 0.30
+ 0.23
− 0.16
+ 0.16
+ 0.24
+ 0.28
− 0.20
+ 0.12
+ 0.42
+ 0.09
+ 0.12
+ 0.10
(OWM)
(OWM)
(OWM)
(OWM)
(SC)
(SC)
(peS2o)
(peS2o)
(OWM)
(SC)
(OWM)
(peS2o)
(OWM)
(SC)
Table 7: Effect of distribution shift on unsupervised MIAs (excluding SelfCrit), over the seven domains with shifted variants. I / D : number of domains on which the AUC increases / decreases significantly under the shift ( Δ = OLMo-Detect (Shifted) − OLMo-Detect , averaged over model sizes; 95% paired instance bootstrap). Largest Δ : the largest change and the domain where it occurs. OWM: OpenWebMath; SC: StarCoder. Per-domain values are in Appendix O .
Domain
Comparison
PPL
Zlib
MinK
DCPDD
PAC
Avg
GSM8K
OLMo 2-7B / OLMo 3-7B
0.85 / 0.97
0.89 / 0.98
0.86 / 0.97
0.91 / 0.98
0.83 / 0.95
0.87 / 0.97
OLMo 2-32B / OLMo 3-32B
0.98 / 0.99
0.99 / 1.00
0.97 / 0.99
0.98 / 0.98
0.92 / 0.98
0.97 / 0.99
DCLM-Baseline
OLMo 2-1B / DCB-1B
0.60 / 0.63
0.59 / 0.61
0.57 / 0.60
0.57 / 0.60
0.61 / 0.63
0.59 / 0.61
OLMo 2-1B / SmolLM2-1.7B
0.60 / 0.62
0.59 / 0.62
0.57 / 0.59
0.57 / 0.61
0.61 / 0.64
0.59 / 0.62
OpenWebMath
OLMo 2-1B / DCB-1B
0.51 / 0.51
0.56 / 0.54
0.55 / 0.54
0.61 / 0.58
0.54 / 0.56
0.55 / 0.54
OLMo 2-7B / DCB-7B
0.51 / 0.53
0.57 / 0.56
0.55 / 0.55
0.57 / 0.59
0.56 / 0.57
0.55 / 0.56
Table 8: AUCs of the five best-performing unsupervised MIAs on OLMo 2 and other model families.
Appendix figures & tables77 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Type
Tokens
Words
Bytes
Docs
DCLM-Baseline
Web pages
3.71T
3.32T
21.32T
2.95B
StarCoder (filtered)
Code
83.0B
70.0B
459B
78.7M
peS2o
Academic papers
58.6B
51.1B
413B
38.8M
arXiv
STEM papers
20.8B
19.3B
77.2B
3.95M
OpenWebMath
Math web pages
12.2B
11.1B
47.2B
2.89M
Algebraic Stack
Math proofs/code
11.8B
10.8B
44.0B
2.83M
Appendix
Table 9: Composition of OLMo 2 Mix 1124, the pre-training corpus of the OLMo 2 suite, reproduced from Table 4 of OLMo et al. (2025) .
Domain
Type
Tokens
Words
Bytes
Docs
High-Quality Subset (five domains)
Filtered DCLM-Baseline
High-quality web
752B
670B
4.56T
606M
FLAN
Instruction data
17.0B
14.4B
98.2B
57.3M
peS2o
Academic papers
58.6B
51.1B
413B
38.8M
Wikipedia/Wikibooks
Encyclopedic
3.7B
3.16B
16.2B
6.17M
Stack Exchange
Q&A
1.26B
1.14B
7.72B
2.48M
Appendix
Table 10: Composition of Dolmino Mix 1124, the mid-training corpus of the OLMo 2 suite, reproduced from Table 5 of OLMo et al. (2025) . The High-Quality Subset contributes five domains directly; the Dolmino Math Mix is itself a single domain whose composition is given by seven subsets. “OWM-filtered” indicates that the subset is further filtered using the OpenWebMath classifier designed by OLMo et al. (2025) . † Although CodeSearchNet (OWM-filtered) is listed as part of the Dolmino Math Mix in OLMo et al. (2025) , it was not used in the final training.
Domain
Tokens
50B sample
100B sample
300B sample
Domain %
Mix %
Domain %
Mix %
Domain %
Mix %
Filtered DCLM-Baseline
752B
3.23
47.2
6.85
50.2
20.78
51.9
FLAN
17.0B
50.0
16.6
100
16.7
200
11.3
Stack Exchange
1.26B
100
2.45
200
2.47
400
1.68
peS2o
58.6B
5.15
5.85
16.7
9.52
100
19.4
Wikipedia/Wikibooks
3.7B
100
7.11
100
3.57
400
4.86
Appendix
Table 11: Per-domain composition of the three token-budget samples (50B, 100B, 300B) drawn from Dolmino Mix 1124 for mid-training, reproduced from Table 13 of OLMo et al. (2025) . For each domain: “Domain %” is the fraction of that domain’s full token count drawn into the sample, with values above 100 indicating repetition (e.g., 400 = 4 × repeat); “Mix %” is the domain’s share of the sample’s total tokens (each Mix % column sums to 100%).
Subset
Count
# in SFT
Sampleable
General
Tülu 3 Hardcoded
240
240
OpenAssistant
88,838
7,132
✓
No Robots
9,500
9,500
WildChat (GPT-4 subset)
241,307
100,000
✓
Knowledge Recall
Appendix
Table 12: Subsets of OLMo 2’s SFT mix, reproduced from Table 7 of Lambert et al. (2025) . “Count” is the total number of prompts available in the upstream source corpus; “# in SFT” corresponds to tulu-3-sft-olmo-2-mixture (7B and 13B; 939,344 prompts). The 1B and 32B models use tulu-3-sft-olmo-2-mixture-0225 (866,138 prompts), which draws from the same 19 subsets but further removes date-cutoff instructions and applies majority voting over 5 completions to the † -marked subsets. The Sampleable column marks subsets with non-member candidates available for OLMo-Detect : ✓ = sampleable remainder available; blank = source corpus fully consumed (# in SFT = Count).
Stage
Domain
Subset
Prompts ∗
1B
7B
13B
32B
Sampleable
Post-training
DPO
SFT Reused
117,025
✓
✓
✓
✓
∼
SFT IF
65,792
✓
✓
✓
✓
WildChat Unused ‡
84,105
✓
✓
✓
✓
✓
WildChat Reused ‡
17,703
✓
✓
✓
✓
✓
WildChat IF ‡
10,794
✓
✓
✓
✓
UltraFeedback (cleaned)
60,816
✓
✓
✓
✓
Appendix
Table 13: Prompt subsets for OLMo 2’s DPO stage across all four model sizes. ∗ Per-subset prompt counts are reproduced from Table 27 of OLMo et al. (2025) and apply to the 7B and 13B mixes; the 1B and 32B mixes draw from the same eight subset families with comparable but not identical per-subset counts. The 7B mix omits WildChat IF; chosen responses in the 1B and 32B mixes are additionally filtered to remove date-cutoff mentions. Subsets labeled “IF” contain prompts augmented with verifiable instruction-following constraints (IFEval-style). The Sampleable column marks subsets with uncontaminated candidates available for OLMo-Detect : ✓ = sampleable remainder available; blank = source corpus fully consumed; ∼ = unknown (SFT Reused aggregates prompts from many SFT source datasets with varying sampleability, making it impossible to determine remainder without per-source attribution). ‡ All three WildChat subsets draw from the same upstream source corpus and together count as one of the three sampleable sources ; the remaining two are DaringAnteater IF and Tülu 3 Personas IF.
Stage
Component
1B
7B
13B
32B
Pre-training
OLMo 2 Mix 1124
✓
✓
✓
✓
Mid-training
Dolmino Mix 1124, 50B sample
×1
×3
Dolmino Mix 1124, 100B sample
×3
×3
Dolmino Mix 1124, 300B sample
×1
×1
SFT
tulu-3-sft-olmo-2-mixture-0225
×2
×2
tulu-3-sft-olmo-2-mixture
×2
×2
Appendix
Table 14: Per-model training corpora for infini-gram indexing. Entries denote inclusion in the model’s indexed corpus, with multiplicities reflecting epoch and model-soup repetition (✓ = included once; ×N = included N times). ∗ Model-specific preference mixes: olmo-2-0425-1b-preference-mix (1B), olmo-2-1124-7b-preference-mix (7B), olmo-2-1124-13b-preference-mix (13B), olmo-2-0325-32b-preference-mix (32B). † The 1B model’s additional MATH-only RLVR pass is inferred from the OLMo 2 training documentation in the open-instruct repository ( https://github.com/allenai/open-instruct/blob/main/docs/olmo2.md ).
Stage
Domain
Subset
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Pre-training
DCLM-Baseline
NM
Dev
16
21,745
FastText: 0.01808 ± 0.00004
2018-01-18 ± 930d
0.07
Test
134
195,372
FastText: 0.01809 ± 0.00008
2018-01-19 ± 930d
Total
150
217,117
FastText: 0.01809 ± 0.00007
2018-01-19 ± 930d
M
Dev
16
21,721
FastText: 0.01811 ± 0.00000
2018-01-17 ± 933d
Test
134
195,295
FastText: 0.01811 ± 0.00000
2018-01-18 ± 941d
Total
150
217,016
FastText: 0.01811 ± 0.00000
2018-01-18 ± 940d
Appendix
Table 15: Detailed per-domain statistics for the pre-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set. The Subset column lists constituent subsets; blank entries indicate single-subset domains whose subset shares the domain’s name.
Stage
Domain
Subset
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Mid-training
Math Mix
GSM8K
NM
Dev
103
21,755
—
—
0.04
Test
937
195,408
—
—
Total
1,040
217,163
M
Dev
103
21,729
—
—
Test
937
195,403
—
—
Total
1,040
217,132
Appendix
Table 16: Sampling statistics for the mid-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set. The Subset column lists constituent subsets.
Stage
Domain
Subset
Model
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Post-training
SFT
WildChat, Aya
1B, 32B
NM
Dev
21
21,697
—
—
0.11
Test
204
195,360
—
—
Total
225
217,057
M
Dev
21
21,696
—
—
Test
204
195,463
—
—
Total
225
217,159
Appendix
Table 17: Detailed per-stage statistics for the post-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set. The Subset column lists constituent subsets; blank entries indicate single-subset domains whose subset shares the domain’s name.
Table 18: Detailed per-subset statistics for StarCoder in the pre-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set.
Table 19: Detailed per-subset statistics for Stack Exchange in the mid-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set.
Stage
Domain
Model
Subset
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Post-training
SFT
1B, 32B
Wildchat
NM
Dev
7
16,404
—
—
0.14
Test
64
148,039
—
—
Total
71
164,443
M
Dev
7
16,438
—
—
Test
64
148,115
—
—
Total
71
164,553
Appendix
Table 20: Detailed per-subset statistics for SFT in the post-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set.
Stage
Domain
Model
Subset
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Post-training
RLVR
1B, 7B, 13B, 32B
CoT-GSM8K
NM
Dev
14
11,835
—
—
0.01
Test
128
107,609
—
—
Total
142
119,444
M
Dev
14
11,796
—
—
Test
128
107,563
—
—
Total
142
119,359
Appendix
Table 21: Detailed per-subset statistics for RLVR in the post-training stage of OLMo-Detect . NM : non-member; M : member. Dev : development set; Test : test set.
Pre-training
Mid-training
Post-training
DCLM-Baseline
OpenWebMath
peS2o
StarCoder
Math Mix
Stack Exchange
SFT
DPO
RLVR
Avg
1B
4.07 ±4.87
8.33 ±5.69
3.11 ±4.01
9.31 ±5.24
6.65 ±4.77
5.71 ±5.34
6.08 ±5.24
2.12 ±3.01
2.06 ±4.55
5.27
7B
4.07 ±4.87
8.33 ±5.69
3.11 ±4.01
9.31 ±5.24
6.66 ±4.75
5.71 ±5.34
6.54 ±5.29
1.98 ±3.05
2.06 ±4.55
5.31
13B
4.07 ±4.87
8.33 ±5.69
3.11 ±4.01
9.31 ±5.24
6.65 ±4.75
5.71 ±5.34
6.55 ±5.30
2.52 ±3.37
2.06 ±4.55
5.37
32B
4.07 ±4.87
8.33 ±5.69
3.11 ±4.01
9.31 ±5.24
6.64 ±4.75
5.71 ±5.35
6.12 ±5.31
2.20 ±3.15
2.06 ±4.55
5.28
Avg
4.07
8.33
3.11
9.31
6.65
5.71
6.32
2.20
2.06
5.31
Appendix
Table 22: Per-model-size 13-gram overlap η (%; mean ±std over non-members) for each domain against the training corpus of the corresponding OLMo 2 model size. Multi-subset domains are macro-averaged across subsets (Table 23 ), with the domain-level standard deviation computed as the root mean square of the subset-level standard deviations.
Domain
Subset
1B
7B
13B
32B
Avg
StarCoder
Python
7.17 ±5.28
7.17 ±5.28
7.17 ±5.28
7.17 ±5.28
7.17
Java
11.45 ±5.20
11.45 ±5.20
11.45 ±5.20
11.45 ±5.20
11.45
Stack Exchange
Math Stack Exchange
5.67 ±4.91
5.67 ±4.91
5.67 ±4.91
5.68 ±4.92
5.67
Stack Overflow
5.74 ±5.74
5.74 ±5.74
5.74 ±5.75
5.75 ±5.74
5.74
SFT
WildChat
5.16 ±4.49
4.53 ±4.53
4.55 ±4.53
5.14 ±4.59
4.84
Aya
7.01 ±5.89
8.55 ±5.95
8.55 ±5.96
7.10 ±5.95
7.80
Appendix
Table 23: Per-subset 13-gram overlap η (%; mean ±std over non-members) for multi-subset domains. Domain-level values are macro-averaged across subsets; for DPO, they are averaged across the prompt, chosen, and rejected fields.
Pre-training
Mid-training
Post-training
DCLM-Baseline
OpenWebMath
peS2o
StarCoder
Math Mix
Stack Exchange
SFT
DPO
RLVR
Avg
1B
0.49
0.52
0.48
0.50
0.52
0.48
0.51
0.47
0.51
0.50
7B
0.49
0.52
0.48
0.50
0.52
0.48
0.52
0.51
0.51
0.50
13B
0.49
0.52
0.48
0.50
0.52
0.48
0.52
0.49
0.51
0.50
32B
0.49
0.52
0.48
0.50
0.52
0.48
0.51
0.48
0.51
0.50
Avg
0.49
0.52
0.48
0.50
0.52
0.48
0.51
0.49
0.51
0.50
Appendix
Table 24: Per-model-size blind detectability, measured by BoW classifier AUC. Multi-subset domains are macro-averaged across subsets, with per-subset AUCs reported in Table 25 .
Domain
Subset
1B
7B
13B
32B
Avg
StarCoder
Python
0.48
0.48
0.48
0.48
0.48
Java
0.51
0.51
0.51
0.51
0.51
Stack Exchange
Math Stack Exchange
0.47
0.47
0.47
0.47
0.47
Stack Overflow
0.49
0.49
0.49
0.49
0.49
SFT
WildChat
0.52
0.54
0.54
0.52
0.53
Aya
0.50
0.51
0.51
0.50
0.50
Appendix
Table 25: Per-subset blind detectability for multi-subset domains, measured by BoW classifier AUC.
Method
Reference
Repository
PPL
Yeom et al. (2018)
— (standard baseline)
Zlib
Carlini et al. (2021)
— (standard baseline)
Lower
Carlini et al. (2021)
— (standard baseline)
MinK
Shi et al. (2023)
https://github.com/swj0419/detect-pretrain-code
MinK++
Zhang et al. (2024a)
https://github.com/zjysteven/mink-plus-plus
DCPDD
Zhang et al. (2024b)
https://github.com/zhang-wei-chao/DC-PDD
Appendix
Table 26: Public repositories followed by our implementations of the compared MIAs. The final three rows are the supervised methods; all others are unsupervised.
Method
Hyperparameter
Search space
Selected value
1B
7B
13B
32B
MinK
k
{0.05,…,1.0}
0.2
MinK++
k
{0.05,…,1.0}
0.2
DCPDD
α
{0.001,…,0.5}
0.001
0.2
0.05
0.2
ReCaLL
# prefix shots n
{1,…,12}
10
3
7
12
CAMIA
p -value composition
{Edgington, Fisher, George}
George
Fisher
George
George
Appendix
Table 27: Tunable hyperparameters, search spaces, and the values selected by development-set AUC on OLMo-Detect .
Method
AUC
95% CI
PPL
0.65
[0.64, 0.67]
Zlib
0.68
[0.67, 0.70]
Lower
0.63
[0.62, 0.65]
MinK
0.66
[0.64, 0.67]
MinK++
0.56
[0.54, 0.57]
DCPDD
0.67
[0.65, 0.68]
Appendix
Table 28: Bootstrap 95% confidence intervals for the overall AUC of each unsupervised MIA on OLMo-Detect , i.e. the All column of Table 4 averaged over model sizes. Members and non-members are resampled with replacement within every subset and the full subset-to-domain-to-stage aggregation is recomputed on each of 2,000 resamples, so the interval applies to the aggregate rather than to any single evaluation. SelfCrit is omitted because it covers only post-training. Point estimates match Table 4 .
Method
Setting
AUC
95% CI
Sup-CAMIA
In-domain
0.68
[0.67, 0.68]
Sup-CAMIA
LODO
0.61
[0.60, 0.62]
MIA-Tuner
In-domain
0.55
[0.54, 0.56]
MIA-Tuner
LODO
0.49
[0.48, 0.50]
Appendix
Table 29: Bootstrap 95% confidence intervals for the overall AUC of the supervised MIAs on OLMo-Detect , i.e. the All column of Table 5 averaged over model sizes, computed as in Table 28 . FSD is omitted because it is reported as a gain over its base detector rather than as an absolute AUC. Point estimates match Table 5 .
Stage
Domain
Subsets
Split
Samples
Tokens
Quality
Temporal
Lexical
Pre-training
DCLM-Baseline
Non-member
150
217,117
FastText: 0.01809 ± 0.00007
2018-01-19 ± 930d
0.08
Member
150
217,295
FastText: 0.13129 ± 0.18279
2017-02-21 ± 1208d
peS2o
Non-member
268
216,981
—
2022-12-14 ± 11d
0.13
Member
268
217,039
—
2014-02-03 ± 2657d
OpenWebMath
Non-member
151
217,183
—
2019-10-13 ± 853d
0.32
Member
151
217,379
—
2011-10-01 ± 2936d
Appendix
Table 30: Overall statistics of OLMo-Detect (Shifted) , spanning seven of the nine domains of OLMo-Detect (GSM8K and RLVR are excluded, as neither has any shiftable axis). Non-members are unchanged from OLMo-Detect , while members are randomly sampled subject only to matched instance and token counts, leaving Text Quality , Temporal Range, and Lexical Similarity unaligned. “—” denotes an axis that does not apply to a given domain, i.e., no quality thresholding is applied, or no timestamp metadata exists. NM : non-member; M : member. The Subsets column lists constituent subsets; blank entries indicate single-subset domains whose subset shares the domain’s name. † SFT and DPO are constructed separately for each model size because their training data differ across sizes; the reported values correspond to the 1B model.
Stage
Domain
Subset
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Pre-training
DCLM-Baseline
NM
Dev
16
21,745
FastText: 0.01808 ± 0.00004
2018-01-18 ± 930.40d
0.08
Test
134
195,372
FastText: 0.01809 ± 0.00008
2018-01-19 ± 930.19d
Total
150
217,117
FastText: 0.01809 ± 0.00007
2018-01-19 ± 930.21d
M
Dev
16
21,745
FastText: 0.13136 ± 0.19309
2017-02-21 ± 1204.63d
Test
134
195,550
FastText: 0.13128 ± 0.18152
2017-02-21 ± 1208.34d
Total
150
217,295
FastText: 0.13129 ± 0.18279
2017-02-21 ± 1207.94d
Appendix
Table 31: Detailed per-domain statistics for the pre-training stage of OLMo-Detect (Shifted) . NM : non-member; M : member. Dev : development set; Test : test set. The Subset column lists constituent subsets; blank entries indicate single-subset domains whose subset shares the domain’s name.
Table 32: Detailed per-domain statistics for the mid-training stage of OLMo-Detect (Shifted) . NM : non-member; M : member. Dev : development set; Test : test set. The Subset column lists constituent subsets.
Stage
Domain
Subset
Model
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Post-training
SFT
WildChat, Aya
1B, 32B
NM
Dev
21
21,697
—
—
0.34
Test
204
195,360
—
—
Total
225
217,057
M
Dev
21
21,727
—
—
Test
204
195,511
—
—
Total
225
217,238
Appendix
Table 33: Detailed per-domain statistics for the post-training stage of OLMo-Detect (Shifted) . NM : non-member; M : member. Dev : development set; Test : test set. The Subset column lists constituent subsets.
Table 35: Detailed per-subset statistics for Stack Exchange in OLMo-Detect (Shifted) . NM : non-member; M : member. Dev : development set; Test : test set.
Stage
Domain
Model
Subset
Split
Partition
Samples
Tokens
Quality
Temporal
Lexical
Post-training
SFT
1B, 32B
Wildchat
NM
Dev
7
16,404
—
—
0.43
Test
64
148,039
—
—
Total
71
164,443
M
Dev
7
16,388
—
—
Test
64
148,120
—
—
Total
71
164,508
Appendix
Table 36: Detailed per-subset statistics for SFT in OLMo-Detect (Shifted) . NM : non-member; M : member. Dev : development set; Test : test set.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.60
0.59
0.60
0.57
0.46
0.57
0.61
0.46
0.52
0.60
0.64
0.54
0.50
0.50
0.55
7B
0.64
0.64
0.62
0.61
0.53
0.64
0.65
0.50
0.54
0.65
0.61
0.49
0.43
0.49
0.57
13B
0.65
0.65
0.63
0.63
0.52
0.64
0.68
0.62
0.53
0.66
0.62
0.47
0.47
0.51
0.59
32B
0.66
0.67
0.66
0.64
0.61
0.65
0.69
0.53
0.50
0.64
0.61
0.47
0.64
0.51
0.61
Model-Avg
0.64
0.64
0.63
0.61
0.53
0.62
0.66
0.53
0.52
0.64
0.62
0.49
0.51
0.50
0.58
Appendix
Table 37: Comparison of unsupervised MIAs on the DCLM-Baseline domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.47
0.49
0.48
0.50
0.51
0.52
0.48
0.51
0.63
0.50
0.49
0.44
0.49
0.50
0.50
7B
0.47
0.49
0.48
0.48
0.49
0.51
0.47
0.55
0.70
0.47
0.51
0.52
0.52
0.50
0.51
13B
0.48
0.50
0.50
0.49
0.50
0.51
0.51
0.51
0.70
0.47
0.51
0.51
0.54
0.50
0.52
32B
0.49
0.51
0.50
0.51
0.48
0.52
0.51
0.52
0.71
0.47
0.53
0.44
0.64
0.49
0.52
Model-Avg
0.48
0.50
0.49
0.50
0.50
0.51
0.49
0.52
0.69
0.48
0.51
0.48
0.55
0.50
0.51
Appendix
Table 38: Comparison of unsupervised MIAs on the peS2o domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.58
0.65
0.60
0.60
0.49
0.65
0.69
0.60
0.53
0.55
0.53
0.43
0.59
0.52
0.57
7B
0.60
0.65
0.62
0.61
0.50
0.67
0.68
0.60
0.53
0.54
0.53
0.55
0.46
0.54
0.58
13B
0.60
0.64
0.59
0.60
0.50
0.68
0.70
0.60
0.52
0.60
0.52
0.45
0.54
0.56
0.58
32B
0.60
0.63
0.58
0.62
0.48
0.69
0.70
0.56
0.64
0.61
0.52
0.58
0.45
0.54
0.59
Model-Avg
0.59
0.64
0.60
0.61
0.49
0.67
0.69
0.59
0.56
0.58
0.53
0.50
0.51
0.54
0.58
Appendix
Table 39: Comparison of unsupervised MIAs on the StarCoder domain of OLMo-Detect , evaluated via AUC (macro-averaged over its subsets). Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.51
0.56
0.61
0.55
0.48
0.61
0.54
0.44
0.54
0.56
0.54
0.47
0.52
0.51
0.53
7B
0.51
0.57
0.59
0.55
0.50
0.57
0.56
0.49
0.53
0.46
0.52
0.42
0.24
0.48
0.50
13B
0.52
0.58
0.61
0.56
0.53
0.63
0.58
0.48
0.52
0.45
0.55
0.44
0.42
0.49
0.53
32B
0.57
0.64
0.59
0.60
0.59
0.65
0.60
0.44
0.60
0.43
0.55
0.43
0.51
0.48
0.55
Model-Avg
0.53
0.59
0.60
0.56
0.52
0.61
0.57
0.46
0.55
0.47
0.54
0.44
0.42
0.49
0.53
Appendix
Table 40: Comparison of unsupervised MIAs on the OpenWebMath domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.73
0.76
0.72
0.76
0.57
0.69
0.72
0.32
0.50
0.75
0.64
0.49
0.46
0.52
0.62
7B
0.85
0.89
0.83
0.86
0.52
0.91
0.83
0.05
0.51
0.88
0.72
0.50
0.51
0.55
0.67
13B
0.98
0.99
0.93
0.99
0.58
0.99
0.96
0.01
0.48
0.99
0.89
0.48
0.53
0.68
0.75
32B
0.98
0.99
0.93
0.97
0.51
0.98
0.92
0.02
0.50
0.99
0.91
0.55
0.51
0.74
0.75
Model-Avg
0.89
0.91
0.85
0.89
0.55
0.89
0.86
0.10
0.50
0.90
0.79
0.51
0.50
0.62
0.70
Appendix
Table 41: Comparison of unsupervised MIAs on the GSM8K subset of OLMo-Detect ’s Math Mix domain, evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.55
0.59
0.50
0.54
0.49
0.53
0.59
0.54
0.56
0.58
0.55
0.54
0.54
0.51
0.54
7B
0.58
0.64
0.55
0.60
0.56
0.61
0.63
0.53
0.57
0.59
0.56
0.45
0.40
0.53
0.56
13B
0.60
0.67
0.56
0.62
0.61
0.67
0.66
0.66
0.65
0.61
0.58
0.49
0.64
0.52
0.61
32B
0.63
0.70
0.53
0.65
0.61
0.68
0.69
0.73
0.69
0.64
0.60
0.54
0.64
0.50
0.63
Model-Avg
0.59
0.65
0.53
0.60
0.57
0.62
0.64
0.61
0.62
0.61
0.57
0.51
0.55
0.51
0.59
Appendix
Table 42: Comparison of unsupervised MIAs on the Stack Exchange domain of OLMo-Detect , evaluated via AUC (macro-averaged over its subsets). Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.62
0.64
0.49
0.63
0.73
0.56
0.51
0.61
0.57
0.49
0.54
0.48
0.51
0.51
0.56
7B
0.73
0.70
0.55
0.71
0.77
0.70
0.59
0.74
0.59
0.74
0.60
0.51
0.43
0.50
0.63
13B
0.65
0.61
0.50
0.64
0.70
0.60
0.55
0.65
0.55
0.57
0.51
0.54
0.44
0.51
0.57
32B
0.64
0.64
0.50
0.65
0.69
0.60
0.51
0.69
0.55
0.54
0.52
0.44
0.45
0.50
0.57
Model-Avg
0.66
0.65
0.51
0.66
0.72
0.61
0.54
0.67
0.56
0.58
0.54
0.49
0.46
0.51
0.58
Appendix
Table 43: Comparison of unsupervised MIAs on the SFT domain of OLMo-Detect , evaluated via AUC (macro-averaged over its subsets). Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
1B
0.44
0.55
0.64
0.44
0.45
0.47
0.62
0.54
0.60
0.59
0.54
0.53
0.45
0.53
0.43
0.52
7B
0.41
0.39
0.50
0.40
0.40
0.40
0.43
0.48
0.50
0.60
0.46
0.48
0.30
0.53
0.53
0.45
13B
0.38
0.47
0.64
0.38
0.45
0.39
0.53
0.59
0.49
0.57
0.49
0.45
0.50
0.53
0.43
0.49
32B
0.46
0.53
0.62
0.48
0.47
0.48
0.62
0.65
0.39
0.55
0.60
0.47
0.47
0.48
0.55
0.52
Model-Avg
0.42
0.48
0.60
0.43
0.44
0.43
0.55
0.57
0.49
0.58
0.52
0.48
0.43
0.52
0.48
0.50
Appendix
Table 44: Comparison of unsupervised MIAs on the UF-Wildchat subset of OLMo-Detect ’s DPO domain, evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
1B
0.77
0.79
0.64
0.79
0.60
0.74
0.69
0.36
0.66
0.68
0.51
0.50
0.43
0.48
0.50
0.61
7B
0.89
0.88
0.80
0.90
0.69
0.87
0.77
0.25
0.73
0.82
0.60
0.53
0.48
0.48
0.48
0.68
13B
0.95
0.94
0.81
0.95
0.62
0.92
0.75
0.18
0.76
0.85
0.58
0.47
0.50
0.46
0.46
0.68
32B
0.94
0.91
0.90
0.94
0.72
0.92
0.77
0.28
0.47
0.89
0.60
0.47
0.50
0.49
0.41
0.68
Model-Avg
0.89
0.88
0.79
0.90
0.66
0.86
0.75
0.27
0.65
0.81
0.57
0.49
0.48
0.48
0.46
0.66
Appendix
Table 45: Comparison of unsupervised MIAs on the RLVR domain of OLMo-Detect , evaluated via AUC (macro-averaged over its subsets). Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
Python
1B
0.54
0.61
0.58
0.55
0.50
0.61
0.69
0.52
0.49
0.58
0.52
0.38
0.57
0.52
0.55
7B
0.55
0.62
0.60
0.56
0.53
0.61
0.68
0.51
0.50
0.58
0.52
0.48
0.51
0.51
0.56
13B
0.55
0.62
0.58
0.56
0.52
0.62
0.67
0.56
0.51
0.62
0.51
0.48
0.54
0.52
0.56
32B
0.54
0.60
0.55
0.55
0.53
0.61
0.66
0.53
0.62
0.60
0.53
0.56
0.49
0.56
0.57
Model-Avg
0.54
0.62
0.58
0.55
0.52
0.61
0.68
0.53
0.53
0.60
0.52
0.48
0.53
0.53
0.56
Appendix
Table 46: Comparison of unsupervised MIAs on the subsets of the StarCoder domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
Math Stack Exchange
1B
0.55
0.60
0.49
0.54
0.49
0.55
0.62
0.57
0.58
0.60
0.57
0.60
0.59
0.52
0.56
7B
0.57
0.63
0.53
0.58
0.55
0.59
0.67
0.51
0.62
0.61
0.57
0.41
0.36
0.54
0.55
13B
0.60
0.67
0.55
0.62
0.62
0.68
0.71
0.63
0.69
0.64
0.60
0.45
0.63
0.51
0.61
32B
0.62
0.71
0.51
0.65
0.61
0.70
0.71
0.79
0.74
0.66
0.61
0.57
0.59
0.50
0.64
Model-Avg
0.59
0.65
0.52
0.60
0.57
0.63
0.68
0.63
0.66
0.63
0.59
0.51
0.54
0.52
0.59
Appendix
Table 47: Comparison of unsupervised MIAs on the subsets of the Stack Exchange domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
Aya
1B
0.56
0.58
0.50
0.54
0.67
0.54
0.50
0.48
0.58
0.54
0.54
0.49
0.48
0.50
0.53
7B
0.77
0.70
0.53
0.72
0.76
0.72
0.67
0.69
0.59
0.76
0.69
0.52
0.47
0.53
0.65
13B
0.69
0.60
0.49
0.66
0.64
0.62
0.60
0.56
0.51
0.67
0.59
0.51
0.49
0.55
0.59
32B
0.55
0.56
0.49
0.54
0.59
0.55
0.49
0.60
0.44
0.50
0.51
0.38
0.51
0.51
0.52
Model-Avg
0.64
0.61
0.51
0.61
0.66
0.61
0.57
0.58
0.53
0.62
0.58
0.47
0.49
0.52
0.57
Appendix
Table 48: Comparison of unsupervised MIAs on the subsets of the SFT domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
CoT-GSM8K
1B
0.85
0.89
0.73
0.88
0.62
0.80
0.78
0.21
0.80
0.75
0.49
0.51
0.42
0.51
0.47
0.65
7B
0.98
0.99
0.87
0.99
0.78
0.98
0.89
0.10
0.97
0.98
0.62
0.51
0.50
0.53
0.50
0.75
13B
1.00
1.00
0.90
1.00
0.64
0.99
0.78
0.01
1.00
0.98
0.56
0.48
0.41
0.49
0.39
0.71
32B
1.00
1.00
0.95
1.00
0.79
1.00
0.87
0.04
0.50
0.99
0.61
0.49
0.51
0.50
0.35
0.71
Model-Avg
0.96
0.97
0.86
0.97
0.71
0.94
0.83
0.09
0.82
0.93
0.57
0.50
0.46
0.51
0.43
0.70
Appendix
Table 49: Comparison of unsupervised MIAs on the subsets of the RLVR domain of OLMo-Detect , evaluated via AUC. Methods are grouped by family as in Table 4 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR.
Model
Stage
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
1B
Pre
0.07
0.09
0.08
0.09
0.07
0.12
0.12
0.07
0.10
0.09
0.05
0.03
0.07
0.06
—
0.08
Mid
0.14
0.17
0.15
0.15
0.09
0.10
0.14
0.07
0.09
0.15
0.10
0.06
0.07
0.06
—
0.11
Post
0.20
0.18
0.12
0.20
0.15
0.12
0.13
0.10
0.09
0.17
0.05
0.05
0.04
0.07
0.07
0.12
All
0.13
0.15
0.12
0.15
0.10
0.11
0.13
0.08
0.09
0.14
0.07
0.05
0.06
0.06
0.07
0.10
7B
Pre
0.08
0.13
0.10
0.11
0.09
0.15
0.13
0.08
0.18
0.12
0.06
0.05
0.05
0.06
—
0.10
Appendix
Table 50: Comparison of unsupervised MIAs on OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family: likelihood-based, perturbation-based ( Pert. ), prompt-based and output-distribution-based ( Out.-Dist. ). Pre / Mid / Post report macro-averaged TPRs over the domains of each stage, while All macro-averages the three stages. Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR and thus has only Post and All values, averaged over these two domains. Per-domain results are provided in Tables 51 – 63 .
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.09
0.06
0.14
0.13
0.01
0.12
0.16
0.03
0.06
0.13
0.09
0.04
0.03
0.06
0.08
7B
0.15
0.11
0.16
0.21
0.06
0.18
0.20
0.03
0.07
0.22
0.08
0.06
0.04
0.04
0.12
13B
0.16
0.13
0.14
0.20
0.05
0.15
0.26
0.06
0.08
0.23
0.10
0.04
0.03
0.06
0.12
32B
0.21
0.18
0.13
0.22
0.11
0.18
0.34
0.06
0.12
0.26
0.11
0.04
0.07
0.07
0.15
Model-Avg
0.15
0.12
0.14
0.19
0.06
0.16
0.24
0.04
0.08
0.21
0.10
0.05
0.04
0.06
0.12
Appendix
Table 51: Comparison of unsupervised MIAs on the DCLM-Baseline domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.06
0.07
0.05
0.09
0.07
0.07
0.06
0.05
0.18
0.06
0.04
0.05
0.04
0.05
0.07
7B
0.07
0.07
0.06
0.07
0.07
0.07
0.07
0.05
0.47
0.09
0.05
0.03
0.10
0.05
0.09
13B
0.07
0.07
0.08
0.07
0.05
0.09
0.10
0.05
0.44
0.05
0.07
0.05
0.07
0.05
0.09
32B
0.08
0.09
0.08
0.10
0.08
0.10
0.09
0.07
0.48
0.05
0.06
0.05
0.11
0.04
0.10
Model-Avg
0.07
0.07
0.07
0.08
0.07
0.08
0.08
0.05
0.39
0.06
0.05
0.05
0.08
0.05
0.09
Appendix
Table 52: Comparison of unsupervised MIAs on the peS2o domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.08
0.17
0.07
0.08
0.13
0.16
0.17
0.17
0.09
0.11
0.05
0.01
0.16
0.07
0.11
7B
0.08
0.27
0.10
0.08
0.08
0.23
0.22
0.22
0.10
0.11
0.04
0.05
0.06
0.12
0.13
13B
0.09
0.24
0.08
0.09
0.07
0.20
0.27
0.09
0.11
0.10
0.07
0.03
0.01
0.07
0.11
32B
0.08
0.19
0.04
0.08
0.03
0.24
0.26
0.08
0.10
0.12
0.06
0.04
0.06
0.10
0.11
Model-Avg
0.08
0.22
0.07
0.08
0.08
0.21
0.23
0.14
0.10
0.11
0.06
0.03
0.07
0.09
0.11
Appendix
Table 53: Comparison of unsupervised MIAs on the StarCoder domain of OLMo-Detect , evaluated via TPR@5% FPR (macro-averaged over its subsets). Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.04
0.07
0.08
0.06
0.08
0.12
0.07
0.01
0.09
0.07
0.04
0.03
0.05
0.07
0.06
7B
0.04
0.07
0.07
0.06
0.14
0.12
0.02
0.01
0.09
0.06
0.07
0.04
0.00
0.04
0.06
13B
0.06
0.10
0.04
0.06
0.11
0.10
0.04
0.04
0.15
0.04
0.07
0.03
0.08
0.03
0.07
32B
0.10
0.17
0.04
0.07
0.15
0.13
0.11
0.03
0.10
0.04
0.07
0.02
0.04
0.02
0.08
Model-Avg
0.06
0.10
0.06
0.06
0.12
0.12
0.06
0.03
0.11
0.05
0.06
0.03
0.04
0.04
0.07
Appendix
Table 54: Comparison of unsupervised MIAs on the OpenWebMath domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.21
0.25
0.17
0.25
0.14
0.12
0.16
0.03
0.04
0.22
0.11
0.04
0.04
0.07
0.13
7B
0.43
0.54
0.21
0.48
0.13
0.60
0.37
0.00
0.05
0.49
0.22
0.05
0.06
0.07
0.26
13B
0.92
0.97
0.54
0.93
0.21
0.94
0.78
0.00
0.03
0.95
0.53
0.05
0.04
0.17
0.50
32B
0.89
0.96
0.61
0.84
0.14
0.92
0.67
0.00
0.06
0.96
0.63
0.12
0.06
0.24
0.51
Model-Avg
0.61
0.68
0.38
0.63
0.16
0.65
0.49
0.01
0.05
0.66
0.37
0.06
0.05
0.14
0.35
Appendix
Table 55: Comparison of unsupervised MIAs on the GSM8K subset of OLMo-Detect ’s Math Mix domain, evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.06
0.08
0.12
0.05
0.03
0.08
0.12
0.10
0.13
0.07
0.10
0.07
0.10
0.06
0.08
7B
0.06
0.12
0.09
0.08
0.06
0.18
0.12
0.03
0.09
0.06
0.10
0.05
0.05
0.06
0.08
13B
0.07
0.17
0.13
0.08
0.07
0.16
0.19
0.10
0.10
0.05
0.10
0.07
0.06
0.05
0.10
32B
0.09
0.20
0.17
0.13
0.12
0.21
0.25
0.16
0.17
0.07
0.10
0.10
0.14
0.05
0.14
Model-Avg
0.07
0.14
0.13
0.08
0.07
0.16
0.17
0.10
0.12
0.06
0.10
0.07
0.09
0.05
0.10
Appendix
Table 56: Comparison of unsupervised MIAs on the Stack Exchange domain of OLMo-Detect , evaluated via TPR@5% FPR (macro-averaged over its subsets). Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
1B
0.22
0.14
0.09
0.21
0.33
0.12
0.08
0.12
0.04
0.15
0.07
0.05
0.05
0.05
0.12
7B
0.40
0.29
0.13
0.36
0.50
0.28
0.17
0.17
0.07
0.22
0.06
0.06
0.02
0.06
0.20
13B
0.19
0.14
0.07
0.18
0.26
0.10
0.06
0.11
0.06
0.14
0.09
0.10
0.07
0.08
0.12
32B
0.28
0.15
0.07
0.32
0.33
0.19
0.05
0.17
0.07
0.21
0.07
0.08
0.00
0.08
0.15
Model-Avg
0.27
0.18
0.09
0.27
0.36
0.17
0.09
0.14
0.06
0.18
0.07
0.07
0.04
0.07
0.15
Appendix
Table 57: Comparison of unsupervised MIAs on the SFT domain of OLMo-Detect , evaluated via TPR@5% FPR (macro-averaged over its subsets). Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
1B
0.07
0.06
0.14
0.08
0.04
0.07
0.12
0.16
0.08
0.22
0.04
0.04
0.03
0.10
0.12
0.09
7B
0.02
0.03
0.07
0.01
0.02
0.01
0.01
0.07
0.09
0.07
0.04
0.05
0.00
0.10
0.04
0.04
13B
0.03
0.06
0.28
0.04
0.04
0.03
0.06
0.33
0.10
0.08
0.03
0.03
0.06
0.11
0.06
0.09
32B
0.04
0.09
0.09
0.06
0.07
0.04
0.13
0.19
0.01
0.06
0.03
0.03
0.07
0.07
0.11
0.07
Model-Avg
0.04
0.06
0.15
0.05
0.04
0.04
0.08
0.19
0.07
0.11
0.03
0.04
0.04
0.09
0.08
0.07
Appendix
Table 58: Comparison of unsupervised MIAs on the UF-Wildchat subset of OLMo-Detect ’s DPO domain, evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR.
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
1B
0.29
0.35
0.12
0.33
0.08
0.16
0.19
0.02
0.15
0.14
0.03
0.06
0.04
0.04
0.02
0.14
7B
0.66
0.65
0.33
0.66
0.16
0.60
0.25
0.02
0.51
0.53
0.11
0.04
0.05
0.03
0.05
0.31
13B
0.82
0.81
0.41
0.78
0.06
0.71
0.30
0.00
0.52
0.62
0.11
0.02
0.05
0.05
0.05
0.36
32B
0.71
0.68
0.43
0.74
0.15
0.71
0.34
0.03
0.04
0.67
0.17
0.03
0.06
0.05
0.02
0.32
Model-Avg
0.62
0.62
0.32
0.63
0.11
0.54
0.27
0.02
0.30
0.49
0.10
0.04
0.05
0.05
0.04
0.28
Appendix
Table 59: Comparison of unsupervised MIAs on the RLVR domain of OLMo-Detect , evaluated via TPR@5% FPR (macro-averaged over its subsets). Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
Python
1B
0.10
0.23
0.10
0.10
0.12
0.20
0.16
0.09
0.06
0.17
0.06
0.00
0.14
0.08
0.11
7B
0.10
0.22
0.12
0.10
0.12
0.19
0.23
0.09
0.08
0.13
0.06
0.07
0.05
0.02
0.11
13B
0.11
0.22
0.10
0.12
0.07
0.18
0.26
0.06
0.08
0.16
0.04
0.06
0.01
0.07
0.11
32B
0.10
0.14
0.03
0.10
0.07
0.17
0.22
0.09
0.10
0.16
0.04
0.04
0.06
0.13
0.10
Model-Avg
0.10
0.21
0.09
0.10
0.09
0.18
0.22
0.08
0.08
0.15
0.05
0.04
0.06
0.07
0.11
Appendix
Table 60: Comparison of unsupervised MIAs on the subsets of the StarCoder domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
Math Stack Exchange
1B
0.07
0.07
0.16
0.04
0.04
0.11
0.20
0.14
0.20
0.11
0.11
0.09
0.16
0.08
0.11
7B
0.05
0.07
0.07
0.07
0.02
0.25
0.16
0.04
0.14
0.08
0.11
0.02
0.02
0.05
0.08
13B
0.07
0.16
0.15
0.07
0.04
0.20
0.27
0.11
0.11
0.02
0.12
0.09
0.07
0.04
0.11
32B
0.11
0.25
0.21
0.14
0.11
0.25
0.39
0.23
0.20
0.05
0.11
0.11
0.12
0.05
0.17
Model-Avg
0.08
0.14
0.15
0.08
0.05
0.20
0.25
0.13
0.16
0.06
0.11
0.08
0.09
0.06
0.12
Appendix
Table 61: Comparison of unsupervised MIAs on the subsets of the Stack Exchange domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
Aya
1B
0.12
0.02
0.16
0.16
0.24
0.09
0.04
0.09
0.06
0.14
0.08
0.06
0.04
0.05
0.10
7B
0.48
0.27
0.12
0.40
0.35
0.26
0.20
0.18
0.09
0.41
0.07
0.10
0.05
0.11
0.22
13B
0.28
0.15
0.06
0.28
0.23
0.15
0.10
0.09
0.04
0.26
0.09
0.13
0.11
0.13
0.15
32B
0.17
0.04
0.11
0.14
0.10
0.06
0.04
0.22
0.07
0.08
0.10
0.04
0.01
0.14
0.09
Model-Avg
0.26
0.12
0.11
0.25
0.23
0.14
0.09
0.15
0.06
0.22
0.08
0.08
0.05
0.11
0.14
Appendix
Table 62: Comparison of unsupervised MIAs on the subsets of the SFT domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold.
Subset
Model
Likelihood-based
Pert.
Prompt
Out.-Dist.
Method- Avg
PPL
Zlib
Lower
MinK
MinK++
DCPDD
PAC
ReCaLL
EMMIA
CAMIA
Neigh.
Guided
DCQ
CDD
SelfCrit †
CoT-GSM8K
1B
0.45
0.55
0.22
0.43
0.10
0.19
0.30
0.00
0.25
0.14
0.01
0.06
0.04
0.03
0.02
0.19
7B
0.94
0.96
0.46
0.97
0.24
0.91
0.38
0.00
0.97
0.95
0.11
0.02
0.07
0.04
0.03
0.47
13B
1.00
1.00
0.59
1.00
0.09
0.96
0.41
0.00
1.00
0.97
0.06
0.03
0.02
0.05
0.02
0.48
32B
1.00
1.00
0.63
1.00
0.21
0.98
0.52
0.00
0.05
0.95
0.16
0.03
0.09
0.06
0.01
0.45
Model-Avg
0.85
0.88
0.48
0.85
0.16
0.76
0.41
0.00
0.57
0.75
0.09
0.03
0.05
0.04
0.02
0.40
Appendix
Table 63: Comparison of unsupervised MIAs on the subsets of the RLVR domain of OLMo-Detect , evaluated via TPR@5% FPR. Methods are grouped by family as in Table 50 . Model-Avg averages across model sizes, while Method-Avg averages the methods with a value in that row. The best-performing method in each row is highlighted in bold. † SelfCrit is evaluated only on DPO and RLVR.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.59
0.64
0.65
0.67
0.64
Sup-CAMIA
In-domain
0.51
0.55
0.54
0.56
0.54
LODO
0.55
0.64
0.58
0.57
0.58
MIA-Tuner
In-domain
0.45
0.45
0.53
0.55
0.50
LODO
0.47
0.51
0.49
0.47
0.49
Appendix
Table 64: Supervised MIAs on the DCLM-Baseline domain of OLMo-Detect , evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.49
0.49
0.50
0.51
0.50
Sup-CAMIA
In-domain
0.55
0.47
0.52
0.50
0.51
LODO
0.50
0.52
0.48
0.55
0.51
MIA-Tuner
In-domain
0.54
0.57
0.56
0.57
0.56
LODO
0.55
0.51
0.39
0.57
0.51
Appendix
Table 65: Supervised MIAs on the peS2o domain of OLMo-Detect , evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.56
0.57
0.58
0.64
0.59
Sup-CAMIA
In-domain
0.60
0.59
0.59
0.58
0.59
LODO
0.54
0.56
0.53
0.56
0.55
MIA-Tuner
In-domain
0.57
0.58
0.52
0.54
0.55
LODO
0.59
0.54
0.56
0.51
0.55
Appendix
Table 66: Supervised MIAs on the OpenWebMath domain of OLMo-Detect , evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.65
0.65
0.64
0.63
0.64
Sup-CAMIA
In-domain
0.57
0.54
0.56
0.56
0.56
LODO
0.64
0.69
0.67
0.67
0.67
MIA-Tuner
In-domain
0.61
0.54
0.53
0.48
0.54
LODO
0.46
0.55
0.44
0.50
0.49
Appendix
Table 67: Supervised MIAs on the StarCoder domain of OLMo-Detect (macro-averaged over its subsets), evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.76
0.89
0.99
0.99
0.91
Sup-CAMIA
In-domain
0.82
0.94
1.00
1.00
0.94
LODO
0.68
0.85
0.77
0.90
0.80
MIA-Tuner
In-domain
0.51
0.51
0.50
0.51
0.51
LODO
0.49
0.51
0.51
0.52
0.51
Appendix
Table 68: Supervised MIAs on the GSM8K subset of OLMo-Detect ’s Math Mix domain, evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.59
0.64
0.67
0.70
0.65
Sup-CAMIA
In-domain
0.61
0.61
0.72
0.68
0.66
LODO
0.59
0.61
0.56
0.50
0.56
MIA-Tuner
In-domain
0.69
0.72
0.70
0.55
0.66
LODO
0.51
0.46
0.44
0.54
0.49
Appendix
Table 69: Supervised MIAs on the Stack Exchange domain of OLMo-Detect (macro-averaged over its subsets), evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.64
0.70
0.61
0.64
0.65
Sup-CAMIA
In-domain
0.51
0.73
0.66
0.55
0.61
LODO
0.55
0.61
0.56
0.49
0.55
MIA-Tuner
In-domain
0.53
0.55
0.50
0.52
0.53
LODO
0.50
0.46
0.48
0.51
0.49
Appendix
Table 70: Supervised MIAs on the SFT domain of OLMo-Detect (macro-averaged over its subsets), evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.55
0.39
0.47
0.53
0.48
Sup-CAMIA
In-domain
0.60
0.58
0.58
0.57
0.58
LODO
0.40
0.42
0.40
0.44
0.42
MIA-Tuner
In-domain
0.49
0.55
0.56
0.47
0.52
LODO
0.41
0.44
0.42
0.48
0.44
Appendix
Table 71: Supervised MIAs on the UF-Wildchat subset of OLMo-Detect ’s DPO domain, evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.79
0.88
0.94
0.91
0.88
Sup-CAMIA
In-domain
0.73
0.86
0.91
0.90
0.85
LODO
0.62
0.83
0.68
0.94
0.77
MIA-Tuner
In-domain
0.56
0.54
0.53
0.54
0.54
LODO
0.51
0.48
0.48
0.52
0.50
Appendix
Table 72: Supervised MIAs on the RLVR domain of OLMo-Detect (macro-averaged over its subsets), evaluated via AUC. Top: AUCs of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its AUC gain ( Δ AUC) over the base MIA (in parentheses) on this domain. Base AUC : the base MIA’s AUC on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Pre
Mid
Post
All
Zlib (unsup.)
—
0.09
0.17
0.18
0.15
0.13
0.33
0.32
0.26
0.13
0.57
0.34
0.35
0.16
0.58
0.31
0.35
0.13
0.41
0.29
0.28
Sup-CAMIA
In-domain
0.08
0.24
0.19
0.17
0.07
0.38
0.30
0.25
0.08
0.57
0.36
0.34
0.06
0.56
0.27
0.30
0.07
0.44
0.28
0.26
LODO
0.07
0.09
0.08
0.08
0.06
0.22
0.25
0.18
0.07
0.07
0.11
0.08
0.10
0.24
0.28
0.21
0.07
0.16
0.18
0.14
MIA-Tuner
In-domain
0.07
0.13
0.10
0.10
0.08
0.20
0.09
0.12
0.04
0.15
0.08
0.09
0.06
0.07
0.06
0.06
0.06
0.14
0.08
0.09
LODO
0.07
0.05
0.05
0.06
0.04
0.07
0.05
0.05
0.04
0.05
0.06
0.05
0.05
0.06
0.06
0.06
0.05
0.06
0.05
0.05
Appendix
Table 73: Supervised MIAs on OLMo-Detect , evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on the evaluated domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Pre / Mid / Post macro-average over each stage’s domains and All macro-averages the three stages, as in Table 4 . Bottom: FSD’s gain ( Δ ) over its base MIA (in parentheses) under the All aggregation; Base TPR@5% FPR is the base MIA’s own score averaged across model sizes, and positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.06
0.11
0.13
0.18
0.12
Sup-CAMIA
In-domain
0.01
0.05
0.04
0.06
0.04
LODO
0.05
0.08
0.04
0.17
0.09
MIA-Tuner
In-domain
0.01
0.01
0.00
0.06
0.02
LODO
0.02
0.02
0.03
0.01
0.02
Appendix
Table 74: Supervised MIAs on the DCLM-Baseline domain of OLMo-Detect , evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.07
0.07
0.07
0.09
0.07
Sup-CAMIA
In-domain
0.08
0.04
0.05
0.08
0.06
LODO
0.05
0.03
0.06
0.05
0.05
MIA-Tuner
In-domain
0.09
0.08
0.06
0.09
0.08
LODO
0.07
0.12
0.01
0.13
0.08
Appendix
Table 75: Supervised MIAs on the peS2o domain of OLMo-Detect , evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.07
0.07
0.10
0.17
0.10
Sup-CAMIA
In-domain
0.10
0.06
0.10
0.04
0.07
LODO
0.05
0.09
0.03
0.09
0.06
MIA-Tuner
In-domain
0.05
0.09
0.06
0.05
0.06
LODO
0.11
0.01
0.06
0.07
0.06
Appendix
Table 76: Supervised MIAs on the OpenWebMath domain of OLMo-Detect , evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.17
0.27
0.24
0.19
0.22
Sup-CAMIA
In-domain
0.12
0.13
0.13
0.08
0.12
LODO
0.12
0.04
0.14
0.10
0.10
MIA-Tuner
In-domain
0.12
0.11
0.02
0.05
0.08
LODO
0.09
0.02
0.05
0.01
0.04
Appendix
Table 77: Supervised MIAs on the StarCoder domain of OLMo-Detect (macro-averaged over its subsets), evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.25
0.54
0.97
0.96
0.68
Sup-CAMIA
In-domain
0.39
0.69
0.97
0.97
0.76
LODO
0.13
0.38
0.07
0.43
0.25
MIA-Tuner
In-domain
0.06
0.05
0.06
0.06
0.06
LODO
0.05
0.07
0.05
0.06
0.06
Appendix
Table 78: Supervised MIAs on the GSM8K subset of OLMo-Detect ’s Math Mix domain, evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.08
0.12
0.17
0.20
0.14
Sup-CAMIA
In-domain
0.08
0.06
0.17
0.16
0.12
LODO
0.05
0.06
0.08
0.06
0.06
MIA-Tuner
In-domain
0.19
0.36
0.24
0.09
0.22
LODO
0.06
0.08
0.04
0.05
0.06
Appendix
Table 79: Supervised MIAs on the Stack Exchange domain of OLMo-Detect (macro-averaged over its subsets), evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.14
0.29
0.14
0.15
0.18
Sup-CAMIA
In-domain
0.09
0.20
0.14
0.08
0.13
LODO
0.06
0.23
0.05
0.10
0.11
MIA-Tuner
In-domain
0.09
0.10
0.07
0.04
0.07
LODO
0.05
0.03
0.07
0.03
0.04
Appendix
Table 80: Supervised MIAs on the SFT domain of OLMo-Detect (macro-averaged over its subsets), evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.06
0.03
0.06
0.09
0.06
Sup-CAMIA
In-domain
0.21
0.06
0.21
0.05
0.13
LODO
0.06
0.06
0.11
0.08
0.08
MIA-Tuner
In-domain
0.05
0.08
0.12
0.06
0.08
LODO
0.06
0.07
0.07
0.07
0.07
Appendix
Table 81: Supervised MIAs on the UF-Wildchat subset of OLMo-Detect ’s DPO domain, evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Method
Setting
1B
7B
13B
32B
Model-Avg
Zlib (unsup.)
—
0.35
0.65
0.81
0.68
0.62
Sup-CAMIA
In-domain
0.27
0.64
0.72
0.67
0.58
LODO
0.12
0.45
0.18
0.66
0.35
MIA-Tuner
In-domain
0.15
0.10
0.07
0.07
0.10
LODO
0.04
0.04
0.03
0.08
0.05
Appendix
Table 82: Supervised MIAs on the RLVR domain of OLMo-Detect (macro-averaged over its subsets), evaluated via TPR@5% FPR. Top: scores of Sup-CAMIA and MIA-Tuner under the In-domain setting (trained on this domain’s dev set) and the LODO setting (trained on the other eight domains’ dev sets); Zlib, the strongest unsupervised MIA, is repeated as a reference and is excluded from the bold, which marks the best supervised entry per column. Bottom: FSD measures the change in a base MIA’s score after fine-tuning the target model on non-members, so we report its gain ( Δ ) over the base MIA (in parentheses) on this domain. Base TPR@5% FPR : the base MIA’s own score on this domain averaged across model sizes; positive gains are bolded. Model-Avg averages across model sizes.
Domain
N
1B
7B
13B
32B
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
DCLM-Baseline
150
0.52
0.57
0.56
0.57
0.58
0.56
0.58
0.62
0.62
0.63
0.58
0.56
0.59
0.60
0.60
0.58
0.60
0.63
0.63
0.64
peS2o
268
0.52
0.53
0.55
0.55
0.55
0.52
0.55
0.55
0.55
0.55
0.49
0.50
0.49
0.49
0.48
0.49
0.49
0.50
0.49
0.49
OpenWebMath
151
0.54
0.57
0.60
0.62
0.62
0.52
0.55
0.58
0.61
0.61
0.52
0.57
0.58
0.60
0.60
0.55
0.60
0.61
0.63
0.63
StarCoder
156
0.61
0.63
0.65
0.65
0.66
0.59
0.61
0.65
0.67
0.67
0.59
0.61
0.64
0.65
0.66
0.61
0.64
0.66
0.66
0.68
Math Mix
1040
0.83
0.83
0.84
0.84
0.84
0.94
0.94
0.95
0.95
0.95
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Appendix
Table 83: AUC of Sup-CAMIA as a function of the labeled-data budget on OLMo-Detect . For each subset and model size, half of the members and half of the non-members are held out as a fixed evaluation set; Sup-CAMIA is trained on nested subsets of the remaining half containing 10%–50% of the subset’s members and non-members (10% equals the dev-set budget of the main tables). Entries are the mean AUC on the held-out half over five random splits, averaged over the domain’s subsets (DPO over the chosen and rejected responses); N is the number of members (equal to the number of non-members) in the domain. SFT and DPO members come from the size-specific post-training mixes (SFT: one mix shared by 1B and 32B and another by 7B and 13B; DPO: one preference mix per size), so their N varies across model sizes and is given as a range. Macro-Avg is the mean over the nine domains. At the subset level the standard deviation across splits is below 0.05 in 211 of 260 cases; it reaches 0.17 for the smallest subsets (at most 100 members) at the 10%–20% budgets and is at most 0.09 at budgets of 30% or more. Absolute values are not comparable to Table 5 , which uses the 90% test split.
Method
Given the domain
Domain unknown
AUC (macro)
Acc.
AUC (pooled)
Acc.
PPL L
0.60
0.57
0.56
0.56
Zlib L
0.64
0.59
0.59
0.57
Lower L
0.61
0.57
0.59
0.57
MinK L
0.61
0.58
0.58
0.57
MinK++ L
0.54
0.52
0.52
0.53
Appendix
Table 84: Two evaluation settings on OLMo-Detect , averaged over model sizes (1B–32B). Every domain is subsampled to 100 members and 100 non-members (20 random draws; multi-subset domains pooled over their subsets; methods, dev sets and hyperparameters exactly as in the main tables). Given the domain : AUC per domain, macro-averaged (the paper’s protocol), and accuracy with a decision threshold tuned within the evaluated domain (2-fold). Domain unknown : one AUC over the pooled instances of all nine domains, and accuracy with a single threshold tuned on the pooled data; supervised MIAs are retrained on the pooled dev sets of all nine domains. Thresholds maximise accuracy on their tuning data. Best per column and block in bold. † SelfCrit is evaluated on DPO and RLVR only. Accuracies are omitted for EMMIA, which reports the best of its four base scores per subset and thus has no single score to threshold and for SelfCrit.
Method
DCLM
peS2o
OWM
SC
SE
SFT
DPO
Largest
PPL
+ 0.06 ‡
+ 0.22 ‡
+ 0.29 ‡
+ 0.04 †
− 0.06 ‡
− 0.08 ‡
− 0.01
OWM
Zlib
+ 0.01
− 0.20 ‡
− 0.31 ‡
+ 0.25 ‡
+ 0.05 ‡
− 0.01
+ 0.01
OWM
Lower
+ 0.05 ‡
+ 0.06 ‡
− 0.30 ‡
+ 0.14 ‡
+ 0.04 ‡
+ 0.06 ‡
+ 0.01
OWM
MinK
+ 0.02
+ 0.21 ‡
+ 0.23 ‡
+ 0.03
− 0.02
− 0.06 ‡
+ 0.01
OWM
MinK++
− 0.06 ‡
− 0.03 †
− 0.02
− 0.16 ‡
+ 0.07 ‡
− 0.02
+ 0.01
SC
DCPDD
+ 0.03
+ 0.04 ‡
− 0.05 ‡
+ 0.16 ‡
+ 0.02
− 0.10 ‡
− 0.04 ‡
SC
Appendix
Table 85: Per-domain effect of distribution shift on unsupervised MIAs, Δ = AUC on OLMo-Detect (Shifted) minus AUC on OLMo-Detect , averaged over the four model sizes. Significance is assessed with a paired instance-level bootstrap: † the 95% interval excludes zero, ‡ the 99% interval excludes zero; significant entries are in bold. Largest : the domain carrying that MIA’s largest change. GSM8K and RLVR have no shifted variant. OWM: OpenWebMath; SC: StarCoder; SE: Stack Exchange. Aggregate counts are in Table 7 .
Large Language Models (LLMs) utilize large amounts of data for their training, some of which may come from copyrighted sources. Membership Inference Attacks (MIA) aim to detect those documents and whether they have been included in the training corpora of the LLMs. The black-box MIAs require a significant amount of data manipulation; therefore, their comparison is often challenging. We study state-of-the-art (SOTA) MIAs under the black-box assumptions and compare them to each other using a unified set of datasets to determine if any of them can reliably detect membership under SOTA LLMs. In addition, a new method, called the Familiarity Ranking, was developed to showcase a possible approach to black-box MIAs, thereby giving LLMs more freedom in their expression to understand their reasoning better. The results indicate that none of the methods are capable of reliably detecting membership in LLMs, as shown by an AUC-ROC of approximately 0.5 for all methods across several LLMs. The higher TPR and FPR for more advanced LLMs indicate higher reasoning and generalizing capabilities, showcasing the difficulty of detecting membership in LLMs using black-box MIAs.
Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques. In particular, subtle distribution shifts between member and non-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. Therefore, all open-source models with intermediate checkpoints and public training data can be converted into MIA testbeds. We apply our framework to a half-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters. To facilitate further privacy research, we open-source a modular library for designing and implementing attacks in this setting: https://github.com/safr-ai-lab/pandora_llm.
Large Language Models (LLMs) raise growing concerns about privacy leakage and copyright compliance. Membership inference is a key tool for assessing such risks, but existing studies mainly focus on whether specific samples or sample-based data units are used for training. We argue that LLMs exhibit a human-memory-like behavior: an LLM may not memorize a specific sample verbatim, yet it can accumulate and reveal knowledge about a real-world entity from scattered mentions. This analogy motivates us to examine whether an LLM can be interrogated like a human interviewee to reveal its exposure to entity-related information. Motivated by this question, we propose entity-level membership inference, which determines whether information related to a target entity is used in LLM training. We study this task in the practical label-only black-box setting, where only generated texts are observable. We formalize the task under clue, input, and model constraints, establish the necessary and sufficient conditions for its feasibility, and instantiate five interrogation strategies based on this formalization. The strategies use limited entity clues to construct prompts, elicit entity-related responses, and infer membership from semantic features among the generated texts. We construct entity-level datasets and adapt state-of-the-art sample-level label-only methods to the entity-level setting as baselines. Experiments on person entities show that our methods achieve AUC up to 0.97 and bring gains of 6.0%--17.5% in Balanced Accuracy over the best adapted baseline.