What Carries Forward? Pretraining Disclosure Across Language Model Releases

Md Sakib Ul Islam; S M Mehedi Zaman

cs.CLOct 27, 2026

What Carries Forward? Pretraining Disclosure Across Language Model Releases

Authors: Md Sakib Ul Islam, S M Mehedi Zaman

Organizations: Victorian Institute of Technology, Australia · Rutgers University, USA

Abstract

Understanding how language models were pretrained requires information about their data, processing procedures, and computational resources. However, it remains unclear whether successive releases preserve or expand earlier disclosure. We conduct a document-based audit of 41 model entries from 17 families using 24 pretraining-related indicators from the 2025 Foundation Model Transparency Index (FMTI). We examine overall scores and indicator profiles, tracking gains, losses, and retained disclosures across 24 adjacent comparisons in ten families. Scores decrease in 11 comparisons, increase in six, and remain unchanged in seven, with 38 indicator losses and 28 gains overall. Stable or improving scores can conceal losses: OLMo-2 and OLMo-3 receive identical totals despite three gains and three losses, while Llama-3.1 improves over Llama-2 while losing credit for two data-related criteria. Gaps also persist in corpus composition and training compute. These findings show that later releases do not consistently preserve earlier pretraining disclosure and motivate documentation that explicitly identifies changes and the applicability of inherited information.

Figures & tables

Figure 1: Pretraining disclosure across Llama releases, based on their technical reports ( Touvron et al., 2023a ; Touvron et al., 2023b ; Dubey et al., 2024 ) . Later reports do not provide an equivalent to Llama 1’s named-source sampling breakdown, although processing disclosure recovers in Llama 3.1.

Group	$n$	Mean	Median	Range
Open weights	27	5.19	5	1–10
Open source/reproducible	14	16.07	16	14–18
All entries	41	8.90	6	1–18

Table 1: Pretraining disclosure scores by operational openness group. Scores count requirements met out of 24;

n

counts releases or grouped variants.

Figure 2: Pretraining disclosure scores across selected family sequences. Scores are out of 24. Release months and years appear below the version labels; the original Falcon group spans May–September 2023. Entries are equally spaced by order, not elapsed time. Connecting lines do not imply direct model derivation. MM denotes multimodal.

Figure 3: Indicator gains and losses across 24 adjacent comparisons, grouped by (a) family and (b) subdomain. Parentheses give the number of comparisons per family. Orange indicates losses (

1\rightarrow 0

); blue indicates gains (

0\rightarrow 1

). Counts include repeated transitions and are not normalized by comparisons or indicators.

Comparison	Scores	Gained	Lost	Retained
OLMo 2 $\rightarrow$ 3	$17\rightarrow 17$	3	3	14
Llama 2 $\rightarrow$ 3.1	$6\rightarrow 8$	4	2	4

Table 2: Two illustrative release comparisons. Gained, lost, and retained denote

0\rightarrow 1

1\rightarrow 0

, and

1\rightarrow 1

transitions. Scores are out of 24.

Figure 4: Selected disclosure criteria for open-weight (

n=27

) and open-source/reproducible (

n=14

) entries. Bars show percentages meeting each criterion; labels give counts. A zero denotes an unmet requirement.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Family / group	Entry	Order	Score	Primary report reference(s)
Llama / OW	Llama 1	1	10	https://arxiv.org/abs/2302.13971
Llama / OW	Llama 2	2	6	https://arxiv.org/abs/2307.09288
Llama / OW	Llama 3.1	3	8	https://arxiv.org/abs/2407.21783
Falcon / OW	Falcon 7B 40B 180B	1	10	https://arxiv.org/abs/2311.16867 https://huggingface.co/datasets/tiiuae/falcon-refinedweb
Falcon / OW	Falcon 2 11B	2	4	https://arxiv.org/abs/2407.14885
Falcon / OW	Falcon Mamba 7B	3	7	https://arxiv.org/abs/2410.05355

Appendix

Table 3: Selected entries and report references. Scores are out of 24.

ID	Indicator	OW	OS	All	Balanced (%)	Families
0	Data acquisition methods	5/27	14/14	19/41	66.7	13
1	Public datasets	1/27	8/14	9/41	37.3	7
2	Crawling	0/27	1/14	1/41	2.0	1
3	Usage data used in training	2/27	14/14	16/41	62.3	12
4	Notice of usage data used in training	1/27	14/14	15/41	60.3	11
5	Licensed data sources	1/27	13/14	14/41	54.4	10

Appendix

Table 4: All selected indicators: entry counts, equal-family prevalence, and family coverage.

Family	Earlier entry	Later entry	Scores	G	L	R	N
Llama	Llama 1	Llama 2	10–6	3	7	3	11
Llama	Llama 2	Llama 3.1	6–8	4	2	4	14
Falcon	Falcon 7B 40B 180B	Falcon 2 11B	10–4	0	6	4	14
Falcon	Falcon 2 11B	Falcon Mamba 7B	4–7	3	0	4	17
Falcon	Falcon Mamba 7B	Falcon H1	7–6	0	1	6	17
Gemma	Gemma 1	Gemma 2	5–5	0	0	5	19

Appendix

Table 5: All adjacent comparisons. G: gained; L: lost; R: retained; N: continued non-satisfaction.

Entry	Total	Acq.	Proc.	Prop.	Comp.	Meth.
Llama 1	10	2	3	1	2	2
Llama 2	6	1	0	2	1	2
Llama 3.1	8	0	3	1	2	2
Falcon 7B 40B 180B	10	1	3	2	2	2
Falcon 2 11B	4	0	0	1	1	2
Falcon Mamba 7B	7	0	3	1	1	2

Appendix

Table 6: Complete entry and subdomain scores. Denominators: 24, 7, 3, 5, 4, 3, 1, 1. (Part 1 of 2.)

Entry	Total	Acq.	Proc.	Prop.	Comp.	Meth.	Dep.
DeepSeek-V3.2 Base	4	0	0	1	0	2	1
DeepSeek-V4-Flash	3	0	0	1	0	2	0
Mistral 7B family	1	0	0	0	0	1	0
Mixtral 8x7B	1	0	0	0	0	1	0
Ministral 3	3	0	0	0	0	2	1
Pythia Suite	17	6	3	2	2	3	1

Appendix

Table 7: Complete entry and subdomain scores. Denominators: 24, 7, 3, 5, 4, 3, 1, 1. (Part 2 of 2.)

Entry	0	1	3	12	13	14	15	16
Llama 1	1	1	0	1	1	1	1	0
Llama 2	0	0	1	0	0	0	1	1
Llama 3.1	0	0	0	1	1	1	1	0
Falcon 7B 40B 180B	1	0	0	1	1	1	1	1
Falcon 2 11B	0	0	0	0	0	0	1	0
Falcon Mamba 7B	0	0	0	1	1	1	1	0

Appendix

Table 8: Final 41-entry binary indicator matrix. Indicators 0–16.

Entry	22	23	24	29	30
Llama 1	1	1	0	1	1
Llama 2	0	0	1	1	1
Llama 3.1	0	1	1	1	1
Falcon 7B 40B 180B	0	1	1	1	1
Falcon 2 11B	0	1	0	1	1
Falcon Mamba 7B	0	1	0	1	1

Appendix

Table 9: Final 41-entry binary indicator matrix. Indicators 17–45.

Relationship	All entries	Open weights	Open source/ reproducible
Details accompanying broader disclosures
Data size $\rightarrow$ language or domain composition	5/37 (13.5%)	2/23 (8.7%)	3/14 (21.4%)
Hardware or provider $\rightarrow$ compute usage	2/24 (8.3%)	0/11 (0.0%)	2/13 (15.4%)
Related disclosure criteria
Processing purpose $\rightarrow$ processing methods	24/28 (85.7%)	10/14 (71.4%)	14/14 (100.0%)
Processing methods $\rightarrow$ processing techniques	20/24 (83.3%)	6/10 (60.0%)	14/14 (100.0%)

Appendix

Table 10: Selected disclosure relationships. Each cell reports entries meeting both conditions divided by entries meeting the first condition.

Comparison	Scores	$\Delta$	G	L	N	R	Gained IDs	Lost IDs
Llama 1 $\to$ Llama 2	$10\to 6$	-4	3	7	11	3	3, 16, 24	0, 1, 12, 13, 14, 22, 23
Llama 2 $\to$ Llama 3.1	$6\to 8$	+2	4	2	14	4	12, 13, 14, 23	3, 16
Falcon 7B 40B 180B $\to$ Falcon 2 11B	$10\to 4$	-6	0	6	14	4	–	0, 12, 13, 14, 16, 24
Falcon 2 11B $\to$ Falcon Mamba 7B	$4\to 7$	+3	3	0	17	4	12, 13, 14	–
Falcon Mamba 7B $\to$ Falcon H1	$7\to 6$	-1	0	1	17	6	–	23
Gemma 1 $\to$ Gemma 2	$5\to 5$	+0	0	0	19	5	–	–

Appendix

Table 11: All ordered release comparisons and changed criteria. G: gained (

0\to 1

); L: lost (

1\to 0

); N: still unmet (

0\to 0

); R: retained (

1\to 1

). Changed criteria use original FMTI IDs; – denotes none. MM abbreviates multimodal.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 639 papers

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training
Oct 19, 2026Haonan Huang, Chengkun CaiLLM Sycophancy Language Model Post-Training
Princeton University · University College London
Prefill Awareness in Large Language Models
Jun 10, 2026Andy Wang, Parv Mahajan, David Demitri Africa +3Language Model Safety Evaluation LLM Auditing
UK AI Security Institute
Seamless Deception: Larger Language Models Are Better Knowledge Concealers
Mar 15, 2026Dhananjay Ashok, Ruth-Ann Armstrong, Jonathan MayDeception in Language Models LLM Auditing
Information Sciences Institute, University of Southern California

Oct 19, 2026J/K move · Enter open · S save

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Haonan Huang, Chengkun Cai

Princeton University · University College London

LLM Sycophancy Language Model Post-Training LLM Auditing

Open paper AskPDF arXiv

Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal states add predictive information without changing this ranking. Similar average scores conceal different vulnerabilities, and training history matters for auditing a released model.

Abstract

LLM Auditing

Explore similar work

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Prefill Awareness in Large Language Models

Seamless Deception: Larger Language Models Are Better Knowledge Concealers