Understanding how language models were pretrained requires information about their data, processing procedures, and computational resources. However, it remains unclear whether successive releases preserve or expand earlier disclosure. We conduct a document-based audit of 41 model entries from 17 families using 24 pretraining-related indicators from the 2025 Foundation Model Transparency Index (FMTI). We examine overall scores and indicator profiles, tracking gains, losses, and retained disclosures across 24 adjacent comparisons in ten families. Scores decrease in 11 comparisons, increase in six, and remain unchanged in seven, with 38 indicator losses and 28 gains overall. Stable or improving scores can conceal losses: OLMo-2 and OLMo-3 receive identical totals despite three gains and three losses, while Llama-3.1 improves over Llama-2 while losing credit for two data-related criteria. Gaps also persist in corpus composition and training compute. These findings show that later releases do not consistently preserve earlier pretraining disclosure and motivate documentation that explicitly identifies changes and the applicability of inherited information.
Figures & tables
Figure 1: Pretraining disclosure across Llama releases, based on their technical reports ( Touvron et al., 2023a ; Touvron et al., 2023b ; Dubey et al., 2024 ) . Later reports do not provide an equivalent to Llama 1’s named-source sampling breakdown, although processing disclosure recovers in Llama 3.1.
Group
n
Mean
Median
Range
Open weights
27
5.19
5
1–10
Open source/reproducible
14
16.07
16
14–18
All entries
41
8.90
6
1–18
Table 1: Pretraining disclosure scores by operational openness group. Scores count requirements met out of 24; n counts releases or grouped variants.
Figure 2: Pretraining disclosure scores across selected family sequences. Scores are out of 24. Release months and years appear below the version labels; the original Falcon group spans May–September 2023. Entries are equally spaced by order, not elapsed time. Connecting lines do not imply direct model derivation. MM denotes multimodal.
Figure 3: Indicator gains and losses across 24 adjacent comparisons, grouped by (a) family and (b) subdomain. Parentheses give the number of comparisons per family. Orange indicates losses ( 1→0 ); blue indicates gains ( 0→1 ). Counts include repeated transitions and are not normalized by comparisons or indicators.
Comparison
Scores
Gained
Lost
Retained
OLMo 2 → 3
17→17
3
3
14
Llama 2 → 3.1
6→8
4
2
4
Table 2: Two illustrative release comparisons. Gained, lost, and retained denote 0→1 , 1→0 , and 1→1 transitions. Scores are out of 24.
Figure 4: Selected disclosure criteria for open-weight ( n=27 ) and open-source/reproducible ( n=14 ) entries. Bars show percentages meeting each criterion; labels give counts. A zero denotes an unmet requirement.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal states add predictive information without changing this ranking. Similar average scores conceal different vulnerabilities, and training history matters for auditing a released model.
Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can recognize and act on the fact their prior assistant messages have been inserted or edited, the effectiveness and validity of these methods could be compromised. We investigate whether frontier language models can distinguish between tampered and untampered assistant-side context, a capability we call prefill awareness. To do so, we construct a binary preference benchmark across three prefill mechanisms, filtering for cases where models show consistent stances. We find that frontier models show substantial prefill awareness: Claude Opus 4.5 detects prefills opposing its preferences in 9-35% of cases with a 0% false positive rate when prompted; additionally, models often revert towards baseline behavior without explicitly reporting that the prefill was foreign. Controlled ablations later also show that detection and resistance rely on different cues, where stylistic mismatch mainly affects whether models flag a prefill as foreign, while preference mismatch mainly affects whether they revert toward their baseline answer. We also examine more realistic agentic settings such as misalignment-continuation evaluations and SWE-bench trajectories, where frontier models sometimes disavow prefilled assistant turns in ways that depend strongly on dataset, task success, and hidden formatting artifacts. Our results indicate that prefill awareness is already a substantial confound for some prefill-based methods. We recommend that model developers track this capability in frontier systems.
Language Models (LMs) may acquire harmful knowledge, and yet feign ignorance of these topics when under audit. Inspired by the recent discovery of deception-related behaviour patterns in LMs, we aim to train classifiers that detect when a LM is actively concealing knowledge. Initial findings on smaller models show that classifiers can detect concealment more reliably than human evaluators, with gradient-based concealment proving easier to identify than prompt-based methods. However, contrary to prior work, we find that the classifiers do not reliably generalize to unseen model architectures and topics of hidden knowledge. Most concerningly, the identifiable traces associated with concealment become fainter as the models increase in scale, with the classifiers achieving no better than random performance on any model exceeding 70 billion parameters. Our results expose a key limitation in black-box-only auditing of LMs and highlight the need to develop robust methods to detect models that are actively hiding the knowledge they contain.
Dhananjay Ashok, Ruth-Ann Armstrong, Jonathan May
Information Sciences Institute, University of Southern California