Interpretability methods for neural networks are predominantly reactive: they analyse activations produced during specific forward passes, requiring known inputs to find hidden capabilities such as backdoors. We propose Weight Oracles, fine-tuned language models that diagnose properties of a target network by reading its raw weights directly, without behavioural testing. We investigate this paradigm in two phases. Phase I establishes feasibility: through a staged curriculum and an external chain-of-computation that delegates parameter-free operations to deterministic code, an explainer LLM learns to simulate the forward pass of small transformers from their weights, achieving 99% holdout accuracy on unseen targets. Phase II repurposes this infrastructure for safety auditing. We train an oracle on natural language diagnostic questions about weight anomalies using only benign pathologies as training signal, and evaluate it zero-shot on backdoors absent from training. The oracle achieves AUROC 0.93 on attention-routed backdoors and 0.81 across a diversified threat distribution including stealth and adversarially regularized variants. Hand-crafted statistical detectors are sharp on the threat models they implicitly target but collapse on threat-model shift, while the oracle remains uniformly competent across attack types. Scaling to realistic model sizes remains the principal open challenge.
Figures & tables
Figure 1: Weight Oracle architecture. Each weight matrix of the target transformer is sliced into rows; each row is projected from its native dimension d∈{8,16} into the LLM’s hidden dimension (2048) by a learned linear layer, then summed with a type embedding τ identifying the source matrix. The resulting weight tokens are concatenated with a natural-language question and processed by Qwen 2.5 3B. The same injection pipeline is used in both phases; the output head differs (Phase I: predicted forward-pass output; Phase II: text answer).
Figure 2: Per-attack-type AUROC breakdown across the four threat models.
Figure 3: Layer-wise probing: where backdoor detection emerges in the oracle.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: External chain-of-computation. Each forward pass of the target transformer is decomposed into five LLM passes, one per linear weight projection, interleaved with deterministic external operations for LayerNorm, attention output projection, GELU, biases, and residuals. The LLM learns only the dot-product primitive W⋅h ; all composition is handled outside it.
Category
Pathology
Construction
Weight signature
% mix
Baseline
Clean
Standard training, 200–2000 steps on random token → class task
Small-magnitude amplification with mild rank perturbations
Weak attention-pathway anomalies near the clean distribution
Shortcut
Dominant token
Mixed-objective training: normal task + “token t→ class c at any position”
Token-specific routing through WV,WO ; closest benign proxy to a backdoor
16
Soft trigger
Positional bias added to logits during training
Mild attention asymmetry; position-dependent class bias
Appendix
Table 1: Training pathology specification.
Stage
Identifier
Natural-language question
Weight property probed
A
trained_or_random
Is this model trained or randomly initialised?
Init vs. post-training weight statistics
accuracy_level
What is this model’s accuracy level?
Overall model competence
confidence_level
How confident is this model on average?
Output entropy / softmax peakedness
attn_value_magnitude
How large are the value and output projection weights of the attention layer?
∥WV∥F , ∥WO∥F
attn_vo_balance
How does the value/output pathway magnitude compare to the query/key pathway?
(∥WV∥+∥WO∥)/(∥WQ∥+∥WK∥)
attn_sv_concentration
How concentrated are the singular values of the attention value weights?
Top-SV / sum-SV of WV
Appendix
Table 2: The 17 oracle question types. Each row gives the stage identifier, the natural-language wording, and the probed weight-space property.
Figure 5: Clean Model (typical)
Figure 6: Backdoored Model (high confidence detection)
Figure 7: Backdoored Model (hardest case — missed)
Figure 8: Target model architecture.
Figure 9: Backdoor signature of pos0 backdoors: W V singular-value concentration against V/O pathway balance (left) and V/O Frobenius norms (right) for clean vs. backdoored models.
Figure 10: ROC curve for zero-shot backdoor detection on the pos0-only cohort.
Figure 11: Sim vs. scratch initialisation by threat model. Sim initialisation helps on V/O-routed attacks (pos0) but degrades sharply on pathway-shifted attacks (lastpos, stealth), consistent with the negative transfer discussed in Section 4.
Figure 12: Weight matrix type ablation: AUROC change from zeroing each matrix type.
Figure 13: Gradient attribution by weight type (left) and differential attribution between backdoored and clean models (right).
Figure 14: PCA of oracle hidden states: clean vs. backdoored separation (left), coloured by V/O norm ratio (right).
Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.
Recent defenses for safeguarding open-weight large language models (LLMs) are intended to prevent adversarial usage. Underlying these defenses is an assumption that new harmful behavior is learned through fine-tuning rather than elicited by jailbreaking the model. Yet, pretrained LLMs already encode substantial harmful knowledge across many domains, which raises an important question: can an adversary jailbreak safeguarded models, to achieve harmful usage without fine-tuning at all? In this paper, we show that open-weight safeguards are susceptible to simpler strategies that, despite being well known, have not been systematically evaluated against these safeguards. Specifically, we evaluate two low-cost attacks--abliteration and prefilling--that do not rely on gradient-based optimization. Across three harmfulness evaluation benchmarks (BeaverTails, HarmBench, and AdvBench), these attacks increase attack success rates against safeguarded open-weight models from below 10% to a range of 16%-96%. To mitigate this vulnerability, we introduce abliteration-resistant tuning (ART), which incorporates an abliteration-based objective into training. ART can be layered onto existing defenses and reduces the success rates of abliteration, prefilling, and their combination by 10%-20%. These findings indicate that the attack surface for open-weight models is broader than previously characterized, and that evaluations of safeguarding defenses should incorporate a more diverse set of attack strategies beyond adversarial fine-tuning.
Kevin Kuo, Virginia Smith, Chhavi Yadav
Carnegie Mellon University · Simons Institute, UC Berkeley
Autonomous systems increasingly rely on Large Language Models (LLMs) yet the safety infrastructure surrounding these models introduces latency and compute overhead. This limits utility in resource-constrained, time-critical deployments. Existing external guardrail models remain blind to the model's internal workings, creating a fundamental assurance gap. We ask: does the model already know when the content is harmful? We extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes (12.6M parameters) to detect harmful prompts. Evaluated on WildJailbreak, Beavertails, and AEGIS 2.0, our probes achieve F1 scores of 99%, 83%, and 84%, respectively competitive with 1000x larger guard models while cutting latency and compute costs.