Weight Oracles: Reading Neural Network Weights with Language Models

Krishna Kabra; Constantin Venhoff; Christian Schroeder de Witt

cs.LGOct 5, 2026

Weight Oracles: Reading Neural Network Weights with Language Models

Authors: Krishna Kabra, Constantin Venhoff, Christian Schroeder de Witt

Organizations: University of Oxford

Abstract

Interpretability methods for neural networks are predominantly reactive: they analyse activations produced during specific forward passes, requiring known inputs to find hidden capabilities such as backdoors. We propose Weight Oracles, fine-tuned language models that diagnose properties of a target network by reading its raw weights directly, without behavioural testing. We investigate this paradigm in two phases. Phase I establishes feasibility: through a staged curriculum and an external chain-of-computation that delegates parameter-free operations to deterministic code, an explainer LLM learns to simulate the forward pass of small transformers from their weights, achieving 99% holdout accuracy on unseen targets. Phase II repurposes this infrastructure for safety auditing. We train an oracle on natural language diagnostic questions about weight anomalies using only benign pathologies as training signal, and evaluate it zero-shot on backdoors absent from training. The oracle achieves AUROC 0.93 on attention-routed backdoors and 0.81 across a diversified threat distribution including stealth and adversarially regularized variants. Hand-crafted statistical detectors are sharp on the threat models they implicitly target but collapse on threat-model shift, while the oracle remains uniformly competent across attack types. Scaling to realistic model sizes remains the principal open challenge.

Figures & tables

Panel of Figure 1 — Figure 1: Weight Oracle architecture. Each weight matrix of the target transformer is sliced into rows; each row is projected from its native dimension $d\in\{8,16\}$ into the LLM’s hidden dimension (2048) by a learned linear layer, then summed with a type embedding $\tau$ identifying the source matrix. The resulting weight tokens are concatenated with a natural-language question and processed by Qwen 2.5 3B. The same injection pipeline is used in both phases; the output head differs (Phase I: predicted forward-pass output; Phase II: text answer).

Figure 2: Per-attack-type AUROC breakdown across the four threat models.

Figure 3: Layer-wise probing: where backdoor detection emerges in the oracle.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: External chain-of-computation. Each forward pass of the target transformer is decomposed into five LLM passes, one per linear weight projection, interleaved with deterministic external operations for LayerNorm, attention output projection, GELU, biases, and residuals. The LLM learns only the dot-product primitive $W\cdot h$ ; all composition is handled outside it.

Category	Pathology	Construction	Weight signature	% mix
Baseline	Clean	Standard training, 200–2000 steps on random token $\to$ class task	$\\|W_{V}\\|\approx 1.6$ , SV concentration $\approx 0.27$ ; accuracy 95–100%	18
Attention pathway	Amplified	Post-training scaling of $W_{V},W_{O}$ by $1.1$ – $1.8\times$	$\\|W_{V}\\|$ : $1.6\to 2.5$ ; $\\|W_{O}\\|$ : $1.7\to 2.7$	32
	Rank-1 injection	Add $uv^{\top}$ to $W_{V},W_{O}$	Top SV of $W_{V}$ : $0.8\to 2.1$ ; remaining SVs unchanged
	Subtle variants	Small-magnitude amplification with mild rank perturbations	Weak attention-pathway anomalies near the clean distribution
Shortcut	Dominant token	Mixed-objective training: normal task $+$ “token $t\to$ class $c$ at any position”	Token-specific routing through $W_{V},W_{O}$ ; closest benign proxy to a backdoor	16
Shortcut	Soft trigger	Positional bias added to logits during training	Mild attention asymmetry; position-dependent class bias	16

Appendix

Table 1: Training pathology specification.

Stage	Identifier	Natural-language question	Weight property probed
A	trained_or_random	Is this model trained or randomly initialised?	Init vs. post-training weight statistics
	accuracy_level	What is this model’s accuracy level?	Overall model competence
	confidence_level	How confident is this model on average?	Output entropy / softmax peakedness
	attn_value_magnitude	How large are the value and output projection weights of the attention layer?	$\\|W_{V}\\|_{F}$ , $\\|W_{O}\\|_{F}$
	attn_vo_balance	How does the value/output pathway magnitude compare to the query/key pathway?	$(\\|W_{V}\\|\!+\!\\|W_{O}\\|)\,/\,(\\|W_{Q}\\|\!+\!\\|W_{K}\\|)$
	attn_sv_concentration	How concentrated are the singular values of the attention value weights?	Top-SV / sum-SV of $W_{V}$

Appendix

Table 2: The 17 oracle question types. Each row gives the stage identifier, the natural-language wording, and the probed weight-space property.

Figure 6: Backdoored Model (high confidence detection)

Figure 7: Backdoored Model (hardest case — missed)

Figure 9: Backdoor signature of pos0 backdoors: W V singular-value concentration against V/O pathway balance (left) and V/O Frobenius norms (right) for clean vs. backdoored models.

Figure 10: ROC curve for zero-shot backdoor detection on the pos0-only cohort.

Figure 11: Sim vs. scratch initialisation by threat model. Sim initialisation helps on V/O-routed attacks (pos0) but degrades sharply on pathway-shifted attacks (lastpos, stealth), consistent with the negative transfer discussed in Section 4.

Figure 12: Weight matrix type ablation: AUROC change from zeroing each matrix type.

Figure 13: Gradient attribution by weight type (left) and differential attribution between backdoored and clean models (right).

Figure 14: PCA of oracle hidden states: clean vs. backdoored separation (left), coloured by V/O norm ratio (right).

Explore similar work

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles
Jul 25, 2026Tobias Bersia, Tatiana GaintsevaModel Activations Representation-Level Misalignment
BAISH · Queen Mary University of London
Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks
May 26, 2026Kevin Kuo, Virginia Smith, Chhavi YadavAttacker Large Language Model Large Language Model Fine-Tuning
Carnegie Mellon University · Simons Institute, UC Berkeley
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiLarge Language Model Safety Guardrail
Wrynx Inc. · Independent Researcher

Jul 25, 2026, cs.CLJ/K move · Enter open · S save

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Tobias Bersia, Tatiana Gaintseva

BAISH · Queen Mary University of London

Model Activations Representation-Level Misalignment Verbalization Blind Spots

Open paper AskPDF arXiv

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

Abstract

Explore similar work

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models