Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring "Weight-Space Programming"

Nazmus Sakib Ahmed

cs.LGNov 5, 2026

Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring "Weight-Space Programming"

Authors: Nazmus Sakib Ahmed

Organizations: BRAC University, Dhaka, Bangladesh

Abstract

Large language models (LLMs) have recently been reported to read and write the raw weights of neural networks directly, a capability termed weight-space programming. We ask whether the reading side of this capability reflects reasoning about weight flows or a combination of local arithmetic at the network's edges and memorized priors over public datasets. We test Claude Opus 5.5 on small convolutional networks of about 125,000 parameters trained on five image datasets, providing every weight as plain text with no tools, no forward passes, and no dataset or class names. The model performs accurate local calculations: batch-normalization statistics lead it to infer about three optimizer steps on networks stopped after three updates, and output-layer sign patterns recover the vehicle/animal split of CIFAR-10. The sessions report reading the first and last layers closely and sampling the middle, and several claims built on partial tracing fail verification, although we did not run a direct forward-pass test. Semantic judgments lean on familiar datasets: the model names CIFAR-10 for CIFAR-10 networks but also for all three fast-food networks, relabels KMNIST as MNIST, reads a 99.57%-accurate SAT-6 network as CIFAR-10 animal photos, and calls "frog" the output favored by a feature that fires most on trees. Permuting CIFAR-10's labels preserves the output split at the new indices, but every class pair the model interprets is then wrong, indicating reliance on remembered label order. These are exploratory results from mostly single runs. We conclude that zero-shot weight reading by this model is better described as local arithmetic plus dataset priors than as weight-flow tracing, and we provide a protocol for auditing future weight-space claims.

Figures & tables

Figure 1: Conceptual illustration of the central finding. The model inspects the outer layers of a trained network and fills the semantic gap with a familiar template; here, a geometric frog is imposed on a satellite forest patch. The illustration is not a literal rendering of the experiments.

Dataset	Classes	Train / val / test	Majority (%)	Networks	Test accuracy (%)
CIFAR-10	10	45,000 / 5,000 / 10,000	10.00	M1 / M2 / R	82.36 / 82.28 / 83.95
KMNIST	10	54,000 / 6,000 / 10,000	10.00	M1 / M2 / R	97.84 / 97.58 / 97.95
Cats vs Dogs	2	18,728 / 2,341 / 2,341	50.15	M1 / M2	87.70 / 87.95
Fast food	10	2,400 / 300 / 300	10.00	M1 / M2 / R	54.67 / 52.67 / 58.00
SAT-6	6	64,800 / 8,100 / 8,100	37.12	M2 high / M2 low	99.57 / 36.01

Table 1: Datasets, splits, and test accuracies. M1, M2, and R denote Model 1, Model 2, and Random. The SAT-6 low-accuracy network is the three-update model.

Panel of Figure 2 — Figure 2: Experimental pipeline. A small CNN is trained; its weights are serialized as text; a fresh session with no tools or memory receives the weights and a question; the specific values we could locate in each answer are checked against the actual parameters.

Figure 3: Output-layer structure on CIFAR-10, Model 1. Seven hidden units split outputs 0–9 into vehicles $\{0,1,8,9\}$ and animals $\{2,\dots,7\}$ , matching the standard CIFAR-10 label order.

Control	Manipulation	Acc. (%)	Model answer	Conf. (%)
None	Original weights	82.36	CIFAR-10	90
Reordered outputs	Vehicles moved to outputs 2, 4, 5, 7	82.37	CIFAR-10	72
Shuffled weights	Values scrambled within each array	7.78	CIFAR-10 (best guess)	45

Table 2: CIFAR-10 controls on Model 1, under the original question. Test accuracy for the two controls is measured on weights rounded as in the prompt, with labels renumbered for the reordered control. The original weights also score 82.37% after the same rounding, against 82.36% before it.

Figure 4: Reading versus naming on the reordered CIFAR-10 network. Left: the model correctly groups outputs 2, 4, 5, and 7. Right: the class names the standard order assigns to those outputs, all of which are wrong (for example, output 2 is really airplane, not bird).

True dataset	Model 1 answer	Model 2 answer
KMNIST (Japanese characters)	MNIST (80%)	MNIST (85%)
Cats vs Dogs	CIFAKE (30%)	Cats vs Dogs (35%)
Fast-food photos	CIFAR-10 (45%)	CIFAR-10 (50%)

Table 3: Model answers for less-familiar datasets under the original question. Confidence is in parentheses.

Dataset	Architecture	Acc. (%)	Image type	In position	Conf.
CIFAR-10	Model 1 $(16,32,64)$	82.36	Correct	10 of 10	95% / 90%
CIFAR-10	Random $(62,68,25)$	83.95	Correct	10 of 10	92% / 85%
KMNIST	Model 1 $(16,32,64)$	97.84	Partial: grayscale strokes, called digits	0 of 10	85% / 65%
KMNIST	Random $(62,68,25)$	97.95	No answer	No answer	No answer

Table 4: Architecture comparison under the revised question. “In position” counts outputs whose class name is correct at that index. “Conf.” gives confidence in the image type and in the classes.

Network	Acc. (%)	F1 (%)	Image type	Any position	Name given	Conf.
High accuracy	99.57	99.51	Wrong: animal photos	0 of 6	CIFAR-10, animal classes	70% / 45%
Low accuracy	36.01	22.91	Partial: scenes (“only a guess”)	4 of 6	Unknown	30% / 3%

Table 5: SAT-6 accuracy comparison on Model 2 under the revised question, with test accuracy and macro F1. “Any position” counts correct class names regardless of output index. “Conf.” gives confidence in the image type and in the classes.

What the session read	What is true in the network	Session’s conclusion
A “dark green” feature favors output 4	It correlates with greenness (+0.64), not brightness ( $-0.01$ ), and fires most on trees. Its top output across all hidden units is output 4, which is trees	Output 4 is frog
Outputs 1 and 3 form a strong pair (+0.54)	Outputs 1 and 3 are building and road	Cat and dog
A “blue” feature favors output 0	It correlates $-0.44$ with blueness, fires most on grassland and road, and is below average on water	Output 0 is bird

Table 6: SAT-6 high-accuracy session: what the model read, what is true in the network, and what it concluded.

Figure 5: Same reading, wrong meaning. Conceptual illustration: a satellite forest patch and a frog are joined by a line of numbers, which are decorative and not data. The session associates a green feature with output 4, which is trees, and calls that output “frog”, in line with its reading of the network as CIFAR-10 animals.

Figure 6: Where the sessions report looking. Conceptual summary, not a measurement: conv1 and fc2 are read closely, while conv2 , conv3 , and fc1 are sampled sparsely.

Component	Verdict	Deciding evidence
Label order breaks it	Supported, one run	Renumbered outputs kept their remembered CIFAR-10 meanings; all nine interpreted class pairs were wrong.
Input and output shape bias the guess	Supported indirectly	Same-shape fast food became CIFAR-10. Every network used $32\times 32$ inputs, so input size was not varied, and KMNIST shows that shape alone does not decide the answer.
It is not following weight flows	Supported indirectly	Sessions report sampling the middle layers, and several claims based on partial tracing failed our checks. No forward-pass test was run.
It leans on famous datasets	Supported	KMNIST became MNIST, and the 99.57%-accurate SAT-6 network became CIFAR-10 animal photos.
Non-standard architectures break it	Unresolved	Unusual hidden layer widths did not break CIFAR-10 recognition, but input and output dimensions stayed standard.
Higher accuracy helps recognition	No evidence, one pair	On SAT-6, 99.57% accuracy came with more confidence in a wrong answer, not a correct one.

Table 7: Evidence for the components of the deflationary account.

Panel of Figure 7 — Figure 7: Confidence in image type under the revised question, labeled with each answer’s score. The SAT-6 high-accuracy session reports 70% for a wrong image type.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

True dataset	Network	Acc. (%)	Model answer	Conf. (%)	Score
CIFAR-10	Model 1	82.36	CIFAR-10	90	Correct*
CIFAR-10	Model 2	82.28	CIFAR-10	92	Correct*
CIFAR-10 (reordered)	Model 1	82.37	CIFAR-10	72	Fail
CIFAR-10 (shuffled)	Model 1	7.78	CIFAR-10 (best guess)	45	Pass
KMNIST	Model 1	97.84	MNIST	80	Partial
KMNIST	Model 2	97.58	MNIST	85	Partial

Appendix

Table 8: Original question: naming the dataset. An asterisk marks the two runs that had no preset scoring rule.

True dataset	Network	Acc. (%)	Image type	In position	Any position	Name given
CIFAR-10	Model 1	82.36	Correct	10 of 10	10 of 10	CIFAR-10
CIFAR-10	Random	83.95	Correct	10 of 10	10 of 10	CIFAR-10
KMNIST	Model 1	97.84	Partial	0 of 10	0 of 10	MNIST
KMNIST	Random	97.95	No answer	No answer	No answer	No answer
SAT-6	Model 2 high	99.57	Wrong	0 of 6	0 of 6	CIFAR-10, animals
SAT-6	Model 2 low	36.01	Partial	0 of 6	4 of 6	Unknown

Appendix

Table 9: Revised question: describing the dataset. “In position” counts outputs whose class name is correct at that index; “any position” counts correct names regardless of index.

Session	Checked	Mismatched	Note
CIFAR-10, Model 1 (original)	18	0	One sign count was 62 against “about 61”
CIFAR-10, Model 2 (original)	34	0
CIFAR-10, reordered	21	0	Similarity estimates within 0.03
Fast food, Model 1	12	2	One wrong filter total ( $+0.09$ against $-0.006$ ); correlation estimates up to 0.02 off
Fast food, Model 2	19	0
Fast food, Random	30	0

Appendix

Table 10: Tallies of quoted items checked against the weights.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 342 papers

Nov 5, 2026

Computation, not weight access: large language models as auditors of clinical prediction networks

Frontier large language models (LLMs) have been reported to read, edit and write the weights of small neural networks, raising the prospect of auditing clinical prediction models from their files alone. Public demonstrations, however, mixed reading raw weights with running code. We separated these conditions in internally pre-specified simulation experiments. Tanh networks of 105 to 38,401 weights were trained on simulated cohorts of patients with subarachnoid haemorrhage, each with a planted relationship or none, and one network was trained on labels that under-recorded deaths in a subgroup. GPT-6 Astra and four other GPT models answered single prompts under a default or an explicit system instruction, either with the weights in the prompt and no tools, or with a calculation-only tool that exposed the weights (white box) or only the network's predictions (black box). Without tools, Astra recovered planted relationships in 353-weight networks but became unreliable at 3,457 weights, and under the default instruction it declined 11 of 12 audit and repair tasks as unidentifiable; the explicit instruction removed these refusals. With the tool, it recovered the planted relationship in all 29 runs on networks of up to 38,401 weights, raised no false alarm in 15 runs on null networks, and found the biased subgroup in 6 of 6 runs. Black-box access performed as well as white-box access, and other GPT-6 models read as accurately. A well-designed script matched the comparator models on the audit and the best LLM repair. For auditing these networks, what mattered was computation and elicitation, not access to the weights.

Zhiyu Liu, Xinyu Yang

Tianjin Medical University, Tianjin, China · Tianjin Medical University General Hospital, Tianjin, China

LLM Auditing LLM Evaluation LLM Tool Use

Ask PDF

Oct 5, 2026

Weight Oracles: Reading Neural Network Weights with Language Models

Interpretability methods for neural networks are predominantly reactive: they analyse activations produced during specific forward passes, requiring known inputs to find hidden capabilities such as backdoors. We propose Weight Oracles, fine-tuned language models that diagnose properties of a target network by reading its raw weights directly, without behavioural testing. We investigate this paradigm in two phases. Phase I establishes feasibility: through a staged curriculum and an external chain-of-computation that delegates parameter-free operations to deterministic code, an explainer LLM learns to simulate the forward pass of small transformers from their weights, achieving 99% holdout accuracy on unseen targets. Phase II repurposes this infrastructure for safety auditing. We train an oracle on natural language diagnostic questions about weight anomalies using only benign pathologies as training signal, and evaluate it zero-shot on backdoors absent from training. The oracle achieves AUROC 0.93 on attention-routed backdoors and 0.81 across a diversified threat distribution including stealth and adversarially regularized variants. Hand-crafted statistical detectors are sharp on the threat models they implicitly target but collapse on threat-model shift, while the oracle remains uniformly competent across attack types. Scaling to realistic model sizes remains the principal open challenge.

Krishna Kabra, Constantin Venhoff, Christian Schroeder de Witt

University of Oxford

LLM Auditing Backdoor Detection Neural Network Interpretability

Ask PDF

Aug 7, 2026

Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Space

Open-weight large language models (LLMs) are increasingly developed through complex, multi-stage pipelines, leading to intricate lineage relationships that reflect model origin, ownership, and evolution. Understanding these relationships is important for model provenance, governance, and supply-chain integrity. In this work, we investigate the notion of LLM "biometrics" (analogous to human biometrics) to ask whether LLMs exhibit intrinsic fingerprints in weight space alone, without access to input data, that reveal their origin and lineage. We formulate this as a lineage discrimination problem, distinguishing among independent-origin, same-series, and shared-base models. To characterize these relationships, we propose a unified geometric fingerprinting framework that analyzes weight matrices from two complementary perspectives: (i) spectral energy, captured by singular value distributions to encode global magnitude patterns, and (ii) subspace alignment, quantified via subspace deviations to capture directional geometry. Our analysis uncovers a clear hierarchy of structural similarity in weight space: spectral energy reliably distinguishes independently trained models and different model families, while subspace alignment enables fine-grained discrimination among closely related models, including variations in dataset scale and post-training procedures. Extensive experiments on over 110 diverse open-weight LLM pairs demonstrate that weight-space geometry provides a robust and interpretable signal for model lineage, enabling coarse-grained regime separation and fine-grained discrimination within shared-base models.

Yiwei Chen, Bingqi Shang, Sijia Liu

Michigan State University

Language Model Fingerprinting Open-Weight Language Models Weight-Space Analysis

Ask PDF

Abstract

Neural Network Interpretability

Explore similar work

Computation, not weight access: large language models as auditors of clinical prediction networks

Weight Oracles: Reading Neural Network Weights with Language Models

Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Space