Computation, not weight access: large language models as auditors of clinical prediction networks

Zhiyu Liu; Xinyu Yang

cs.LGNov 5, 2026

Computation, not weight access: large language models as auditors of clinical prediction networks

Authors: Zhiyu Liu, Xinyu Yang

Organizations: Tianjin Medical University, Tianjin, China · Tianjin Medical University General Hospital, Tianjin, China

Abstract

Frontier large language models (LLMs) have been reported to read, edit and write the weights of small neural networks, raising the prospect of auditing clinical prediction models from their files alone. Public demonstrations, however, mixed reading raw weights with running code. We separated these conditions in internally pre-specified simulation experiments. Tanh networks of 105 to 38,401 weights were trained on simulated cohorts of patients with subarachnoid haemorrhage, each with a planted relationship or none, and one network was trained on labels that under-recorded deaths in a subgroup. GPT-6 Astra and four other GPT models answered single prompts under a default or an explicit system instruction, either with the weights in the prompt and no tools, or with a calculation-only tool that exposed the weights (white box) or only the network's predictions (black box). Without tools, Astra recovered planted relationships in 353-weight networks but became unreliable at 3,457 weights, and under the default instruction it declined 11 of 12 audit and repair tasks as unidentifiable; the explicit instruction removed these refusals. With the tool, it recovered the planted relationship in all 29 runs on networks of up to 38,401 weights, raised no false alarm in 15 runs on null networks, and found the biased subgroup in 6 of 6 runs. Black-box access performed as well as white-box access, and other GPT-6 models read as accurately. A well-designed script matched the comparator models on the audit and the best LLM repair. For auditing these networks, what mattered was computation and elicitation, not access to the weights.

Figures & tables

Element	Levels
Networks	Tanh multilayer perceptrons trained on simulated cohorts: 105 weights (11 inputs); S, 353; M, 1,217; L, 3,457; D3 (three hidden layers), 2,817; XL, 11,009; XXL, 38,401 (20 inputs); and a 209-weight network trained on biased labels, with an unbiased counterpart
Planted relationships	Threshold; interaction; each at half strength; none (null). In the 105-weight networks: a threshold, a hypertension-by-occupation interaction, an age dip
Tasks	Read (explain and write a surrogate model); audit (find subgroups whose predicted risk departs from what their other characteristics imply); repair (edit the weights)
Instruction	Default (one sentence); explicit (open choices left to the model; Appendix B )
Access	Weights in the prompt, no tools; calculator with the weights (white box); calculator with predictions only (black box)
Models	GPT-6 Astra (main); gpt-6.1-sol , gpt-6-sol , gpt-6-luna , gpt-5.6-sol

Table 1: Study design at a glance.

Figure 1: Reading planted relationships across network sizes (GPT-6 Astra). (a) Strength of the planted relationship in the model’s surrogate divided by its strength in the network; the dashed line marks equality and the dotted line the pre-specified threshold of one half. (b)

R^{2}

of the surrogate against the network’s log-odds, including null networks. Default and explicit refer to the system instruction; the calculator held either the weights (white box) or only a prediction function (black box). Points are single runs, offset slightly on the horizontal axis.

Network (weights)	Planted	Default, no tool	Explicit, no tool	Calculator, white box	Calculator, black box
S (353)	Threshold	0.89; 0.41–0.88 [2/2]	–	–	–
S (353)	Interaction	0.56–0.81; 0.84 [2/2]	–	0.85; 0.88 [1/1]	–
S (353)	Threshold, half strength	0.96; 0.82 [1/1]	–	–	–
S (353)	Interaction, half strength	0.70; 0.79 [1/1]	–	–	–
S (353)	None (null)	0.89–0.90 [0/2]	–	0.91 [0/1]	–
M (1,217)	Threshold	0.44–0.91; 0.85–0.90 [1/2]	–	–	–

Table 2: Reading planted relationships (GPT-6 Astra). Each cell gives the strength ratio and the surrogate

R^{2}

(range over runs) and, in brackets, the number of runs in which the relationship was recovered by the pre-specified rule; for null networks,

R^{2}

and the number of false alarms.

Figure 2: Recovery of the biased subgroup. Each point is one run on the biased network: the Jaccard overlap between the true subgroup (women farmers over 65) and the reported subgroup closest to it. Crosses mark runs that declined to report a subgroup; the dotted line marks the pre-specified threshold of 0.5. Rows without a model name are GPT-6 Astra.

Condition	Biased: found	Biased: declined	Best Jaccard	Unbiased: false alarm
Default instructions, weights in prompt	0/2	2/2	0.00	0/1
Default + stated additive reference	0/2	2/2	0.00	1/1
Explicit instructions, weights in prompt	1/2	0/2	0.01–0.53	0/1
Explicit + stated additive reference	1/2	0/2	0.59–1.00	1/1
Explicit + calculator, white box	3/3	0/3	0.99–1.00	1/3
Explicit + calculator, black box	3/3	0/3	1.00	0/3

Table 3: Auditing the biased network and the unbiased control. Found: pre-specified rule (Jaccard

\geq

0.5, correct sign, at least half the network’s shift). False alarm: a stated shift of 0.5 or more on the unbiased network. Rows without a model name are GPT-6 Astra.

Condition	Runs	Declined	Invalid	Change in subgroup	Mean $\|$ change $\|$ elsewhere	Bias after
E8 repair, default (subgroup not given)	2	2	0	–	0.00	$-$ 1.61 to $-$ 1.60
E8 repair, explicit (subgroup not given)	2	0	0	–	0.14	$-$ 1.44
Subgroup given, default	2	2	0	0.00	0.00	$-$ 1.60
Subgroup given, explicit + tolerance	2	0	0	1.24–1.40	0.07–0.09	$-$ 0.37 to $-$ 0.20
+1.6 exact, default	2	2	0	0.00	0.00	$-$ 1.60
+1.6 exact, explicit	2	0	0	0.80	0.80	$-$ 0.81

Table 4: Repairs. Change in subgroup: mean change in log-odds for women farmers over 65 (target

+

1.6, or removal of the departure). Bias after: remaining difference from the unbiased network inside the subgroup (before:

-

1.60). Invalid: an answer that broke the shape constraint. Rows above the line are GPT-6 Astra.

Model	No tool, 105-weight networks (E13)		Calculator, white box (E19)
	Found	$R^{2}$	Interaction ratio (L, XXL)	Null $R^{2}$ (L, XXL)	Audit (Jaccard; false alarm)
gpt-6-astra	6/6	0.94–0.97	0.85–1.08	0.91–0.98	0.99–1.00; 1/3
gpt-6.1-sol	3/3	0.96–0.97	0.89–0.97	0.94–0.98	0.53; 0/1
gpt-6-sol	2/3	0.30–0.91	0.95–0.99	0.92–0.98	0.52; 1/1
gpt-6-luna	2/3	0.49–0.84	0.99–1.13	0.93–0.97	0.05; 1/1
gpt-5.6-sol	3/3	0.33–0.93	–	–	–

Table 5: Other models. Left: the 105-weight networks without tools (pre-specified shape rules; three planted relationships). Right: the calculator with white-box access on the 3,457- and 38,401-weight networks and the audit.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Experiment	Plan commit	Committed (UTC+8)	First run started
Planted relationships in 105-weight networks	3a6bd3c	2026-11-04 20:40	2026-11-04 20:42
Size ladder, half strength and network groups; biased-label audit and repair	e259471	2026-11-05 00:35	2026-11-05 00:35
Function-preserving rewrites	449c348	2026-11-05 00:42	2026-11-05 03:21
Addendum: claims at zero locations and exact scan	09a62cd	2026-11-05 00:51	script, after the commit
Other models without tools (105-weight prompts)	5bc52e3	2026-11-05 00:59	2026-11-05 00:59
Stated reference for audit and repair	1425aa0	2026-11-05 01:03	2026-11-05 01:03

Appendix

Table 6: Plan commits and the start of the first run of each experiment.

LLM Auditing

Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.

See all 666 papers

Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring "Weight-Space Programming"
Nov 5, 2026Nazmus Sakib AhmedNeural Network Interpretability Memorization in Language Models
BRAC University, Dhaka, Bangladesh
Weight Oracles: Reading Neural Network Weights with Language Models
Oct 5, 2026Krishna Kabra, Constantin Venhoff, Christian Schroeder de WittLLM Auditing Backdoor Detection
University of Oxford
Weight-space programming for auditing clinical decision networks
Nov 3, 2026Humza Nusrat, Omar NusratWeight-Space Analysis
Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan 48202, USA · Department of Radiology, Michigan State University, East Lansing, Michigan 48824, USA · Toronto Metropolitan University, Toronto, Ontario, Canada

Nov 5, 2026J/K move · Enter open · S save

Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring "Weight-Space Programming"

Nazmus Sakib Ahmed

BRAC University, Dhaka, Bangladesh

Neural Network Interpretability Memorization in Language Models LLM Auditing

Open paper AskPDF arXiv

Large language models (LLMs) have recently been reported to read and write the raw weights of neural networks directly, a capability termed weight-space programming. We ask whether the reading side of this capability reflects reasoning about weight flows or a combination of local arithmetic at the network's edges and memorized priors over public datasets. We test Claude Opus 5.5 on small convolutional networks of about 125,000 parameters trained on five image datasets, providing every weight as plain text with no tools, no forward passes, and no dataset or class names. The model performs accurate local calculations: batch-normalization statistics lead it to infer about three optimizer steps on networks stopped after three updates, and output-layer sign patterns recover the vehicle/animal split of CIFAR-10. The sessions report reading the first and last layers closely and sampling the middle, and several claims built on partial tracing fail verification, although we did not run a direct forward-pass test. Semantic judgments lean on familiar datasets: the model names CIFAR-10 for CIFAR-10 networks but also for all three fast-food networks, relabels KMNIST as MNIST, reads a 99.57%-accurate SAT-6 network as CIFAR-10 animal photos, and calls "frog" the output favored by a feature that fires most on trees. Permuting CIFAR-10's labels preserves the output split at the new indices, but every class pair the model interprets is then wrong, indicating reliance on remembered label order. These are exploratory results from mostly single runs. We conclude that zero-shot weight reading by this model is better described as local arithmetic plus dataset priors than as weight-flow tracing, and we provide a protocol for auditing future weight-space claims.

Abstract

LLM Auditing

Explore similar work

Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring "Weight-Space Programming"

Weight Oracles: Reading Neural Network Weights with Language Models

Weight-space programming for auditing clinical decision networks