Weight-space programming for auditing clinical decision networks

Humza Nusrat; Omar Nusrat

physics.med-phNov 3, 2026

Weight-space programming for auditing clinical decision networks

Authors: Humza Nusrat, Omar Nusrat

Organizations: Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan 48202, USA · Department of Radiology, Michigan State University, East Lansing, Michigan 48824, USA · Toronto Metropolitan University, Toronto, Ontario, Canada

Abstract

Machine learning models in radiotherapy are commissioned by testing them on cases with known answers, which may not reveal a shortcut that the test cases share with the training data. Frontier language models have recently been reported to read and write the raw weights of small neural networks, an ability called weight-space programming. The purpose of this study was to determine whether such a model, given only a clinical protocol and a network's weights, can audit and repair the network. Networks with 433 to 4,801 parameters were trained to screen head-and-neck planning requests against four rules. Flaws were planted through the training data: confounds that hid each rule, a 3 Gy shift in the cord threshold, and a hidden trigger. Claude Opus 5.5 audited each network blind, without tools, data or code execution, and every claim and repair was tested on about 420,000 requests. The language model named the broken rule in 16 of 18 flawed networks (95% CI 67% to 97%), including all 15 with a confound or a threshold shift. It repaired each of the 16 by changing a median of 3 parameters. Fine-tuning on protocol-labeled requests also repaired every network, but changed nearly every parameter. Each confound and shift it found was visible in single units, and a simple statistic chosen for each flaw type found the same flaws. It cleared 2 of 5 healthy networks without being forced and found 1 of 3 triggers. Testing against the protocol detected every flaw, and black-box testing found the confounds and the shift faster. The results indicate that, in small networks, weight-space programming can locate and repair flaws that silence an input or shift a threshold, but cannot yet certify that a network is correct.

Figures & tables

Input	Definition	Range
Coverage goal	PTV D95% at or above this dose	55 to 78 Gy
Hot-spot cap	PTV D0.03cc at or below this dose	54 to 81 Gy
Mandible limit	Mandible D0.1cc at or below this dose	45 to 85 Gy
Mandible in PTV	Volume of mandible inside the PTV	0 to 28 cc
PTV volume	Volume of the highest-dose PTV	80 to 400 cc
Cord limit	Spinal cord D0.03cc at or below this dose	38 to 50 Gy

Table 1: Inputs to the decision networks. Doses are in Gy and volumes in cc. The cord floor came from a separate feasibility calculation.

Planted flaw	Error in planted region	Detected from the weights	Parameters changed	Correct decisions made wrong	Detected by black-box testing	Found by a weight statistic
Cap rule hidden	98.2% to 99.9%	3/3	3, 3, 39	0, 0, 0	1/1	3/3
Mandible rule hidden	100%	3/3	21, 24, 53	0, 0, 0	1/1	3/3
Cord rule hidden	99.9% to 100%	3/3	5, 4, 3	0, 0, 0	1/1	3/3
Missing-structure rule hidden	53.6% to 68.3%	3/3	1, 1, 1	0, 0, 0	1/1	3/3
Cord threshold shifted 3 Gy (drift)	99.9% to 100%	3/3	3, 1, 2	2, 18, 0	1/1	3/3
Hidden trigger	58.4% to 96.0%	1/3	1, 1, 0	1,181, 0, 0	1/3	0/3

Table 2: Results for the flaw panel, audited by Claude Opus 5.5. Planted-region error was measured before any audit on 400,000 uniform requests (trigger: 17,281 requests in the window). Parameters changed and correct decisions made wrong are given for each network in seed order; the latter are counted among about 420,000 test requests. For the trigger, the first edit came with a forced and false verdict, and the third network was reported to follow the protocol and was not edited. Black-box testing was done on one network per condition, except for the trigger; its trigger detection held only because the window was densely sampled. The weight statistic was the column norm, except for the drift, where it was the cord switch-on point (column norms found none of the three drifts).

Figure 1: Rule-B decisions for pilot network 3, against the coverage goal minus the mandible limit and the mandible volume forced to receive the coverage dose. The dashed outline is the region the protocol escalates. (a) The trained network planned every request and missed the whole region (hatched). (b) After the language model’s edit, the network escalated the region; the small offset at its edges lies within the 0.3 Gy and 0.3 cc tolerance. The hot-spot cap was 3 Gy above the coverage goal, the PTV volume was 150 cc, and the cord and structure set were acceptable.

Figure 2: The edit to pilot network 3. The edit changed 22 of the 433 parameters: the seven connections drawn here, three biases, and twelve weights set to zero so that

u_{1}

and

u_{2}

each read only two inputs. Two first-layer units that were almost never active became

u_{1}

and

u_{2}

, and a second-layer unit that never fired became

g

, which turns on only when both rule-B conditions hold. The equations are in raw units, with doses in Gy and volumes in cc; the dots in

g

stand for small terms from unchanged weights.

Figure 3: Repairs of the 18 flawed panel networks. The language model edited 17 of them without data: 16 repairs of detected flaws (filled dots) and one forced edit of a trigger network, which did not repair it (open dot). It made no edit to the third trigger network. A script that was told which rule was broken, and fine-tuning on 200,000 protocol-labeled requests at a constant learning rate and with the networks’ training schedule, were applied to all 18. Each dot is one network; for fine-tuning, it is the median of five runs with different random seeds. Vertical bars mark the medians of the repairs. (a) Parameters changed, of 433. (b) Previously correct decisions made wrong, of 416,809 test requests.

Language model	Output limit	Cap rule hidden	Mandible rule hidden	Cord drift	Healthy
Opus 5.5	128,000	Detected, repaired	Detected, repaired	Detected, repaired	Cleared
Sonnet 5.5	128,000	Detected, repaired	Detected, repaired	Detected, repaired (1.1% error left)	Cleared (forced)
Haiku 5.5	128,000	Detected, repaired	Detected, repaired	Repaired; region covered 26% of failures	False alarm
Opus 4.6	64,000	No verdict	No verdict	No verdict	No verdict
Fable 5.1	64,000	Not tested	Detected, repaired (after extra message)	Not tested	No verdict (after extra message)

Table 3: Results for other language models on the first network of each listed condition. The output limit is the approximate maximum length of one reply, in tokens.

Weight-Space Analysis

Weight-space analysis studies neural network parameters as structured objects to understand, compare, edit, and reuse learned models.

See all 42 papers

Weight Oracles: Reading Neural Network Weights with Language Models
Oct 5, 2026Krishna Kabra, Constantin Venhoff, Christian Schroeder de WittLLM Auditing Backdoor Detection
University of Oxford
OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
Oct 27, 2026Humza Nusrat, Kundan ThindMedical AI Agents LLM Agent Evaluation
Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan, USA · Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
A Radiation-Hardened, Fault-Aware Quantized Neural Network That Self-Heals
Oct 20, 2026Peter Xiangyuan Ma, Joshua S. BloomNeural Network Fault Tolerance
Department of Astronomy, University of California, Berkeley, CA 94720, USA

Oct 5, 2026J/K move · Enter open · S save

Weight Oracles: Reading Neural Network Weights with Language Models

Krishna Kabra, Constantin Venhoff, Christian Schroeder de Witt

University of Oxford

LLM Auditing Backdoor Detection Neural Network Interpretability

Open paper AskPDF arXiv

Interpretability methods for neural networks are predominantly reactive: they analyse activations produced during specific forward passes, requiring known inputs to find hidden capabilities such as backdoors. We propose Weight Oracles, fine-tuned language models that diagnose properties of a target network by reading its raw weights directly, without behavioural testing. We investigate this paradigm in two phases. Phase I establishes feasibility: through a staged curriculum and an external chain-of-computation that delegates parameter-free operations to deterministic code, an explainer LLM learns to simulate the forward pass of small transformers from their weights, achieving 99% holdout accuracy on unseen targets. Phase II repurposes this infrastructure for safety auditing. We train an oracle on natural language diagnostic questions about weight anomalies using only benign pathologies as training signal, and evaluate it zero-shot on backdoors absent from training. The oracle achieves AUROC 0.93 on attention-routed backdoors and 0.81 across a diversified threat distribution including stealth and adversarially regularized variants. Hand-crafted statistical detectors are sharp on the threat models they implicitly target but collapse on threat-model shift, while the oracle remains uniformly competent across attack types. Scaling to realistic model sizes remains the principal open challenge.

Abstract

Weight-Space Analysis

Explore similar work

Weight Oracles: Reading Neural Network Weights with Language Models

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

A Radiation-Hardened, Fault-Aware Quantized Neural Network That Self-Heals