Weight-space programming for auditing clinical decision networks

Humza Nusrat; Omar Nusrat

physics.med-phNov 3, 2026

Weight-space programming for auditing clinical decision networks

Authors: Humza Nusrat, Omar Nusrat

Organizations: Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan 48202, USA · Department of Radiology, Michigan State University, East Lansing, Michigan 48824, USA · Toronto Metropolitan University, Toronto, Ontario, Canada

Abstract

Machine learning models in radiotherapy are commissioned by testing them on cases with known answers, which may not reveal a shortcut that the test cases share with the training data. Frontier language models have recently been reported to read and write the raw weights of small neural networks, an ability called weight-space programming. The purpose of this study was to determine whether such a model, given only a clinical protocol and a network's weights, can audit and repair the network. Networks with 433 to 4,801 parameters were trained to screen head-and-neck planning requests against four rules. Flaws were planted through the training data: confounds that hid each rule, a 3 Gy shift in the cord threshold, and a hidden trigger. Claude Opus 5.5 audited each network blind, without tools, data or code execution, and every claim and repair was tested on about 420,000 requests. The language model named the broken rule in 16 of 18 flawed networks (95% CI 67% to 97%), including all 15 with a confound or a threshold shift. It repaired each of the 16 by changing a median of 3 parameters. Fine-tuning on protocol-labeled requests also repaired every network, but changed nearly every parameter. Each confound and shift it found was visible in single units, and a simple statistic chosen for each flaw type found the same flaws. It cleared 2 of 5 healthy networks without being forced and found 1 of 3 triggers. Testing against the protocol detected every flaw, and black-box testing found the confounds and the shift faster. The results indicate that, in small networks, weight-space programming can locate and repair flaws that silence an input or shift a threshold, but cannot yet certify that a network is correct.

Figures & tables

Input	Definition	Range
Coverage goal	PTV D95% at or above this dose	55 to 78 Gy
Hot-spot cap	PTV D0.03cc at or below this dose	54 to 81 Gy
Mandible limit	Mandible D0.1cc at or below this dose	45 to 85 Gy
Mandible in PTV	Volume of mandible inside the PTV	0 to 28 cc
PTV volume	Volume of the highest-dose PTV	80 to 400 cc
Cord limit	Spinal cord D0.03cc at or below this dose	38 to 50 Gy

Table 1: Inputs to the decision networks. Doses are in Gy and volumes in cc. The cord floor came from a separate feasibility calculation.

Planted flaw	Error in planted region	Detected from the weights	Parameters changed	Correct decisions made wrong	Detected by black-box testing	Found by a weight statistic
Cap rule hidden	98.2% to 99.9%	3/3	3, 3, 39	0, 0, 0	1/1	3/3
Mandible rule hidden	100%	3/3	21, 24, 53	0, 0, 0	1/1	3/3
Cord rule hidden	99.9% to 100%	3/3	5, 4, 3	0, 0, 0	1/1	3/3
Missing-structure rule hidden	53.6% to 68.3%	3/3	1, 1, 1	0, 0, 0	1/1	3/3
Cord threshold shifted 3 Gy (drift)	99.9% to 100%	3/3	3, 1, 2	2, 18, 0	1/1	3/3
Hidden trigger	58.4% to 96.0%	1/3	1, 1, 0	1,181, 0, 0	1/3	0/3

Table 2: Results for the flaw panel, audited by Claude Opus 5.5. Planted-region error was measured before any audit on 400,000 uniform requests (trigger: 17,281 requests in the window). Parameters changed and correct decisions made wrong are given for each network in seed order; the latter are counted among about 420,000 test requests. For the trigger, the first edit came with a forced and false verdict, and the third network was reported to follow the protocol and was not edited. Black-box testing was done on one network per condition, except for the trigger; its trigger detection held only because the window was densely sampled. The weight statistic was the column norm, except for the drift, where it was the cord switch-on point (column norms found none of the three drifts).

Figure 1: Rule-B decisions for pilot network 3, against the coverage goal minus the mandible limit and the mandible volume forced to receive the coverage dose. The dashed outline is the region the protocol escalates. (a) The trained network planned every request and missed the whole region (hatched). (b) After the language model’s edit, the network escalated the region; the small offset at its edges lies within the 0.3 Gy and 0.3 cc tolerance. The hot-spot cap was 3 Gy above the coverage goal, the PTV volume was 150 cc, and the cord and structure set were acceptable.

Figure 2: The edit to pilot network 3. The edit changed 22 of the 433 parameters: the seven connections drawn here, three biases, and twelve weights set to zero so that

u_{1}

and

u_{2}

each read only two inputs. Two first-layer units that were almost never active became

u_{1}

and

u_{2}

, and a second-layer unit that never fired became

g

, which turns on only when both rule-B conditions hold. The equations are in raw units, with doses in Gy and volumes in cc; the dots in

g

stand for small terms from unchanged weights.

Figure 3: Repairs of the 18 flawed panel networks. The language model edited 17 of them without data: 16 repairs of detected flaws (filled dots) and one forced edit of a trigger network, which did not repair it (open dot). It made no edit to the third trigger network. A script that was told which rule was broken, and fine-tuning on 200,000 protocol-labeled requests at a constant learning rate and with the networks’ training schedule, were applied to all 18. Each dot is one network; for fine-tuning, it is the median of five runs with different random seeds. Vertical bars mark the medians of the repairs. (a) Parameters changed, of 433. (b) Previously correct decisions made wrong, of 416,809 test requests.

Language model	Output limit	Cap rule hidden	Mandible rule hidden	Cord drift	Healthy
Opus 5.5	128,000	Detected, repaired	Detected, repaired	Detected, repaired	Cleared
Sonnet 5.5	128,000	Detected, repaired	Detected, repaired	Detected, repaired (1.1% error left)	Cleared (forced)
Haiku 5.5	128,000	Detected, repaired	Detected, repaired	Repaired; region covered 26% of failures	False alarm
Opus 4.6	64,000	No verdict	No verdict	No verdict	No verdict
Fable 5.1	64,000	Not tested	Detected, repaired (after extra message)	Not tested	No verdict (after extra message)

Table 3: Results for other language models on the first network of each listed condition. The output limit is the approximate maximum length of one reply, in tokens.

Weight-Space Analysis

Weight-space analysis studies neural network parameters as structured objects to understand, compare, edit, and reuse learned models.

See all 42 papers

Oct 5, 2026

Weight Oracles: Reading Neural Network Weights with Language Models

Interpretability methods for neural networks are predominantly reactive: they analyse activations produced during specific forward passes, requiring known inputs to find hidden capabilities such as backdoors. We propose Weight Oracles, fine-tuned language models that diagnose properties of a target network by reading its raw weights directly, without behavioural testing. We investigate this paradigm in two phases. Phase I establishes feasibility: through a staged curriculum and an external chain-of-computation that delegates parameter-free operations to deterministic code, an explainer LLM learns to simulate the forward pass of small transformers from their weights, achieving 99% holdout accuracy on unseen targets. Phase II repurposes this infrastructure for safety auditing. We train an oracle on natural language diagnostic questions about weight anomalies using only benign pathologies as training signal, and evaluate it zero-shot on backdoors absent from training. The oracle achieves AUROC 0.93 on attention-routed backdoors and 0.81 across a diversified threat distribution including stealth and adversarially regularized variants. Hand-crafted statistical detectors are sharp on the threat models they implicitly target but collapse on threat-model shift, while the oracle remains uniformly competent across attack types. Scaling to realistic model sizes remains the principal open challenge.

Krishna Kabra, Constantin Venhoff, Christian Schroeder de Witt

University of Oxford

LLM Auditing Backdoor Detection Neural Network Interpretability

Ask PDF

Oct 27, 2026

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

Background and purpose: Radiotherapy planning agents based on large language models (LLMs) decide when to submit a plan or request review but are evaluated mainly by dose. This study developed OpenGray, a benchmark of planning tasks with known correct decisions. Materials and methods: OpenGray adds tasks with known answers to the OpenKBP-Opt head-and-neck dataset, including two contradictions with dose goals proven incompatible and a feasible control with a stored plan meeting every hard goal. Four systems, from a rule-based planning script to a full LLM agent, were evaluated on 30 patients, the LLM-based ones with a commercial (Terra) and a local (Ministral 3 8B) LLM. Decisions, achieved dose, problem recognition, and technical failures were recorded and also read from logs of 39 stereotactic radiosurgery runs of SAGE, a clinical LLM agent. Results: With Terra, the script preceded by an LLM review and the full agent had decision accuracies of 95.4% and 92.1%, respectively, yet escalated only 50% (95% confidence interval, 25-75%) and 25% (10-40%) of overlap contradictions, which required arithmetic on the anatomy. All four systems submitted every feasible control, but 24% of these plans missed a hard goal that the stored plan had met. In four SAGE runs, one goal failed at every attempt; the agent identified it in every reply but had no action for stopping. Conclusions: Favorable aggregate scores concealed specific decision failures. Planning agents should be evaluated on their decisions as well as their dose and given an explicit action for stopping and requesting review.

Humza Nusrat, Kundan Thind

Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan, USA · Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA

Medical AI Agents LLM Agent Evaluation LLM Decision-Making AI Agent Benchmarks

Ask PDF

Oct 20, 2026

A Radiation-Hardened, Fault-Aware Quantized Neural Network That Self-Heals

Deep learning (DL) models are increasingly being deployed into radiation-challenging environments in spacecraft, particle accelerators, and, soon, space-based data centers. Instantiated as a learned orchestration of numbers stored in persistent memory, models are susceptible to continual degradation as high-energy particles corrupt data. At inference time, new data passed through the network fundamentally break the off-line assumptions and guarantees of a static model determined during training. To preserve the quality and efficacy of the models, fault tolerance against bit flips is a clear objective. In this work we developed a runtime method for repairing the quantized weights of neural networks (NNs) after bit flips on commodity, commercial off-the-shelf (COTS) hardware, without storing extra replicas. Our work extends the well-established Fault-Aware Training (FAT) paradigm by adding an alignment penalty to concentrate undamaged activations and weights into a low-rank subspace, and we then reconstruct flagged corrupted coordinates from the uncorrupted survivors. We call this method {\it Projection Healing} and apply it to six regression benchmarks on quantized NNs. Holding a FAT checkpoint fixed and toggling only the runtime heal gives a further

{\sim}23\%

reduction in the native-unit excess integral. Across those datasets the method extends the operational scrub interval by a median of

2.04\times

at a

+5\%

RMSE degradation and

1.96\times

+50\%

, compared with

1.62\times

and

1.07\times

for classical Single Error Correction, Double Error Detection (SECDED). This introduces a new path towards designing resilient data-driven models, complementing established hardware redundancy, ECC, and FAT-based techniques.

Peter Xiangyuan Ma, Joshua S. Bloom

Department of Astronomy, University of California, Berkeley, CA 94720, USA

Neural Network Fault Tolerance

Ask PDF

Abstract

Weight-Space Analysis

Explore similar work

Weight Oracles: Reading Neural Network Weights with Language Models

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

A Radiation-Hardened, Fault-Aware Quantized Neural Network That Self-Heals