Weight-Space Programming for Medical Imaging: Do We Still Need Training Data?
Authors: Maximilian Lindholz, Richard Ruppel, Annette Milnik, Sebastian Halskov, Semil Eminovic, Charlie A. Hamm, Robin Schmidt, Anika Knupfer, Jana Hutter, Johanna P. Müller, Carolina Dominguez Aleixo, Jawed Nawabi, Jens Vogel-Claussen, Tobias Penzkofer
Organizations: Charité – Universitätsmedizin Berlin, Germany · Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Germany · University of Basel, Switzerland · Experimental and Clinical Research Center (ECRC) of Charité and Max Delbrück Center, Berlin, Germany · Hannover Medical School, Germany · Leibniz University Hannover, Germany · King’s College London, United Kingdom · German Center for Lung Research (DZL), Hannover, Germany
Can a large language model (LLM) write the parameters of a medical image classifier from medical knowledge alone, without any training data? Such classifiers normally require labelled images, which are costly and often unavailable. We investigate this question through weight-space programming (WSP), in which an LLM writes or adapts the weights and biases of a neural network directly, without optimisation on training data. Claude Opus 5.5 and GPT-6 Astra each generate the parameters of a small LeNet-style CNN in a single reply, without tool use and without access to any image. We compare these networks against zero-shot MedSigLIP and against the same architecture trained on labels using the Adam optimiser, across 7 classification tasks in four modalities: chest X-ray, breast ultrasound and abdominal CT (all from MedMNIST), and pelvic MRI (SynthUterus). The LLM-written networks perform above chance on all six MedMNIST tasks, with a mean test AUC of 0.719 for Opus and 0.711 for Astra. Both models outperform MedSigLIP on 5 of 6 tasks while using more than 10,000 times fewer parameters. However, the same network trained with Adam on only 256 labelled images achieves a higher AUC on every MedMNIST task (mean AUC 0.828). On SynthUterus, Opus attains an AUC of 0.764, whereas Astra falls below chance (AUC 0.136) owing to a systematic error. These results show that LLMs can write functional medical classifiers, but that they do not yet replace training data. We release the protocol, logs and all generated weights at https://github.com/MaximilianLindholz/weight-space-programming-medical.
Figures & tables
Figure 1: The 7 medical imaging tasks ( Sec. 3.1 ), two examples each. For display only, images are shown at higher resolution (MedMNIST+ [ 48 ] at 64 or 224 pixels, SynthUterus at its released 96); the networks see the same images at 28×28 .
Step
Operation
Output
Parameters
Input
image scaled to [−1,1]
C×28×28
–
1 Convolution
K filters 3×3 (pad 1)
K×28×28
K(9C+1)
2 Rectification
ReLU
K×28×28
–
3 Pooling
average 4×4 , stride 4
K×7×7
–
4 Read-out
linear, n×K maps 7×7
n
n(49K+1)
Output
sigmoid or softmax
n
–
Table 1: ProgNet, layer by layer. C input channels, K filters, n outputs.
LLM-written, 3 calls
No labels
Adam (same network)
ResNet
Dataset
Modality
Cls.
Opus
Astra
MedSigLIP
256 lab.
full
-18
Chest ∙
Chest X-ray
14 (ML)
0.552 ± 0.015
0.538 ± 0.008
0.665
0.553
0.654
0.768
Pneumonia ∙
Chest X-ray
2
0.822 ± 0.036
0.854 ± 0.012
0.273
0.914
0.934
0.944
Breast ∙
Breast US
2
0.670 ± 0.009
0.677 ± 0.067
0.643
0.820
0.841
0.901
OrganA ∙
Abdominal CT
11
0.782 ± 0.000
0.763 ± 0.004
0.656
0.896
0.982
0.997
OrganC ∙
Abdominal CT
11
0.731 ± 0.017
0.712 ± 0.022
0.673
0.908
0.983
0.992
Table 2: Test AUC of LLM-written networks and baselines. LLM-written : weights written in a single tool-free reply, mean ± sd over three calls. No labels : zero-shot MedSigLIP-448. Adam : the same ProgNet trained on 256 labels or on the full training set; the published ResNet-18 (grey) is shown for reference. Bold: best without labels per row. ∙ High, ∘ low risk of LLM training-data exposure. ‡ 117 labels (all training cases), mean of 3 seeds. ML: multi-label.
Figure 2: Where the LLM-written and Adam-trained networks (256 labels) look for selected classes (one map per class and network). Per 7×7 grid cell: contribution to the class logit in test cases of the class minus in the other cases (red: evidence for the class; each map scaled to its own maximum). Opus and Astra: first of their three calls each; below each column: correlation r of each LLM’s map with Adam’s (Opus / Astra).
Figure 3: LLM-written vs . Adam-trained weights. (a) Distribution of all weights of the LLM-written networks (Opus and Astra, three calls each) and of the 256-label Adam networks (SynthUterus: 117 labels), each scaled by its network’s largest weight, pooled over the 7 tasks (log scale). Written weights are mostly exactly zero and take few distinct values. (b) The ten SynthUterus convolution filters (first call each): Opus writes four oriented edge detectors and leaves six filters unused (all zero), Astra uses all ten, Adam learns smooth mixed patterns.
Figure 4: The SynthUterus task: anteflexion (left) vs . retroflexion (right), defined by the angle between the cervical canal and the uterine cavity. Two illustrative training or validation cases per class (the data owner’s picks, or the most similar kept case where de-duplication removed a pick), at the released 96×96 resolution (top) and as the 28×28 network input (bottom). Both radiologists rated each case realistic, correct class, free of anomalies and artefacts.
Recent work shows that large language models (LLMs) can operate directly on the parameters of other neural networks. They can read what a network was trained to do, fix a network that fails, and write working networks from scratch, all without gradient descent. This capability has been termed Weight-Space Programming (WSP). Exploring this new paradigm systematically requires a robust benchmark. We introduce WEIGHTBENCH, a benchmark that evaluates LLMs of increasing capability on 42 WSP tasks across five tracks: reading, fixing and writing weights, recognizing edits that are impossible, and data-free compression. The networks whose weights the LLMs read and edit range from 2K-parameter control policies to 7B-parameter language models. Experimental benchmark results show that frontier agents can already read small networks and write working control policies without ever running them, but cannot yet patch language models, compress them without data, or tell an impossible edit from a hard one. On all 42 tasks, Claude Opus 5.5 reaches a macro score of 0.535 (Sonnet 5.5 0.466, Haiku 5.5 0.402, DeepSeek v4-flash 0.215); on the 38 tasks it was run on, GPT-6.1-sol reaches 0.568 against Opus's 0.546. Code, the task registry, our OS-level sandbox harness, and the live leaderboard are publicly available at https://github.com/Aayush-Marasini/WeightBench.
Aayush Marasini, Barsat Khadka, Nick Rahimi
The University of Southern Mississippi, Hattiesburg, MS 39401, USA
Large language models (LLMs) have recently been reported to read and write the raw weights of neural networks directly, a capability termed weight-space programming. We ask whether the reading side of this capability reflects reasoning about weight flows or a combination of local arithmetic at the network's edges and memorized priors over public datasets. We test Claude Opus 5.5 on small convolutional networks of about 125,000 parameters trained on five image datasets, providing every weight as plain text with no tools, no forward passes, and no dataset or class names. The model performs accurate local calculations: batch-normalization statistics lead it to infer about three optimizer steps on networks stopped after three updates, and output-layer sign patterns recover the vehicle/animal split of CIFAR-10. The sessions report reading the first and last layers closely and sampling the middle, and several claims built on partial tracing fail verification, although we did not run a direct forward-pass test. Semantic judgments lean on familiar datasets: the model names CIFAR-10 for CIFAR-10 networks but also for all three fast-food networks, relabels KMNIST as MNIST, reads a 99.57%-accurate SAT-6 network as CIFAR-10 animal photos, and calls "frog" the output favored by a feature that fires most on trees. Permuting CIFAR-10's labels preserves the output split at the new indices, but every class pair the model interprets is then wrong, indicating reliance on remembered label order. These are exploratory results from mostly single runs. We conclude that zero-shot weight reading by this model is better described as local arithmetic plus dataset priors than as weight-flow tracing, and we provide a protocol for auditing future weight-space claims.
Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine. However, their application in medical settings remains limited by the scarcity of visual question answering (VQA) datasets that capture clinical reasoning and explicit image-text alignment. Here, we leverage de-identified medical images and expert commentaries shared through clinician-oriented online resources. By combining an advanced LLM with clinician-in-the-loop verification, we established a rigorous pipeline to construct ThoughtMed-1M, a long-form medical VQA dataset containing over one million VQA pairs and designed to capture structured clinical reasoning and medical image-text alignment. To demonstrate its utility, we developed a FOundational LLM Trained on ThoughtMed-1M (FOLTMed). FOLTMed achieved state-of-the-art performance across 42 medical VQA benchmark datasets, with a macro accuracy of 85.4 percent. It also generated more clinically coherent responses on the ThoughtMed-1M test set, outperforming state-of-the-art models by 3 to 5 percent across factuality and similarity metrics, highlighting a scalable paradigm for advancing research on clinically grounded multimodal LLMs.
Lingxuan Hou, Yuhua Xie, Yue Hu +14
College of Biomedical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Engineering, Computer & Mathematical Sciences, Auckland University of Technology, Auckland, New Zealand · Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China +2