cs.CVNov 9, 2026

Weight-Space Programming for Medical Imaging: Do We Still Need Training Data?

Authors: Maximilian Lindholz, Richard Ruppel, Annette Milnik, Sebastian Halskov, Semil Eminovic, Charlie A. Hamm, Robin Schmidt, Anika Knupfer, 

Organizations: Charité – Universitätsmedizin Berlin, Germany · Berlin Institute of Health at Charité – Universitätsmedizin Berlin, Germany · University of Basel, Switzerland · Experimental and Clinical Research Center (ECRC) of Charité and Max Delbrück Center, Berlin, Germany · 

Abstract

Can a large language model (LLM) write the parameters of a medical image classifier from medical knowledge alone, without any training data? Such classifiers normally require labelled images, which are costly and often unavailable. We investigate this question through weight-space programming (WSP), in which an LLM writes or adapts the weights and biases of a neural network directly, without optimisation on training data. Claude Opus 5.5 and GPT-6 Astra each generate the parameters of a small LeNet-style CNN in a single reply, without tool use and without access to any image. We compare these networks against zero-shot MedSigLIP and against the same architecture trained on labels using the Adam optimiser, across 7 classification tasks in four modalities: chest X-ray, breast ultrasound and abdominal CT (all from MedMNIST), and pelvic MRI (SynthUterus). The LLM-written networks perform above chance on all six MedMNIST tasks, with a mean test AUC of 0.719 for Opus and 0.711 for Astra. Both models outperform MedSigLIP on 5 of 6 tasks while using more than 10,000 times fewer parameters. However, the same network trained with Adam on only 256 labelled images achieves a higher AUC on every MedMNIST task (mean AUC 0.828). On SynthUterus, Opus attains an AUC of 0.764, whereas Astra falls below chance (AUC 0.136) owing to a systematic error. These results show that LLMs can write functional medical classifiers, but that they do not yet replace training data. We release the protocol, logs and all generated weights at https://github.com/MaximilianLindholz/weight-space-programming-medical.

Figures & tables

Explore similar work

CardsList
  1. WeightBench: Evaluating Language Models as Weight-Space Programmers

    Nov 13, 2026Aayush Marasini, Barsat Khadka, Nick Rahimiweight-space programmingLLM Evaluation

  2. A visual large language foundational model for medical image recognition using clinician-contributed online resources

    Sep 7, 2026Lingxuan Hou, Yuhua Xie, Yue Hu +14Medical Visual Question AnsweringMedical VLMs