Weight-Space Programming for Improving Image-Restoration Diffusion Models: Technical Conjectures and a Recursive Improvement Loop
Abstract
Can an LLM directly edit the weights of a diffusion-based image-restoration model to improve restoration quality---without gradients, paired data, or fine-tuning? Recent experiments show frontier LLMs can fix structural bugs by modifying as little as of parameters, but cannot instill new capabilities. We ask whether weight-space programming (WSP) can move from debugging to improvement in the precise-regression setting of image restoration (deblurring, super-resolution, denoising, inpainting). Restoration diffusion UNets present a middle ground: low-level signal-processing layers are concentrated enough for targeted weight-direction edits, while high-level semantic layers remain distributed and resistant to sparse modification. We propose three technical conjectures: (C1) an LLM can localize the layer behind a quality gap (over-smoothing, color shift, hallucinated texture) within a 200 forward-pass budget; (C2) editing a small set of identified weight directions (e.g., scaling high-frequency filter norms, correcting output-bias channels) can measurably improve PSNR and perceptual quality; (C3) a recursive loop---diagnose edit evaluate re-diagnose---can compound gains across rounds, forming a minimal recursive self-improvement (RSI) loop for vision. We analyze the required AI capability (arguing targeted edits need not superintelligence, while general capability generation does) and detail the challenges of precise pixel-level regression. We provide a PyTorch-style deployment pipeline and outline a benchmark. This manuscript is conjectural, intended to open a new subtopic for image restoration.
Figures & tables
| Quality gap | Concentration | Sparse edit works? |
|---|---|---|
| Color cast / white balance | Concentrated | Yes (output bias) |
| Checkerboard / upconv artifact | Concentrated | Yes (transpose conv weights) |
| Halos on edges | Semi-concentrated | Maybe (skip connection scale) |
| Over-sharpening / ringing | Semi-concentrated | Maybe (high-freq filter norm) |
| Global over-smoothing | Distributed | No (needs retraining) |
| Hallucinated texture | Distributed | No (semantic prior issue) |
| Conjecture | What it does | Expected gain |
|---|---|---|
| C1: Diagnosis | Locate responsible layer(s) | – fwd passes |
| C2: Sparse edit | Improve quality via weight delta | – dB PSNR |
| C3: Recursive loop | Compound improvements | – dB total |
| Dimension | WSP | LoRA | Full FT |
|---|---|---|---|
| Gradient needed? | No | Yes | Yes |
| Params edited | – | ||
| Data needed | Feedback set | Paired dataset | Paired dataset |
| Speed | Minutes | Hours | Days |
| Best for | Targeted fixes | Domain adaptation | Full retrain |