Can an LLM directly edit the weights of a diffusion-based image-restoration model to improve restoration quality---without gradients, paired data, or fine-tuning? Recent experiments show frontier LLMs can fix structural bugs by modifying as little as 0.016% of parameters, but cannot instill new capabilities. We ask whether weight-space programming (WSP) can move from debugging to improvement in the precise-regression setting of image restoration (deblurring, super-resolution, denoising, inpainting). Restoration diffusion UNets present a middle ground: low-level signal-processing layers are concentrated enough for targeted weight-direction edits, while high-level semantic layers remain distributed and resistant to sparse modification. We propose three technical conjectures: (C1) an LLM can localize the layer behind a quality gap (over-smoothing, color shift, hallucinated texture) within a 200 forward-pass budget; (C2) editing a small set of identified weight directions (e.g., scaling high-frequency filter norms, correcting output-bias channels) can measurably improve PSNR and perceptual quality; (C3) a recursive loop---diagnose → edit → evaluate → re-diagnose---can compound gains across rounds, forming a minimal recursive self-improvement (RSI) loop for vision. We analyze the required AI capability (arguing targeted edits need not superintelligence, while general capability generation does) and detail the challenges of precise pixel-level regression. We provide a PyTorch-style deployment pipeline and outline a benchmark. This manuscript is conjectural, intended to open a new subtopic for image restoration.
Figures & tables
Quality gap
Concentration
Sparse edit works?
Color cast / white balance
Concentrated
Yes (output bias)
Checkerboard / upconv artifact
Concentrated
Yes (transpose conv weights)
Halos on edges
Semi-concentrated
Maybe (skip connection scale)
Over-sharpening / ringing
Semi-concentrated
Maybe (high-freq filter norm)
Global over-smoothing
Distributed
No (needs retraining)
Hallucinated texture
Distributed
No (semantic prior issue)
Table 1: Expected editability of common restoration quality gaps.
Conjecture
What it does
Expected gain
C1: Diagnosis
Locate responsible layer(s)
20 – 50 fwd passes
C2: Sparse edit
Improve quality via weight delta
+0.2 – 0.5 dB PSNR
C3: Recursive loop
Compound improvements
+0.5 – 1.5 dB total
Table 2: Summary of the three conjectures.
Dimension
WSP
LoRA
Full FT
Gradient needed?
No
Yes
Yes
Params edited
<0.1%
1 – 10%
100%
Data needed
Feedback set
Paired dataset
Paired dataset
Speed
Minutes
Hours
Days
Best for
Targeted fixes
Domain adaptation
Full retrain
Table 3: WSP vs. LoRA vs. full fine-tuning for restoration.