What Does Weight-Space Programming Add to Diffusion Editing? Program Construction versus Score-Guided Refinement
Organizations: Fudan University · Northwest University · Tencent · Shanghai Institute for Mathematics and Interdisciplinary Sciences
Abstract
Weight-space programming (WSP) lets a language model construct executable weight updates for another network. We study diffusion concept editing through an interface in which programs define training contexts, preservation constraints and update directions, while a shared numerical fitter selects their coefficients. We compare generated programs with published editors, a shared-panel hand-written baseline and budget-matched search over conventional edits on 24 TIMED test edits for Stable Diffusion 1.4. The preregistered rewrite pipeline shows no advantage over the strong baseline. In an exploratory comparison, one-shot Astra programs improve fresh-scene VLM joint success, which requires subject presence together with whole-image target matching, by +10.0 points over the shared-panel baseline, while CLIP relative match shows no detectable change. These programs organise adjustable directions around protected text-state subspaces. Probing and rewriting the Astra programs then raises CLIP by +6.2 points while joint success changes by -9.7. On a fixed 144-image sample, five additional evaluators give point estimates in the same directions under whole-image target matching; checks of subject-local change give less decisive differences. Together, these results identify useful behaviour in complete generated programs and show that score-guided revision need not preserve it: program construction and revision should be evaluated as distinct stages of a WSP pipeline.
Figures & tables
| unit | update directions | coefficients | shares with programs |
|---|---|---|---|
| TIME, UCE | published closed form | fixed | — |
| UCE+P | UCE with the shared panel as preserved concepts | fixed | protection panel |
| U1b (hand-written) | TIME + 6 further templates + preservation term on the shared panel, split into 4 layer groups | CMA-ES, 24 evaluations | panel, fitter, objective, selection |
| RS (random search) | regulariser, protection weight, strength, template/panel/layer-group subsets sampled from the conventional closed-form family | none: each sample is one candidate, 24 candidates | panel, objective, selection, probe seeds |
| U2 (one-shot LLM) | program | CMA-ES, 24 evaluations | — |
| U3 (probe + rewrite) | second program, after the probing record | CMA-ES, 24 evaluations | — |
| CLIP relative match (ViT-bigG-14, TIMED convention) | VLM joint (Qwen3.8-27B: subject present matches target description) | |
|---|---|---|
| native TIMED prompts | efficacy, generality, specificity | joint efficacy, joint generality |
| fresh-scene prompts | generality; specificity on fresh negatives | fresh-scene joint success; subject presence |
| Native prompts | Fresh positives | Fresh neg. | ||||
| J-eff | J-gen | F-joint | F-subj | F-CLIP | F-spec | |
| A Unit means (%) | ||||||
| TIME | 71.2 | 38.1 | 28.0 | 72.1 | 55.1 | 71.7 |
| UCE | 40.6 | 24.5 | 21.5 | 69.6 | 46.9 | 73.0 |
| UCE+P | 39.6 | 23.8 | 20.8 | 71.5 | 40.3 | 86.8 |
| U1b | 74.7 | 48.3 | 27.7 | 64.0 | 81.2 | 78.5 |
| Construction: U2-Astra U1b | Revision: U3-Astra U2-Astra | |||
|---|---|---|---|---|
| Evaluator | Full | Local | Full | Local |
| Gemini 4 carbon | +14.6 [+2.1, +27.1] | +4.2 [ 14.6, +20.8] | 16.7 [ 31.2, 2.1] | 8.3 [ 25.0, +10.4] |
| Gemini 3.1 pro | +16.7 [0.0, +33.3] | +6.2 [ 8.3, +20.8] | 18.8 [ 33.3, 4.2] | 8.3 [ 25.0, +8.3] |
| GPT 6.1 sol | +12.5 [0.0, +25.0] | +12.5 [ 2.1, +29.2] | 22.9 [ 39.6, 6.2] | 0.0 [ 14.6, +16.7] |
| Claude Opus 5.5 | +10.4 [ 4.2, +25.0] | +4.2 [ 12.5, +18.8] | 2.1 [ 16.7, +12.5] | +4.2 [ 10.4, +20.8] |
| Qwen3 VL 235B | +10.4 [ 2.1, +22.9] | +2.1 [ 4.2, +8.3] | 18.8 [ 35.4, 2.1] | +2.1 [ 8.3, +14.6] |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| endpoint | all (24) | object (18) | other (6) |
|---|---|---|---|
| VLM joint success | +10.0 [+4.7, +15.4] | +7.2 [+1.7, +12.7] | +18.4 [+7.3, +29.5] |
| VLM subject present | +4.8 [ 4.8, +14.4] | +0.2 [ 9.6, +9.0] | +18.4 [ 2.4, +42.4] |
| VLM matches target description | +8.9 [+4.2, +14.1] | +6.7 [+2.1, +11.3] | +15.6 [+3.5, +28.5] |
| CLIP relative match | 0.6 [ 7.3, +6.4] | 1.0 [ 6.8, +4.1] | +0.7 [ 19.4, +24.7] |
| CLIP success subject | +3.4 [ 4.4, +11.1] | 1.0 [ 9.1, +6.8] | +16.7 [+1.4, +31.9] |
| TIME | U1b | U3-Opus | U3-Opus TIME (paired, 95% CI) | |
|---|---|---|---|---|
| development panel (16 self-built edits) | 41.1 | 66.1 | 66.7 | +25.5 [+11.5, +40.6] |
| TIMED test (24 pre-drawn edits, main comparison) | 71.2 | 74.7 | 74.3 | +3.1 [-7.3, +13.2] |
| code | U2-Astra | U2-Opus | U3-Astra | U3-Opus |
|---|---|---|---|---|
| calls TIME token alignment (ctx.align) | 24/24 | 24/24 | 3/24 | 16/24 |
| encodes the empty prompt as a text | 24/24 | 0/24 | 20/24 | 0/24 |
| calls an SVD | 24/24 | 0/24 | 22/24 | 0/24 |
| explicit orthogonal projection X H(H ⊤ X) | 24/24 | 0/24 | 18/24 | 0/24 |
| solves a linear system | 24/24 | 24/24 | 16/24 | 21/24 |
| fitter selected a zero update | 0/24 | 4/24 | 0/24 | 1/24 |
| candidate | macro balanced accuracy [95% CI] | accuracy | abstain rate |
|---|---|---|---|
| Qwen3.8-27B, threshold 0.5 (selected) | 0.955 [0.904, 0.977] | 0.914 | 0.000 |
| Qwen3.8-27B, abstain band 0.3–0.7 | 0.900 [0.816, 0.947] | 0.865 | 0.097 |
| Gemini-4-argon | 0.864 [0.784, 0.940] | 0.850 | 0.082 |
| contrast | mean [95% CI] (pp) | p | Holm p | rejected |
|---|---|---|---|---|
| CLIP generality advantage, rule editor TIME | +21.4 [+14.9, +28.6] | 2.0e-05 | 1.4e-04 | yes |
| subject presence, rule editor TIME | -14.5 [-24.2, -5.6] | 3.8e-04 | 1.5e-03 | yes |
| attenuation, rule editor TIME | +21.7 [+13.5, +30.6] | 2.0e-05 | 1.4e-04 | yes |
| attenuation, TIME target-prompt ref. | -10.4 [-16.8, -4.5] | 1.0e-04 | 5.0e-04 | yes |
| attenuation, rule editor target-prompt ref. | +11.3 [+4.0, +19.1] | 1.6e-03 | 4.9e-03 | yes |
| subject-filtered CLIP success, TIME target-prompt ref. (both filtered) | -8.6 [-15.1, -1.5] | 1.9e-02 | 1.9e-02 | yes |
| arm | human: subject present | human: attribute on subject | human: both | judge: subject present | judge-absent human-present |
|---|---|---|---|---|---|
| unedited | 98.3 | 18.7 | 16.2 | 83.5 | 9/16 |
| target-prompt ref. | 80.0 | 76.6 | 60.9 | 67.3 | 7/16 |
| TIME | 90.7 | 46.1 | 46.7 | 70.5 | 9/16 |
| rule editor | 64.3 | 55.3 | 40.1 | 58.6 | 4/16 |
| unit | LLM output tokens (all attempts) | first-try success | median program run time (s) | candidate evaluations / edit | median fitting wall time / edit (s) |
| TIME | — | — | — | 0 | — |
| UCE | — | — | — | 0 | 182 |
| UCE+P | — | — | — | 0 | 181 |
| U1b | — | — | — | 24 | 2740 |
| RS | — | — | — | 24 | 990 |
| U2-Astra | 274k | 24/24 | 54 | 24 | 2087 |
| unit | as reported | affected prompts dropped | change |
|---|---|---|---|
| target-prompt ref. | 92.6 | 92.6 | 0.0 |
| TIME | 63.9 | 63.8 | -0.1 |
| UCE | 49.2 | 50.4 | +1.2 |
| UCE+P | 40.5 | 41.8 | +1.3 |
| U1b | 90.5 | 90.8 | +0.3 |
| RS | 79.8 | 79.6 | -0.2 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| native joint efficacy | ||||||
| U3-Opus U3-Astra | +1.0 [ 5.9, +9.7] | 0.8939 | 1.000 | 1.000 | 1.000 | 0.950 |
| U3-Opus U2-Opus | +14.6 [ 0.7, +31.9] | 0.1120 | 1.000 | 0.237 | 1.000 | 0.189 |
| U3-Opus U2-Astra | +0.7 [ 7.3, +9.7] | 0.9425 | 1.000 | 1.000 | 1.000 | 0.980 |
| U3-Opus RS | 0.7 [ 9.0, +8.0] | 0.9381 | 1.000 | 1.000 | 1.000 | 0.979 |
| U3-Opus U1b | 0.3 [ 6.2, +5.9] | 1.0000 | 1.000 | 1.000 | 1.000 | 1.000 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 2 of 8: native joint generality | ||||||
| U3-Opus U3-Astra | 2.4 [ 6.0, +1.0] | 0.2180 | 1.000 | 0.280 | 1.000 | 0.315 |
| U3-Opus U2-Opus | +3.0 [ 6.4, +12.8] | 0.5650 | 1.000 | 0.656 | 1.000 | 0.689 |
| U3-Opus U2-Astra | 7.0 [ 12.1, 1.7] | 0.0189 | 0.362 | 0.038 | 1.000 | 0.046 |
| U3-Opus RS | 1.2 [ 8.4, +6.4] | 0.7803 | 1.000 | 0.802 | 1.000 | 0.868 |
| U3-Opus U1b | 6.0 [ 10.3, 1.6] | 0.0181 | 0.362 | 0.038 | 1.000 | 0.046 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 3 of 8: fresh joint success | ||||||
| U3-Opus U3-Astra | +2.9 [ 0.8, +6.4] | 0.1547 | 1.000 | 0.278 | 1.000 | 0.240 |
| U3-Opus U2-Opus | +0.4 [ 6.8, +7.5] | 0.9267 | 1.000 | 0.988 | 1.000 | 0.971 |
| U3-Opus U2-Astra | 6.9 [ 12.7, 1.5] | 0.0283 | 0.794 | 0.100 | 1.000 | 0.065 |
| U3-Opus RS | 1.2 [ 8.2, +4.8] | 0.7578 | 1.000 | 0.853 | 1.000 | 0.856 |
| U3-Opus U1b | +3.1 [ 3.0, +8.6] | 0.3316 | 1.000 | 0.477 | 1.000 | 0.442 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 4 of 8: fresh CLIP generality | ||||||
| U3-Opus U3-Astra | 5.9 [ 14.6, +1.6] | 0.1982 | 1.000 | 0.235 | 1.000 | 0.296 |
| U3-Opus U2-Opus | +17.6 [+5.4, +31.9] | 0.0123 | 0.221 | 0.023 | 1.000 | 0.033 |
| U3-Opus U2-Astra | +0.3 [ 7.8, +7.9] | 0.9496 | 1.000 | 0.968 | 1.000 | 0.982 |
| U3-Opus RS | +9.5 [+1.1, +18.8] | 0.0498 | 0.597 | 0.072 | 1.000 | 0.098 |
| U3-Opus U1b | 0.3 [ 8.2, +6.5] | 0.9677 | 1.000 | 0.968 | 1.000 | 0.988 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 5 of 8: fresh-negative CLIP specificity | ||||||
| U3-Opus U3-Astra | +2.0 [ 1.9, +6.2] | 0.3607 | 1.000 | 0.481 | 1.000 | 0.477 |
| U3-Opus U2-Opus | 5.5 [ 11.3, 0.0] | 0.0715 | 1.000 | 0.123 | 1.000 | 0.130 |
| U3-Opus U2-Astra | 4.2 [ 9.1, +0.3] | 0.1058 | 1.000 | 0.173 | 1.000 | 0.181 |
| U3-Opus RS | +3.1 [ 2.0, +7.9] | 0.2428 | 1.000 | 0.350 | 1.000 | 0.343 |
| U3-Opus U1b | +1.7 [ 2.4, +5.8] | 0.4308 | 1.000 | 0.535 | 1.000 | 0.549 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 6 of 8: fresh subject presence | ||||||
| U3-Opus U3-Astra | +1.1 [ 3.8, +6.1] | 0.6899 | 1.000 | 0.892 | 1.000 | 0.803 |
| U3-Opus U2-Opus | 8.6 [ 16.1, 1.6] | 0.0314 | 1.000 | 0.210 | 1.000 | 0.068 |
| U3-Opus U2-Astra | 6.9 [ 15.2, +1.2] | 0.1297 | 1.000 | 0.359 | 1.000 | 0.215 |
| U3-Opus RS | 8.1 [ 15.6, 0.7] | 0.0528 | 1.000 | 0.211 | 1.000 | 0.101 |
| U3-Opus U1b | 2.1 [ 10.4, +5.7] | 0.6440 | 1.000 | 0.892 | 1.000 | 0.757 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 7 of 8: native CLIP efficacy | ||||||
| U3-Opus U3-Astra | 5.2 [ 13.9, +0.3] | 0.2195 | 1.000 | 0.272 | 1.000 | 0.316 |
| U3-Opus U2-Opus | +17.4 [+3.5, +33.3] | 0.0558 | 0.837 | 0.091 | 1.000 | 0.106 |
| U3-Opus U2-Astra | 4.9 [ 14.2, +0.7] | 0.3751 | 1.000 | 0.397 | 1.000 | 0.488 |
| U3-Opus RS | 0.3 [ 5.2, +3.8] | 1.0000 | 1.000 | 1.000 | 1.000 | 1.000 |
| U3-Opus U1b | 6.2 [ 15.6, 0.7] | 0.0788 | 1.000 | 0.118 | 1.000 | 0.139 |
| contrast | difference [95% CI] | Holm (end.) | BH (end.) | Holm (all) | BH (all) | |
|---|---|---|---|---|---|---|
| Table 13 , block 8 of 8: native CLIP generality | ||||||
| U3-Opus U3-Astra | 1.6 [ 7.1, +3.8] | 0.6024 | 1.000 | 0.620 | 1.000 | 0.726 |
| U3-Opus U2-Opus | +20.1 [+8.4, +33.3] | 0.0023 | 0.042 | 0.004 | 0.477 | 0.008 |
| U3-Opus U2-Astra | +1.9 [ 4.2, +7.9] | 0.5587 | 1.000 | 0.592 | 1.000 | 0.685 |
| U3-Opus RS | +7.2 [+2.3, +13.6] | 0.0081 | 0.121 | 0.013 | 1.000 | 0.024 |
| U3-Opus U1b | 3.5 [ 10.0, +2.4] | 0.3147 | 1.000 | 0.354 | 1.000 | 0.424 |