Spectral Steering: Local Inverse Geometry for Frozen Diffusion Generators
Organizations: University of Applied Sciences and Arts of Southern Switzerland (SUPSI), IDSIA · Universit`a della Svizzera italiana (USI) · Politecnico di Milano
Abstract
A frozen diffusion generator may need to change several output properties while preserving others. We study when guidance faithfully realizes such a structured request and whether local inverse correction improves its direction. Our account links this benefit to the interaction between the request and the objective-specific remaining-sampler Jacobian: conditioning alone is insufficient. Spectral steering combines a request-specific cosine-gain predictor with damped inverse correction and fixed-time relinearization for finite edits. Scalar nulls and matched-movement controls test the predicted mechanism. Across 8,923 instances in a retrospectively identified local regime, predicted and realized gains correlate at Spearman 0.79, versus -0.11 for condition number. Experiments span images, audio, robotics, and control and planning, including complex character motion generation. On 50 paired inpainting cases, four corrections improve unobserved-pixel PSNR over Adam by 3.25 dB at equal total derivative traversals. Together, the analysis and controls connect request-dependent geometry to directional gains and establish where these gains translate into downstream performance.
Figures & tables
| Data / simulator Diffusion model | Task / endpoint | Unst. | FO | SGD | Adam | L-BFGS | Spectral (K1) | Spectral (best K) |
| Vision | ||||||||
| CelebA-HQ DDPM EMA-256 | Inpainting PSNR | 16.55 | 19.61 | 25.57 | 25.98 | 25.76 | 23.69 | 27.75 (K4) |
| SR PSNR | 25.05 | 26.32 | 27.69 | 27.79 | 27.58 | 27.39 | 27.90 (K2) | |
| Deblur PSNR | 24.03 | 25.26 | 27.26 | 27.92 ns | 27.90 | 26.61 | 27.94 (K16) ns | |
| Multi-constraint PSNR | 25.81 | 26.12 | 27.69 | 28.61 | 28.54 | 26.89 | 27.75 (K4) | |
| Generated images Stable Diff. 1.5 | Structured edit [0-1] | 0.000 | 0.574 | 0.734 | 0.774 | 0.765 | 0.954 | 1.000 (K8) |
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Endpoint | Unst. | FO | K1 | K2 | K4 | K16 |
|---|---|---|---|---|---|---|---|
| Obstacle | error | 16.64 | 10.32 | 3.54 | 0.95 | 0.23 | 0.01 |
| success % | – | – | 18 | 88 | 98 | 100 | |
| – | – | 0.164 | 0.140 | 0.135 | 0.132 | ||
| Multi-objective | error | 10.62 | 4.85 | 2.52 | 1.40 | 0.79 | 0.19 |
| success % | – | – | 14 | 58 | 96 | 98 | |
| – | – | 0.188 | 0.204 | 0.210 | 0.185 |
| Task | Quantity | FO | K1 | 95% CI | W/L | ||
|---|---|---|---|---|---|---|---|
| Obstacle | fidelity cosine | 0.458 | 0.976 | 50/0 | |||
| leakage | 1.215 | 0.219 | 50/0 | ||||
| bone drift (m) | 0.011 | 0.011 | 27/23 | 0.67 (ns) | |||
| Multi-objective | fidelity cosine | 0.701 | 0.931 | 50/0 | |||
| leakage | 1.477 | 0.507 | 50/0 | ||||
| bone drift (m) | 0.011 | 0.010 | 27/23 | 0.67 (ns) |
| Obstacle | Multi-objective | |||
|---|---|---|---|---|
| Method | cosine | residual | cosine | residual |
| Spectral (K16) | 1.000 | 0.001 | 0.996 | 0.044 |
| DPS-adapted | 0.290 | 1.605 | 0.575 | 1.777 |
| UG-adapted | 0.435 | 1.466 | 0.325 | 1.619 |
| Accounting | Paired methods | PSNR dB | 95% interval | Spectral win % |
|---|---|---|---|---|
| VJP count | Spectral (K1) vs Adam@16 | 26 | ||
| Spectral (K4) vs Adam@64 | 78 | |||
| Total traversals | Spectral (K1) vs Adam@32 | 18 | ||
| Spectral (K4) vs Adam@128 | 86 |
| Method | Passes | Collision % | Request error |
|---|---|---|---|
| Unsteered | 0 | 100.0 | 0.350 |
| FO | 45 | 79.5 | 0.522 |
| Spectral (K1) | 45 | 52.5 | 0.677 |
| Spectral (K4) | 180 | 11.8 | 0.417 |
| Universal Guidance, fixed | 16 | 9.5 | 0.756 |
| Universal Guidance, fixed | 256 | 1.1 | 0.442 |
| Vision operator | UG below prior | Spectral below prior |
|---|---|---|
| Inpainting | 4/50 | 0/150 |
| super-resolution | 50/50 | 0/150 |
| Deblurring | 40/50 | 0/150 |
| Multi-constraint | 50/50 | 0/150 |
| Task / source | Measurement map and target | Evaluation |
|---|---|---|
| Random-mask inpainting (V0) | Retained pixels in DDPM data units. Target: ; unobserved pixels are omitted, not preservation coordinates | PSNR on unobserved pixels |
| super-resolution (V0) | DPS bicubic antialiased resizer . Target: | Full-image PSNR |
| Gaussian deblurring (V0) | Reflection-padded Gaussian convolution . Target: | Full-image PSNR |
| Two-region restoration (V0) | . Target: Restore low-resolution ; preserve exactly observed | PSNR on |
| Multi-region color control (V0) | Mean CIELab coordinates in each named region, . Target: Fixed per-region color displacement; included preservation regions request zero change | Dense on preserved pixels |
| Spatial-statistic editing (V1) | under the smooth luminance weight . Target: Shift the weighted center and spreads; is calibrated from frozen pilot statistics | Objective fidelity |
| Task / source | Measurement map and target | Evaluation |
|---|---|---|
| Localized waveform EQ (A0) | equation 11 , dB cells. Target: Primary request: dB in one cell and dB in another; remaining 22 cells request dB | Requested-cell dB error; LSD and SI-SDR to a reference edit |
| Disjoint-cell EQ ladder (A0) | Per-cell dB on disjoint log-mel cells. Target: Alternating dB pattern at the declared scales; included hold cells request change | Decoded-signal cell error and feasibility gate |
| Bandwidth extension (A1) | , a 601-tap Kaiser low-pass measurement. Target: | High-band LSD above the cutoff |
| Temporal inpainting (A1) | Observed samples of a centered binary temporal mask. Target: Match the observed samples; the missing interval is absent from | Gap-region SNR and splice-boundary discontinuity |
| Known-filter EQ restoration (A1) | log-spaced band energies . Target: , restoring the clean band balance | LTAS distance and waveform metrics |
| Missing-segment boundary match (A0) | Eight boundary-band levels in dB. Target: Match the generated segment’s beginning/end to the adjacent clips; interior coordinates request | Abrupt energy jump across the splice |
| Task / source | Measurement map and target | Evaluation |
|---|---|---|
| Panda Cartesian reach (R0) | Position ; orientation error , ; terminal and . Target: Move position; preserve orientation and terminal linear/angular velocity | Executed Cartesian target error |
| Panda checkpoint/via-point reach (R1) | , with or . Target: Independent checkpoint displacements or geometry-defined via-points | Executed closest-approach via error and final error |
| Reach with orientation hold (R0/R1) | . Target: Move position; request zero orientation change | Executed target error and tool rotation |
| Lateral move at constant height (R0/R1) | Terminal and four checkpoint heights . Target: Move laterally; hold start height at four times | Executed target error and maximum height drift |
| Unmodelled obstacle reach (R0) | with soft-minimum whole-arm clearance. Target: Request goal and clearance; preserve orientation | Contact-free execution and MuJoCo geom distance |
| Spatial locomotion target (R0/R1) | Midpoint/final , terminal , torso height, and pitch, each with its frozen physical scale. Target: Request intermediate/final position and speed; preserve height and pitch | Executed world-position error |
| Task / source | Measurement map and target | Evaluation |
|---|---|---|
| Pendulum stabilization (C0) | . Target: Target ; control effort is preserved | Real-dynamics replay: angle, speed, return, and effort |
| Pendulum reference tracking (C0) | Wrapped angle errors at fixed intermediate checkpoints and , plus . Target: Reach the declared reference angles and finish stationary | Real-dynamics replay and tracking error |
| CartPole target position (C0) | divided by frozen training-distribution scales. Target: Target ; upright orientation and terminal velocities are explicit zeros | Open-loop true-dynamics survival and terminal errors |
| Double-integrator waypoints (P0) | , . Target: Hit all requested positions; finish at | Executed goal/waypoint error and task success |
| Maze2D goal and via-points (P0) | with A* reference waypoints. Target: Reach requested free-space waypoints and goal; finish stationary | Executed goal distance, wall collision, path length |
| Dense obstacle navigation (P0) | , . Target: Request the new goal and zero velocity/clearance violations; preserve smoothness | Executed collision, clearance, goal error |
Diffusion Guidance
Diffusion guidance steers generative sampling toward prompts, rewards, measurements, safety constraints, or other desired outcomes.
See all 125 papers