cs.CVMar 7, 2026
SaveHY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editing
Abstract
Foundation models are transitioning from offline predictors to deployed systems expected to operate over long time horizons. In real deployments, objectives are not fixed: domains drift, user preferences evolve, and new tasks appear after the model has shipped. This elevates continual learning and instant personalization from optional features to core architectural requirements. Yet most adaptation pipelines still follow a static weight paradigm: after training (or after any adaptation step), inference executes a single parameter vector regardless of user intent, domain, or instance-specific constraints. This treats the trained or adapted model as a single point in parameter space. In heterogeneous and continually evolving regimes, distinct objectives can induce separated feasible regions over parameters, forcing any single shared update into compromise, interference, or overspecialization. As a result, continual learning and personalization are often implemented as repeated overwriting of shared weights, risking degradation of previously learned behaviors. We propose HY-WU (Weight Unleashing), a memory-first adaptation framework that shifts adaptation pressure away from overwriting a single shared parameter point. HY-WU implements functional (operator-level) memory as a neural module: a generator that synthesizes weight updates on-the-fly from the instance condition, yielding instance-specific operators without test-time optimization.
Figures & tables
Figure 1: Failure modes of static adaptation vs. conditional parameter generation. (a) Infeasible sharing. When heterogeneous objectives induce separated feasible regions in parameter space, a single shared update (Shared LoRA or SFT) is forced into compromise, instability, or dominance by high-frequency modes. (b) Over-specialization. Training a separate static adapter per domain avoids direct conflict but collapses into a narrow subspace and generalizes poorly under domain shifts. (c) Conditional generation. A generator routes each instance to an update , enabling inference over a family of parameter points rather than a single point.
| Framework concept | Instantiation in Part I (TI2I) | Where it is tested / evidenced |
| Stable invariants (long-term knowledge) | Frozen diffusion backbone (pretrained competence preserved during adaptation) | Static baselines show compromise or over-specialization under conflict (qualitative and quantitative), motivating preservation of the shared base |
| Static parameter memory (single-point adaptation) | SFT / Shared LoRA producing a fixed update | Conflict-controlled editing reveals compromise under sharing; gradient conflict analyses characterize incompatibility |
| Functional memory (operator-level) | Generator synthesizing LoRA updates per instance | Instance-alignment ablations isolate routing as mechanism (Table 7 ); weight-space structure analyses probe semantic organization |
| Routing signal / condition | Hybrid condition from image and instruction features | Instance dependence: same instruction requires different operator shifts depending on content; ablations show that misalignment collapses gains |
| Conditional family of operators | induced by (update family over instances) | Emergent parameter-space neighborhoods align with semantic similarity; clustering and local neighborhood consistency in generated parameters |
| Failure mode under heterogeneity | Separated feasible regions and incompatible gradients | Controlled conflicts (e.g., restoration vs. aging) and gradient-level conflict evidence |
Table 1: Concept-to-implementation mapping for Part I (TI2I). We map the memory-first framing to concrete modules, and we indicate where each concept is validated in Part I.
Figure 2: Comparison of training paradigms for hypernetwork-based parameter generation. (a) Learning to reconstruct parameters from pre-collected checkpoints via reconstruction loss. (b) Learning from pre-collected checkpoints with additional downstream task loss as auxiliary supervision. (c) Our approach: on-the-fly optimization of the parameter generator using only downstream task loss, without relying on pre-collected checkpoints.
Figure 3: Overview of the HY-WU pipeline. The framework extracts conditions from the source image and edit prompt, which are processed by a trainable Neural Network Transformer to synthesize instance-specific parameter tokens. These tokens are then detokenized into LoRA adapters and integrated into a frozen foundation model with , where indicates -th layer. The entire pipeline is optimized end-to-end, where the generator is updated via backpropagation of diffusion loss.
Figure 4: Parameter tokenization and detokenization. LoRA adapters are reorganized into a unified tensor . The red path reverses this process to reconstruct parameters from a tensor.
Figure 5: Architecture of the Neural Network Transformer. The left panel illustrates the overall pipeline, where parameter embeddings and extracted conditions (text/image) are processed through transformer blocks to generate LoRA parameters . The right panel details the internal structure of each block, featuring factorized self-attention to capture structural correlations, and cross-attention for condition injection. The final LoRA projection is zero-initialized to ensure training stability.
| NNT configuration | value |
| parameter count | 8.11B |
| number of layers | 24 |
| number of heads | 32 |
| hidden dimension | 4096 |
| intermediate dimension | 16384 |
Table 2: Configuration for Neural Network Transformer (NNT) and generated LoRA on HY-Image-3.0-Instruct (80B-A13B).
Figure 6: The GSB win-rate comparison of our HY-WU against other baseline models. Each bar reports the pairwise win rate of HY-WU against the corresponding baseline (rates sum to 100%). HY-WU demonstrates a significant performance advantage, particularly against established open-source competitors. Notably, HY-WU also achieves a clear lead over two closed-source competitors (marked with a lock icon), i.e. , Seedream-4.5-2K (55.6%) and GPT-Image-1.5 (55.5%), while remaining highly competitive against the state-of-the-art Nano-Banana series, proving its effectiveness in complex image editing tasks.
| model | public | GEdit-Bench-EN | GEdit-Bench-CN | ||||
| Q_SC | Q_PQ | Q_O | Q_SC | Q_PQ | Q_O | ||
| Seedream 4.5 | 7.920 | 7.427 | 7.587 | 7.735 | 7.442 | 7.445 | |
| GPT Image 1.5 | 7.966 | 7.595 | 7.699 | 7.853 | 7.606 | 7.637 | |
| Nano-Banana-Pro | 7.718 | 7.476 | 7.391 | 7.725 | 7.514 | 7.470 | |
| Step1X-Edit-v1.2 | 7.652 | 7.399 | 7.350 | 7.491 | 7.433 | 7.265 | |
| Qwen-Image-Edit-2511 | 7.898 | 7.380 | 7.530 | 7.723 | 7.335 | 7.378 | |
Table 3: Results on GEdit-Bench (Semantic Consistency as Q_SC, Perceptual Quality as Q_PQ and Overall Performance as Q_O). Best results are shown in bold and second-best results are underlined.
| model | public | add | adjust | extract | replace | remove | background | style | hybrid | action | overall |
| Seedream 4.5 | 4.63 | 3.92 | 4.29 | 3.10 | 3.93 | 4.03 | 3.20 | 3.25 | 3.94 | 3.87 | |
| GPT Image 1.5 | 4.77 | 4.49 | 4.44 | 3.54 | 4.37 | 4.37 | 3.30 | 3.50 | 3.94 | 4.16 | |
| Nano Banana Pro | 4.81 | 4.37 | 4.06 | 3.41 | 4.19 | 4.29 | 3.19 | 3.55 | 4.11 | 4.02 | |
| Step1X-Edit-v1.2 | 4.61 | 4.42 | 2.65 | 3.21 | 3.83 | 4.31 | 2.89 | 3.19 | 3.87 | 3.64 | |
| Qwen-Image-Edit-2511 | 4.69 | 4.24 | 4.05 | 3.28 | 3.02 | 4.33 | 2.94 | 3.50 | 3.80 | 3.77 | |
| LongCat-Image-Edit | 4.72 | 4.22 | 4.44 | 3.41 | 4.31 | 4.38 | 3.14 | 3.19 | 3.99 | 4.07 |
Table 4: Results on ImgEdit-Bench (Category-wise and Overall Performance). Best results are shown in bold and second-best results are underlined.
| model | public | alignment | consistency | structure | quality | overall |
| Seedream 4.5 | 4.55 | 4.02 | 4.24 | 3.89 | 4.18 | |
| GPT Image 1.5 | 4.78 | 3.95 | 4.22 | 3.83 | 4.20 | |
| Nano Banana Pro | 4.58 | 4.02 | 4.23 | 3.89 | 4.18 | |
| Step1X-Edit-v1.2 | 3.83 | 3.52 | 3.72 | 3.35 | 3.61 | |
| Qwen-Image-Edit-2511 | 4.54 | 3.86 | 4.07 | 3.71 | 4.04 | |
| LongCat-Image-Edit | 4.47 | 3.83 | 4.03 | 3.66 | 4.00 |
Table 5: Results on WU-Eval. Best results are shown in bold and second-best results are underlined.
Table 6: Ablation studies about foundation model and scaling behavior. (Foundation model is marked with gray stripe.) For foundation model ablation (a) and (b), HY-WU achieve consistent improvement across different model architecture, highlighting its universality. For scaling behavior ablation (c) and (d), results demonstrate a consistent positive scaling law across both LoRA parameter and Neural Network Transformer (NNT) parameter.
Figure 7: Visual comparison on multi-image editing tasks under consistency constraints. Each task requires transferring attributes from a reference image while preserving key invariants of the base image (e.g., identity, pose, background, or lighting). Competing methods often introduce inconsistencies such as identity drift, structural distortion, or imperfect blending. In contrast, HY-WU produces edits that better preserve subject identity, pose, and scene context while faithfully transferring the target attributes.
Figure 8: Behavioral evidence of structural compromise under shared static adaptation in conflict settings. For each conflicting task pair, we compare: (1) Single LoRA trained per objective, (2) Shared LoRA trained jointly on both objectives, and (3) PG trained jointly. Green check marks indicate correct directional behavior, and red crosses indicate failure. (a) Restoration vs. Aging. Restoration enhances clarity and recovers natural color, whereas aging introduces degradation and discoloration. Single LoRA models exhibit strong directional behavior when trained separately, but they fail on the opposing task. Shared LoRA produces intermediate outputs: restored images remain slightly desaturated, and aged outputs appear insufficiently degraded. PG performs well for both objectives. (b) Deblurring vs. Blurring. Deblurring restores high-frequency details, while blurring suppresses them. Shared LoRA yields softened results in both directions. In contrast, PG maintains sharper deblurring and better blurring effects. Across both settings, single LoRA models exhibit over-specialization behavior and fail on the opposing task. Shared adaptation converges to a compromise operator. PG maintains clearer objective-specific behavior across both objectives, suggesting that condition-dependent parameters reduce the need for compromise within shared parameter space.
Figure 9: Gradient-Level Conflict Structure across Editing Tasks. (a) Mean cosine similarity among tasks. We cluster similar tasks into C0 - C5 for visualization clarity. Each node represents a mean of gradient similarity among samples from two tasks. (b) Conflict ratio matrix, defined as the proportion of negative cosine values. Lighter color indicates higher conflict ratio. (c) Representative histogram distributions illustrating different modes, including intra-task pair (left) and inter-task pairs (middle and right). These distributions demonstrate clear pattern, e.g. , strong positive bias, negative bias, and bimodality alignment. Together, these statistics reveal that heterogeneous editing objectives impose structured and often incompatible optimization directions under shared parameters.
| model | conditional | win rate | alignment | consistency | structure | quality | overall |
| (a) base model | \cmark | - | 4.41 | 3.74 | 3.94 | 3.60 | 3.92 |
| (b) shared LoRA | \xmark | 51.7% | 4.41 | 3.83 | 3.99 | 3.67 | 3.98 |
| (c) SFT | \xmark | 51.5% | 4.45 | 3.84 | 4.02 | 3.65 | 3.99 |
| (d) average PG | \xmark | 48.0% | 4.40 | 3.74 | 3.94 | 3.62 | 3.92 |
| (e) shuffle PG | \cmark (misaligned) | 48.3% | 4.38 | 3.76 | 3.97 | 3.62 | 3.93 |
| (f) full PG | \cmark | 56.5% | 4.42 | 3.89 | 4.06 | 3.72 | 4.02 |
Table 7: Quantitative evaluation of instance-conditioned adaptivity. PG achieves the highest overall performance and dominates in consistency, structure, and quality, while alignment remains comparable across methods. SFT and Shared LoRA, though differing significantly in parameter capacity, yield similar overall performance. Average PG (using averaged generated LoRA across samples as a fixed parameter) collapses to the base-model level, and Shuffle PG (randomly permuting conditional inputs to PG during inference) degrades toward non-conditional baselines, indicating that gains come from input-aligned instance-specific parameter updates rather than architectural capacity.
Figure 10: Emergent semantic structure in parameter space learned by PG. Samples are clustered with K-means in multimodal semantic space (image + text embeddings) and color-coded and visualized in LoRA parameter space via t-SNE. Semantically related edits form coherent regions in parameter space, with representative examples highlighted. Note that clusters may consist of multiple sub-regions, indicating a hierarchical structure, yet remain globally distinct across editing semantics. This suggests that conditional parameter generation induces a structured parameter landscape rather than collapsing into a single shared adapter.
Figure 11: Local semantic consistency induced by PG (kNN in LoRA parameter space) . Neighbor pairs defined in parameter space consistently exhibit higher semantic similarity than random pairs in both image and text modalities, indicating that proximity in the generated parameter space aligns with semantic proximity, providing quantitative evidence of structured, condition-dependent organization in parameter space. We report distributions of (a) image embedding similarity and (b) text embedding similarity , compared against randomly sampled pairs. For each sample, we retrieve its five nearest neighbors in the projected LoRA parameter space and compute pairwise semantic similarity between neighbor pairs with editing instructions and input images.
Figure 12: Demonstration of LoRA weights obtained via direct optimization and generated by PG.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 14: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 15: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 16: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 17: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
More in this topic
Memory-Augmented Neural Networks
Memory-augmented neural networks combine learned computation with explicit or persistent memory for retrieval, long-context reasoning, and adaptation.
See all 91 papers