Foundation models are transitioning from offline predictors to deployed systems expected to operate over long time horizons. In real deployments, objectives are not fixed: domains drift, user preferences evolve, and new tasks appear after the model has shipped. This elevates continual learning and instant personalization from optional features to core architectural requirements. Yet most adaptation pipelines still follow a static weight paradigm: after training (or after any adaptation step), inference executes a single parameter vector regardless of user intent, domain, or instance-specific constraints. This treats the trained or adapted model as a single point in parameter space. In heterogeneous and continually evolving regimes, distinct objectives can induce separated feasible regions over parameters, forcing any single shared update into compromise, interference, or overspecialization. As a result, continual learning and personalization are often implemented as repeated overwriting of shared weights, risking degradation of previously learned behaviors. We propose HY-WU (Weight Unleashing), a memory-first adaptation framework that shifts adaptation pressure away from overwriting a single shared parameter point. HY-WU implements functional (operator-level) memory as a neural module: a generator that synthesizes weight updates on-the-fly from the instance condition, yielding instance-specific operators without test-time optimization.
Figures & tables
Figure 1: Failure modes of static adaptation vs. conditional parameter generation. (a) Infeasible sharing. When heterogeneous objectives induce separated feasible regions in parameter space, a single shared update (Shared LoRA or SFT) is forced into compromise, instability, or dominance by high-frequency modes. (b) Over-specialization. Training a separate static adapter per domain avoids direct conflict but collapses into a narrow subspace and generalizes poorly under domain shifts. (c) Conditional generation. A generator routes each instance to an update Δθ(x)=gϕ(c(x)) , enabling inference over a family of parameter points rather than a single point.
Framework concept
Instantiation in Part I (TI2I)
Where it is tested / evidenced
Stable invariants (long-term knowledge)
Frozen diffusion backbone fθ (pretrained competence preserved during adaptation)
Static baselines show compromise or over-specialization under conflict (qualitative and quantitative), motivating preservation of the shared base
Static parameter memory (single-point adaptation)
SFT / Shared LoRA producing a fixed update Δθstatic
Hybrid condition c(x) from image and instruction features
Instance dependence: same instruction requires different operator shifts depending on content; ablations show that misalignment collapses gains
Conditional family of operators
{θ+Δθ(x)} induced by gϕ (update family over instances)
Emergent parameter-space neighborhoods align with semantic similarity; clustering and local neighborhood consistency in generated parameters
Failure mode under heterogeneity
Separated feasible regions and incompatible gradients
Controlled conflicts (e.g., restoration vs. aging) and gradient-level conflict evidence
Table 1: Concept-to-implementation mapping for Part I (TI2I). We map the memory-first framing to concrete modules, and we indicate where each concept is validated in Part I.
Figure 2: Comparison of training paradigms for hypernetwork-based parameter generation. (a) Learning to reconstruct parameters from pre-collected checkpoints via reconstruction loss. (b) Learning from pre-collected checkpoints with additional downstream task loss as auxiliary supervision. (c) Our approach: on-the-fly optimization of the parameter generator using only downstream task loss, without relying on pre-collected checkpoints.
Figure 3: Overview of the HY-WU pipeline. The framework extracts conditions from the source image and edit prompt, which are processed by a trainable Neural Network Transformer to synthesize instance-specific parameter tokens. These tokens are then detokenized into LoRA adapters and integrated into a frozen foundation model with θ1⋯θL , where θl indicates l -th layer. The entire pipeline is optimized end-to-end, where the generator is updated via backpropagation of diffusion loss.
Figure 4: Parameter tokenization and detokenization. LoRA adapters are reorganized into a unified tensor Tw . The red path reverses this process to reconstruct parameters from a tensor.
Figure 5: Architecture of the Neural Network Transformer. The left panel illustrates the overall pipeline, where parameter embeddings and extracted conditions (text/image) are processed through N transformer blocks to generate LoRA parameters T . The right panel details the internal structure of each block, featuring factorized self-attention to capture structural correlations, and cross-attention for condition injection. The final LoRA B projection is zero-initialized to ensure training stability.
NNT configuration
value
parameter count
8.11B
number of layers
24
number of heads
32
hidden dimension
4096
intermediate dimension
16384
Table 2: Configuration for Neural Network Transformer (NNT) and generated LoRA on HY-Image-3.0-Instruct (80B-A13B).
Figure 6: The GSB win-rate comparison of our HY-WU against other baseline models. Each bar reports the pairwise win rate of HY-WU against the corresponding baseline (rates sum to 100%). HY-WU demonstrates a significant performance advantage, particularly against established open-source competitors. Notably, HY-WU also achieves a clear lead over two closed-source competitors (marked with a lock icon), i.e. , Seedream-4.5-2K (55.6%) and GPT-Image-1.5 (55.5%), while remaining highly competitive against the state-of-the-art Nano-Banana series, proving its effectiveness in complex image editing tasks.
model
public
GEdit-Bench-EN
GEdit-Bench-CN
Q_SC ↑
Q_PQ ↑
Q_O ↑
Q_SC ↑
Q_PQ ↑
Q_O ↑
Seedream 4.5
×
7.920
7.427
7.587
7.735
7.442
7.445
GPT Image 1.5
7.966
7.595
7.699
7.853
7.606
7.637
Nano-Banana-Pro
7.718
7.476
7.391
7.725
7.514
7.470
Step1X-Edit-v1.2
✓
7.652
7.399
7.350
7.491
7.433
7.265
Qwen-Image-Edit-2511
7.898
7.380
7.530
7.723
7.335
7.378
Table 3: Results on GEdit-Bench (Semantic Consistency as Q_SC, Perceptual Quality as Q_PQ and Overall Performance as Q_O). Best results are shown in bold and second-best results are underlined.
model
public
add
adjust
extract
replace
remove
background
style
hybrid
action
overall ↑
Seedream 4.5
×
4.63
3.92
4.29
3.10
3.93
4.03
3.20
3.25
3.94
3.87
GPT Image 1.5
4.77
4.49
4.44
3.54
4.37
4.37
3.30
3.50
3.94
4.16
Nano Banana Pro
4.81
4.37
4.06
3.41
4.19
4.29
3.19
3.55
4.11
4.02
Step1X-Edit-v1.2
✓
4.61
4.42
2.65
3.21
3.83
4.31
2.89
3.19
3.87
3.64
Qwen-Image-Edit-2511
4.69
4.24
4.05
3.28
3.02
4.33
2.94
3.50
3.80
3.77
LongCat-Image-Edit
4.72
4.22
4.44
3.41
4.31
4.38
3.14
3.19
3.99
4.07
Table 4: Results on ImgEdit-Bench (Category-wise and Overall Performance). Best results are shown in bold and second-best results are underlined.
model
public
alignment ↑
consistency ↑
structure ↑
quality ↑
overall ↑
Seedream 4.5
×
4.55
4.02
4.24
3.89
4.18
GPT Image 1.5
4.78
3.95
4.22
3.83
4.20
Nano Banana Pro
4.58
4.02
4.23
3.89
4.18
Step1X-Edit-v1.2
✓
3.83
3.52
3.72
3.35
3.61
Qwen-Image-Edit-2511
4.54
3.86
4.07
3.71
4.04
LongCat-Image-Edit
4.47
3.83
4.03
3.66
4.00
Table 5: Results on WU-Eval. Best results are shown in bold and second-best results are underlined.
Table 6: Ablation studies about foundation model and scaling behavior. (Foundation model is marked with gray stripe.) For foundation model ablation (a) and (b), HY-WU achieve consistent improvement across different model architecture, highlighting its universality. For scaling behavior ablation (c) and (d), results demonstrate a consistent positive scaling law across both LoRA parameter and Neural Network Transformer (NNT) parameter.
Figure 7: Visual comparison on multi-image editing tasks under consistency constraints. Each task requires transferring attributes from a reference image while preserving key invariants of the base image (e.g., identity, pose, background, or lighting). Competing methods often introduce inconsistencies such as identity drift, structural distortion, or imperfect blending. In contrast, HY-WU produces edits that better preserve subject identity, pose, and scene context while faithfully transferring the target attributes.
Figure 8: Behavioral evidence of structural compromise under shared static adaptation in conflict settings. For each conflicting task pair, we compare: (1) Single LoRA trained per objective, (2) Shared LoRA trained jointly on both objectives, and (3) PG trained jointly. Green check marks indicate correct directional behavior, and red crosses indicate failure. (a) Restoration vs. Aging. Restoration enhances clarity and recovers natural color, whereas aging introduces degradation and discoloration. Single LoRA models exhibit strong directional behavior when trained separately, but they fail on the opposing task. Shared LoRA produces intermediate outputs: restored images remain slightly desaturated, and aged outputs appear insufficiently degraded. PG performs well for both objectives. (b) Deblurring vs. Blurring. Deblurring restores high-frequency details, while blurring suppresses them. Shared LoRA yields softened results in both directions. In contrast, PG maintains sharper deblurring and better blurring effects. Across both settings, single LoRA models exhibit over-specialization behavior and fail on the opposing task. Shared adaptation converges to a compromise operator. PG maintains clearer objective-specific behavior across both objectives, suggesting that condition-dependent parameters reduce the need for compromise within shared parameter space.
Figure 9: Gradient-Level Conflict Structure across Editing Tasks. (a) Mean cosine similarity among tasks. We cluster similar tasks into C0 - C5 for visualization clarity. Each node represents a mean of gradient similarity among samples from two tasks. (b) Conflict ratio matrix, defined as the proportion of negative cosine values. Lighter color indicates higher conflict ratio. (c) Representative histogram distributions illustrating different modes, including intra-task pair (left) and inter-task pairs (middle and right). These distributions demonstrate clear pattern, e.g. , strong positive bias, negative bias, and bimodality alignment. Together, these statistics reveal that heterogeneous editing objectives impose structured and often incompatible optimization directions under shared parameters.
model
conditional
win rate ↑
alignment ↑
consistency ↑
structure ↑
quality ↑
overall ↑
(a) base model
\cmark
-
4.41
3.74
3.94
3.60
3.92
(b) shared LoRA
\xmark
51.7%
4.41
3.83
3.99
3.67
3.98
(c) SFT
\xmark
51.5%
4.45
3.84
4.02
3.65
3.99
(d) average PG
\xmark
48.0%
4.40
3.74
3.94
3.62
3.92
(e) shuffle PG
\cmark (misaligned)
48.3%
4.38
3.76
3.97
3.62
3.93
(f) full PG
\cmark
56.5%
4.42
3.89
4.06
3.72
4.02
Table 7: Quantitative evaluation of instance-conditioned adaptivity. PG achieves the highest overall performance and dominates in consistency, structure, and quality, while alignment remains comparable across methods. SFT and Shared LoRA, though differing significantly in parameter capacity, yield similar overall performance. Average PG (using averaged generated LoRA across samples as a fixed parameter) collapses to the base-model level, and Shuffle PG (randomly permuting conditional inputs to PG during inference) degrades toward non-conditional baselines, indicating that gains come from input-aligned instance-specific parameter updates rather than architectural capacity.
Figure 10: Emergent semantic structure in parameter space learned by PG. Samples are clustered with K-means in multimodal semantic space (image + text embeddings) and color-coded and visualized in LoRA parameter space via t-SNE. Semantically related edits form coherent regions in parameter space, with representative examples highlighted. Note that clusters may consist of multiple sub-regions, indicating a hierarchical structure, yet remain globally distinct across editing semantics. This suggests that conditional parameter generation induces a structured parameter landscape rather than collapsing into a single shared adapter.
Figure 11: Local semantic consistency induced by PG (kNN in LoRA parameter space) . Neighbor pairs defined in parameter space consistently exhibit higher semantic similarity than random pairs in both image and text modalities, indicating that proximity in the generated parameter space aligns with semantic proximity, providing quantitative evidence of structured, condition-dependent organization in parameter space. We report distributions of (a) image embedding similarity and (b) text embedding similarity , compared against randomly sampled pairs. For each sample, we retrieve its five nearest neighbors in the projected LoRA parameter space and compute pairwise semantic similarity between neighbor pairs with editing instructions and input images.
Figure 12: Demonstration of LoRA weights obtained via direct optimization and generated by PG.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 14: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 15: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 16: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Figure 17: This visualization clearly demonstrates the ability of our HY-WU to preserve key invariants and generate high-quality and creative edited images without identity drift.
Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in continuous operation, experience accumulation, and personalization, capabilities that are central to adaptive intelligence. While continual learning research has long targeted these goals, its historical focus on in-weight learning (IWL), i.e., updating a single model's parameters to absorb new knowledge, has rendered catastrophic forgetting a persistent challenge. Our position is that combining the strengths of In-Weight Learning (IWL) and the newly emerged capabilities of In-Context Learning (ICL) through the design of modular memory is the missing piece for continual adaptation at scale. We outline a conceptual framework for modular memory-centric architectures that leverage ICL for rapid adaptation and knowledge accumulation, and IWL for stable updates to model capabilities, charting a practical roadmap toward continually learning agents.
Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov +21
Toyota Motor Europe · University of Bremen · University of Florence +17
Adapting pretrained models typically involves a trade-off between the high training costs of backpropagation and the heavy inference overhead of memory-based or in-context learning. We propose FAAST, a forward-only associative adaptation method that analytically compiles labeled examples into fast weights in a single pass. By eliminating memory or context dependence, FAAST achieves constant-time inference and decouples task adaptation from pretrained representation. Across image classification and language modeling benchmarks, FAAST matches or exceeds backprop-based adaptation while reducing adaptation time by over 90% and is competitive to memory/context-based adaptation while saving memory usage by up to 95%. These results demonstrate FAAST as a highly efficient, scalable solution for supervised task adaptation, particularly for resource-constrained models. We release the code and models at https://github.com/baoguangsheng/faast.
Guangsheng Bao, Hongbo Zhang, Han Cui +4
Zhejiang University · Westlake University · Shanghai Polytechnic University +1
Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.
Wojciech Gromski, Patryk Krukowski, Jan Miksa +2
Wrocław University of Science and Technology · IDEAS Research Institute · Jagiellonian University, Krak´ow +2