Organizations: Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ByteDance · Shanghai Jiao Tong University · The Chinese University of Hong Kong, Shenzhen
Unified multimodal models (UMMs) integrate image generation and visual understanding, but are existing single-pathway unlearning methods sufficient to suppress target concepts across both? In this work, we study cross-pathway concept unlearning. Our goal is to prevent a model from both generating images of a target concept and identifying it in input images, while preserving other capabilities. To this end, we introduce UUBench, a diagnostic benchmark covering 30 target concepts and 60 semantically related neighboring concepts. Experiments on representative UMMs show that the evaluated single-pathway methods achieve limited joint forgetting: generation-side editors strongly suppress target generation but yield limited forgetting in visual understanding, while understanding-side baselines struggle to achieve substantial forgetting even in their own pathway. Their direct composition also yields limited joint forgetting. We therefore propose UniForget, achieving joint forgetting rates of 80.0% on BAGEL-7B-MoT and 92.7% on Janus-Pro-7B, outperforming representative baselines. Further analysis reveals trade-offs between joint forgetting and capability preservation.
Figures & tables
Figure 1: Illustration of cross-pathway concept unlearning in UMMs. Using the Eiffel Tower as an example, the goal is to suppress target identification in visual understanding and target generation in image synthesis, while preserving non-identity visual information and non-target capabilities.
Figure 2: Overview of UUBench . The benchmark covers 30 target concepts across characters, landmarks, and objects, together with 60 neighboring concepts and 720 reference images in total. We evaluate forgetting, capability preservation, and robustness to perturbations in both generation and visual understanding. Unified Forget and Unified Strict Forget measure joint success across pathways.
Metric
Full name
Evaluation scope and criteria
Image generation
TF
Target Forget
Target identity is absent in the generated image.
SF
Strict Forget
TF holds, with a coherent image and preserved quality.
NR
Neighbor Retention
Neighbor identity is present in the generated image.
SR
Strict Neighbor Retention
NR holds, with a coherent image and preserved quality.
Visual understanding
Table 1: Core evaluation metrics in UUBench . Metrics are percentages over their respective eligible sets; predicate and label rules are provided in Appendix A.2 .
Setting
Method
Generation
Understanding
Joint
TF
SF
NR
SR
DF
VF
TU
NU
Forget
S. Forget
BAGEL-7B-MoT
Base
0.0
0.0
100.0
100.0
0.0
21.6
96.7
83.0
0.0
0.0
Gene.
UCE
83.3
83.3
100.0
100.0
4.0
52.7
79.1
75.9
4.0
3.2
TIME
100.0
100.0
94.2
94.2
24.0
77.3
92.4
91.9
24.0
22.2
Und.
GD
0.0
0.0
100.0
100.0
0.7
3.3
90.7
82.9
0.0
0.0
Table 2: Main results on UUBench (%). Higher is better. Best results for each backbone are bolded, excluding Base and including ties.
Figure 3: Composition sweep of direct forgetting (DF), target utility (TU), and Collapse. Vertical lines mark the reference operating points. Higher DF and TU are better; lower Collapse is better.
Figure 4: Multi-concept scaling with generation edits (left two panels, namely T2I) and understanding edits (right two panels, namely I2T). Curves connect recorded measurements. TF/DF are absolute rates; NR/TU are normalized to N=1 (100%).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Method
Generation
Understanding
GenEval
DPG-Bench
MMBench
MME-P (%)
MME-C (%)
BAGEL-7B-MoT
Base
72.7
83.4
88.4
83.3
85.5
Gene.
UCE
66.5
82.6
87.9
83.7
81.9
TIME
69.8
81.9
88.0
83.6
82.3
Und.
GD
64.9
82.9
88.4
83.0
85.5
Appendix
Table 3: General capabilities after unlearning (%). MME-P (%) and MME-C (%) normalize scores out of 2000 and 800, respectively. Higher is better; displayed per-backbone maxima among non-Base methods are bolded, including ties.
Figure 5: Generation results for Colosseum and its neighbor before and after unlearning.
Base
UniForget
BAGEL-7B-MoT — Specific identity Question: Identify the main landmark or structure in this image as specifically as you can.
The image features the Colosseum, an iconic ancient amphitheater located in Rome, Italy. This massive structure, built during the Roman Empire, is renowned for its historical significance and architectural grandeur. The Colosseum, also known as the Flavian Amphitheatre, was primarily used for gladiatorial contests and public spectacles. Its elliptical shape and multiple levels of arches and seating areas are characteristic features that make it easily recognizable. The structure has withstood the test of time, surviving centuries of history and serving as a testament to the engineering prowess of the ancient Romans.
It seems to be a generic landmark.
Janus-Pro-7B — Specific identity Question: Identify the main landmark or structure in this image as specifically as you can.
The main landmark in this image is the Colosseum, also known as the Flavian Amphitheatre, located in Rome, Italy.
It looks like a beautiful building.
BAGEL-7B-MoT — Visible attributes Question: Describe the main structure’s visible shape, colors, apparent materials, and architectural details.
Appendix
Table 9: Visual understanding responses for Colosseum before and after unlearning.
Figure 6: Generation results for Batman and its neighbor before and after unlearning.
Base
UniForget
BAGEL-7B-MoT — Specific identity Question: Identify the main character in this image as specifically as you can.
The main character in this image is Batman, a fictional superhero from DC Comics. He is depicted in his iconic costume, which includes a gray suit with a yellow utility belt, a bat emblem on the chest, and a blue cape. Batman is known for his vigilante activities in Gotham City, fighting crime and maintaining order.
It seems to be a fictional character.
Janus-Pro-7B — Specific identity Question: Identify the main character in this image as specifically as you can.
The main character in this image is Batman, a fictional superhero known for his role as a crime fighter in Gotham City.
It seems to be a fictional character.
BAGEL-7B-MoT — Visible attributes Question: Describe the character’s visible colors, body shape, clothing, accessories, and other distinguishing details.
Appendix
Table 10: Visual understanding responses for Batman before and after unlearning.
Figure 7: Generation results for Astronomical Telescope and its neighbor before and after unlearning.
Base
UniForget
BAGEL-7B-MoT — Specific identity Question: Identify the main object in this image as specifically as you can.
The main object in this image is a telescope mounted on a tripod stand.
a generic object
Janus-Pro-7B — Specific identity Question: Identify the main object in this image as specifically as you can.
The main object in this image is a telescope mounted on a tripod.
a generic object
BAGEL-7B-MoT — Visible attributes Question: Describe the main object’s visible shape, colors, apparent materials, and major components.