Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabilities. We argue that this view overlooks an equally important data-centric question: how should each forget example be represented before optimization? Existing methods typically keep image-conditioned failures in their original image-text form, implicitly assuming that pixels are the right route for unlearning whenever pixels appear in the input. We challenge this assumption by constructing modality-decomposed versions of VLM benchmarks, enabling controlled comparisons among image-only, text-only, and multimodal representations of the same examples. Our modality-mixing experiments show that unsafe instruction-following behavior is more effectively suppressed through textual representations than through images alone, while purely text-only unlearning can still weaken visual grounding when the target behavior depends on visual evidence. Motivated by this trade-off, we introduce ModRoute, a gradient-guided per-sample modality routing strategy that selectively converts high visual-pressure examples to text-only form while keeping the remaining examples multimodal. Notably, on VLGuard with LLaVA-1.5-7B, ModRoute has the strongest composite unlearning-utility score of 0.847 at γ=0.5, a 2.5% improvement over the best random-switching score and ≥46% better than pure multimodal or text unlearning. Overall, our results show that effective VLM unlearning depends not only on the objective, but also on the composition and per-sample modality routing of the unlearning data.
Figures & tables
Figure 1: Overview of modality routing for VLM unlearning. (a) Each multimodal forget example can be represented as image-only, text-only, or multimodal input. (b) We find that text-included mixtures often improve unlearning performance over fully multimodal or image-only formulations. (c) ModRoute selectively switches examples to text-only form while keeping others multimodal and improves the unlearning–utility trade-off.
Figure 2: Modality mixing reveals that text-heavy or text-included forget formulations provide stronger unlearning than image-only or fully multimodal formulations. (a) Unsafe R/I rate on VLGuard [ 1 ] over 68 mixes using LLaVA-1.5-7B [ 26 ] where higher R/I rate means stronger unlearning. Image-only modality is capped at 341 examples due to test dataset size. (b) Unsafe R/I rate on VLGuard [ 1 ] using InternVL3-8B [ 27 ] . (c) Forget Accuracy on COVID memes [ 14 ] over 63 mixes using LLaVA-1.5-7B [ 26 ] where lower forget accuracy means stronger unlearning. (d) Forget Accuracy on COVID memes [ 14 ] using InternVL3-8B [ 27 ] .
VLGuard
Setting
MM
Image
Text
Unsafe R/I (forget) ↑
Safe R/I (retain) ↓
Safe Similarity (retain) ↑
Text-heavy tradeoff
500
100
1000
0.941
0.283
0.572
Text–MM tradeoff
500
0
500
0.466
0.168
0.649
Text-heavy best unlearning
500
0
1000
0.975
0.351
0.548
Budget-efficient tradeoff
100
0
500
0.462
0.188
0.633
Multimodal-only
1000
0
0
0.353
0.159
0.654
Table 1: Modality-mixing results for LLaVA-1.5-7B ( 26 ) on VLGuard [ 1 ] and COVID memes [ 14 ] . This includes the best unlearning runs along with several other strong options for forget–utility trade-off and single modality runs. Further matched modality runs for VLGuard can be found in Appendix E .
Method
Switch ( γ )
Unsafe R/I
Safe Sim.
Safe R/I
U–U Score
ModRoute
0.50
0.976
0.612
0.305
0.847
Random Switch
0.75
0.961
0.561
0.314
0.826
Base
–
0.287
0.663
0.106
0.484
NPO [ 7 ]
–
0.460
0.575
0.335
0.524
RMU [ 30 ]
–
0.464
0.595
0.335
0.530
Grad. Diff. [ 31 ]
–
0.690
0.335
0.580
0.565
Table 2: Best performance of ModRoute and random switching across switch percentages γ for LLaVA-1.5-7B on VLGuard [ 1 ] compared against several unlearning baselines. The unlearning–utility tradeoff score formula and the full score breakdown across γ is in the Appendix.
Figure 3: Unlearning–utility scores on COVID memes [ 14 ] comparing ModRoute with random switching across switch fractions γ for (a) LLaVA-1.5-7B [ 26 ] and (b) InternVL3-8B [ 27 ] . The aggregate unlearning–utility score is defined in Appendix F .
Figure 4: Unlearning–Utility score across switch fractions for the VLGuard [ 1 ] dataset comparing random switch and ModRoute using InternVL3-8B [ 27 ] . Full score breakdown and formulation is in the Appendix.
Figure 5: Ablation of routing strategies on VLGuard. Performance of ModRoute and alternative gradient-based routing criteria across routing fractions γ using LLaVA-1.5-7B. Additional ablation studies are in Appendix G.1 .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Backbone
γ
ModRoute
Random
p -value
VLGuard
LLaVA
0
0.520±0.011
—
—
VLGuard
LLaVA
0.25
0.647±0.036
0.536±0.010
0.005
VLGuard
LLaVA
0.50
0.814±0.029
0.563±0.074
0.001
VLGuard
LLaVA
0.75
0.821±0.020
0.693±0.163
0.122
VLGuard
LLaVA
1
0.570±0.013
—
—
VLGuard
InternVL
0
0.610±0.019
—
—
Appendix
Table 3: U–U performance across five random seeds for ModRoute and random switching. Results are reported as mean ± standard deviation. p -values are from paired two-sided t -tests across matched seeds.
Configuration
MM
Image
Text
Unsafe R/I ↑
Safe R/I ↓
Safe Similarity ↑
MM-only
341
0
0
0.326
0.190
0.654
Image-only
0
341
0
0.355
0.215
0.633
Text-only
0
0
341
0.476
0.235
0.652
Appendix
Table 4: Matched-size modality comparison on VLGuard. All configurations use exactly 341 forget examples. Higher Unsafe R/I indicates stronger forgetting, while lower Safe R/I and higher Safe Similarity indicate better utility preservation.
Configuration
MM
Image
Text
Unsafe R/I ↑
Safe R/I ↓
Safe Similarity ↑
MM-heavy
341
50
50
0.362
0.170
0.651
Image-heavy
50
341
50
0.394
0.211
0.625
Text-heavy
50
50
341
0.572
0.315
0.579
Appendix
Table 5: Matched-budget modality mixtures on VLGuard. Each configuration contains 441 total forget examples but differs in which modality dominates the mixture.
Figure 6: Full InternVL3-8B results using ModRoute across different γ s for VLGuard [ 1 ]
Method
Switch ( γ )
Unsafe R/I
Safe Sim.
Safe R/I
U–U Score
ModRoute
0
0.432
0.612
0.272
0.527
ModRoute
0.25
0.549
0.662
0.213
0.619
ModRoute
0.50
0.976
0.612
0.305
0.847
ModRoute
0.75
0.963
0.542
0.298
0.827
ModRoute
1
0.463
0.682
0.194
0.575
Random Switch
0
0.434
0.606
0.281
0.525
Appendix
Table 6: Performance of ModRoute and random switching across switch percentages γ for LLaVA-1.5-7B on VLGuard [ 1 ] .
Figure 7: Full LLaVA-1.5-7B results using ModRoute across different γ s for COVID memes [ 14 ]
Figure 8: Full InternVL3-8B results using ModRoute across different γ s for COVID memes [ 14 ]
Figure 9: Ablation of routing strategies on VLGuard. Comparison of ModRoute with random switching and alternative saliency-based routing strategies across routing fractions γ using LLaVA-1.5-7B. ModRoute provides the strongest overall forgetting–utility trade-off, supporting the use of high-gradient visual salience for selecting examples to route.
γ
Method
Forget Acc. ↓
Test Acc. ↑
Retain Acc. ↑
U–U Score ↑
0.25
ModRoute
0.227
0.688
0.650
0.731
Low-gradient routing
0.494
0.729
0.749
0.599
Random
0.680
0.729
0.757
0.489
0.50
ModRoute
0.275
0.692
0.658
0.705
Low-gradient routing
0.487
0.712
0.701
0.590
Random
0.625
0.743
0.791
0.532
Appendix
Table 7: Routing-direction ablation on COVID Memes. Lower forget accuracy indicates stronger forgetting, while higher test and retain accuracy indicate better utility preservation.
Figure 10: Safe R/I rate and Safe Similarity rate breakdowns for the modality mixes (a) LLaVA-1.5-7B (b) InternVL3-8B on VLGuard [ 1 ]
Figure 11: Retain Accuracy and Test Accuracy breakdowns for the modality mixes (a) LLaVA-1.5-7B (b) InternVL3-8B on VLGuard [ 1 ]
Figure 12: Unsafe R/I rate across modality mixes for LLaVA-1.5-7B on VLGuard [ 1 ] on the modality-decomposed test set.
Figure 13: Example of unlearning performance and potential captioning for modality-decomposition
Figure 14: Example from COVID memes [ 14 ] with the extracted text (not harmful).
Policy
γ
Forget
Retain
Test
U–U Score
Shared
0
0.665
0.786
0.732
0.5046
Random
0.25
0.483
0.768
0.658
0.5954
ModRoute
0.25
0.301
0.738
0.548
0.6766
Random
0.50
0.461
0.751
0.610
0.5956
ModRoute
0.50
0.197
0.748
0.616
0.7546
Random
0.75
0.766
0.768
0.808
0.4556
Appendix
Table 8: Results with Qwen3-VL-4B on COVID Memes.
Policy
γ
Unsafe R/I
Safe Similarity
Safe R/I
U–U Score
Shared
0
0.651
0.814
0.011
0.7512
Random
0.25
0.929
0.790
0.014
0.9126
ModRoute
0.25
0.940
0.799
0.005
0.9228
Random
0.50
0.957
0.814
0.018
0.9334
ModRoute
0.50
0.978
0.833
0.023
0.9488
Random
0.75
0.958
0.794
0.029
0.9278
Appendix
Table 9: Results with Qwen3-VL-4B on VLGuard.
Strategy
γ
Forget Recall ↓
Retain Recall ↑
Finetuned base
–
0.90
0.93
No switching
0
0.68 ± 0.06
0.73 ± 0.02
Random
0.25
0.52 ± 0.08
0.63 ± 0.04
ModRoute
0.25
0.30 ± 0.08
0.65 ± 0.07
Random
0.50
0.44 ± 0.11
0.40 ± 0.12
ModRoute
0.50
0.46 ± 0.08
0.75 ± 0.06
Appendix
Table 10: FIUBench results. Lower forget recall indicates stronger forgetting, while higher retain recall indicates better preservation of retained knowledge. Results are reported as mean ± standard deviation.
Vision-language model unlearning studies how to remove specified concepts, identities, associations, or behaviors while preserving unrelated knowledge and capabilities.