VLA-Forget: Vision-Language-Action Unlearning for Embodied Foundation Models

Ravi Ranjan; Agoritsa Polyzou

cs.CVApr 5, 2026

VLA-Forget: Vision-Language-Action Unlearning for Embodied Foundation Models

Authors: Ravi Ranjan, Agoritsa Polyzou

Organizations: Florida International University Miami, USA

Abstract

Vision-language-action (VLA) models are emerging as embodied foundation models for robotic manipulation, but their deployment introduces a new unlearning challenge: removing unsafe, spurious, or privacy-sensitive behaviors without degrading perception, language grounding, and action control. In OpenVLA-style policies, behavior is produced through a fused visual encoder, a cross-modal projector, and a language backbone that predicts tokenized robot actions, so undesirable knowledge can be distributed across perception, alignment, and reasoning/action layers rather than confined to a single module. Consequently, partial unlearning applied only to the vision stack or only to the language backbone is often insufficient, while conventional unlearning baselines designed for standalone vision or language models may leave residual forgetting or incur unnecessary utility loss in embodied settings. We propose VLA-Forget, a hybrid unlearning framework that combines ratio-aware selective editing for perception and cross-modal specificity with layer-selective reasoning/action unlearning for utility-preserving forgetting. VLA-Forget jointly optimizes three objectives: targeted forgetting, perceptual preservation, and reasoning retention, through staged updates over the visual encoder, projector, and upper action-generating transformer blocks. Across forget-set behavior probes and retain-task evaluations, VLA-Forget improves forgetting efficacy by 10%, preserves perceptual specificity by 22%, retains reasoning and task success by 9%, and reduces post-quantization recovery by 55% relative to strong unlearning baselines.

Figures & tables

Figure 1: Motivating failure case for VLA-Forget. Given the instruction “Pick blue cup,” the policy produces a visually plausible but semantically incorrect action by grasping the red cup. After targeted unlearning, the policy suppresses this spurious instruction-to-action association and recovers instruction-consistent behavior.

Figure 2: Illustration of the unlearning objectives on a selected module. Eq. 6 suppresses the targeted behavior, Eq. 5 preserves non-targeted behavior, and Eq. 7 discourages shallow forgetting by pushing the updated policy away from the original forgotten response.

Figure 3: Overview of VLA-Forget. Given an unlearning request, we construct forget, retain, and mismatch sets, then perform staged adapter-based selective updates over the vision encoder, projector, and LLM/action layers with multi-objective PCGrad stabilization, yielding an unlearned VLA policy with preserved interface and retained task utility.

	Open X-Embodiment (OpenVLA-7B)						lerobot/pusht_image (OpenVLA-7B)
Method	FC $\uparrow$	RC $\uparrow$	FAD $\uparrow$	RAD $\downarrow$	TSR $\uparrow$	SVR $\downarrow$	FC $\uparrow$	RC $\uparrow$	FAD $\uparrow$	RAD $\downarrow$	TSR $\uparrow$	SVR $\downarrow$
SSD	78	83	0.70	0.28	68	17	82	86	0.73	0.20	55	15
SalUn	89	88	0.76	0.26	71	12	89	88	0.78	0.18	60	11
GA	93	60	0.89	0.45	40	5	94	50	0.91	0.50	22	3
NPO	90	88	0.83	0.23	74	8	92	90	0.85	0.15	65	7
VLA-Forget	93	91	0.88	0.21	78	5	95	94	0.90	0.13	69	4

Table 1: Unlearning result on OpenVLA-7B . Left: Open X-Embodiment . Right: lerobot/pusht_image . Higher is better for FC, RC, FAD, and TSR; lower is better for RAD and SVR. Mean

\pm

standard deviation over 5 random seeds, with typical variation in the range of

\pm(1

–

3)

points for rate-based metrics (FC, RC, TSR, and SVR) and

\pm(0.01

–

0.03)

for distance-based metrics (FAD and RAD). Best results are shown in bold and second-best are underlined .

	Open X-Embodiment (Pi0-FAST-Base)
Method	FC $\uparrow$	RC $\uparrow$	FAD $\uparrow$	RAD $\downarrow$	TSR $\uparrow$	SVR $\downarrow$
SSD	76	81	0.68	0.30	65	18
SalUn	87	85	0.74	0.27	68	13
GA	93	57	0.89	0.47	38	6
NPO	89	87	0.82	0.24	72	9
VLA-Forget	94	89	0.88	0.22	75	6

Table 2: Unlearning results on lerobot/pi0fast-base under the Open X-Embodiment protocol. Mean

\pm

standard deviation over 5 random seeds, with typical variation of

\pm(1

–

3)

points for FC, RC, TSR, and SVR, and

\pm(0.01

–

0.04)

for FAD and RAD. Best results are shown in bold and second-best are underlined .

Figure 4: Instruction-conditioned action behavior on pi0fast-base and Open X-Embodiments .

Method	8-bit		4-bit
Method	FC $\uparrow$	SVR $\downarrow$	FC $\uparrow$	SVR $\downarrow$
SSD	76	19	72	23
SalUn	87	8	78	10
GA	82	12	80	16
NPO	85	10	82	13
VLA-Forget	91	6	88	8

Table 3: Quantization robustness after unlearning on OpenVLA-7B with Open X-Embodiment dataset. Results report forget-side metrics under post-training 8-bit and 4-bit quantization.

Figure 5: Safety violation rate (SVR) on Open X-Embodiment for different unlearning methods on OpenVLA-7B, where lower values indicate safer post-unlearning behavior.

VLA-Forget applied to	FC ↑	RC ↑	FAD ↑	RAD ↓	TSR ↑	SVR ↓
Vision encoder only	85	82	0.80	0.27	65	12
Projector only	82	89	0.75	0.22	75	15
Language backbone only	90	88	0.85	0.23	74	8
Vision + Projector	88	86	0.84	0.24	72	10
Projector + Language	92	90	0.86	0.22	77	6
Vision + Language	93	87	0.89	0.25	70	6

Table 4: Ablation results of VLA-Forget on OpenVLA-7B on Open X-Embodiment dataset, across different component selections.

Figure 6: Forget-action comparison across component-level VLA-Forget ablations on OpenVLA-7B and Open X-Embodiment.

Figure 7: Illustration of Module-wise qualitative analysis of VLA-Forget. Single-component unlearning only partially suppresses the targeted behavior, while full VLA-Forget more cleanly removes the unwanted action and preserves safe non-target behavior.

Figure 8: Qualitative example illustration of targeted behavior suppression. Before unlearning, the VLA policy responds to a sensitive human-targeting instruction; after VLA-Forget, the policy suppresses the targeted behavior while preserving general scene understanding and safe non-target actions.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}_{\mathrm{retain}}(B_{r})+\lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}}(B_{r}\cup B_{m}),\quad\mathcal{L}_{\mathrm{forget}}(B_{f}),\quad\mathcal{L}_{\mathrm{mismatch}}(B_{f})

Appendix

Algorithm 1 VLA-Forget : Hybrid Unlearning for Vision-Language-Action Models

\mathcal{L}_{\mathrm{retain}}(B_{r})+\lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}}(B_{r}\cup B_{m}),\quad\mathcal{L}_{\mathrm{forget}}(B_{f}),\quad\mathcal{L}_{\mathrm{mismatch}}(B_{f})

Appendix

Algorithm 2 VLA-Forget : Hybrid Unlearning for Vision-Language-Action Models (continued)

Category	Setting	Value / Details
Model / data	Main model	openvla/openvla-7b ; LoRA-based unlearning in the OpenVLA
	setups	VLA models with vision / projector / reasoning unlearning
	Datasets	OXE , lerobot/pusht_image and a colored-object benchmark
	Data scale	Up to 512 instances for OpenVLA ; up to 4000 instances for main ; split 5000 / 1200 / 1200
Data split	Forget fraction	30% of prompted PushT instances
	Train / val / test	70 / 15 / 15 for PushT; separate train / val / test split

Appendix

Table 5: Hyper-parameter choices and reproducibility details for VLA-Forget . Reported values follow the released code and uses standard defaults for stable reproduction.

Group	Setting / Variant	FC $\uparrow$	RC $\uparrow$	FAD $\uparrow$	RAD $\downarrow$	TSR $\uparrow$	SVR $\downarrow$
A. Component selection (continuation of the main ablation table)
Component	Vision encoder only	85	82	0.80	0.27	65	12
Component	Projector only	82	89	0.75	0.22	75	15
Component	Language backbone only	90	88	0.85	0.23	74	8
Component	Vision + Projector	88	86	0.84	0.24	72	10
Component	Projector + Language	92	90	0.86	0.22	77	6

Appendix

Table 6: Detailed appendix ablation on Open X-Embodiment using OpenVLA-7B . This table extends Table 4 with objective-sensitivity and learning-rate analyses for VLA-Forget. Higher is better for FC, RC, FAD, and TSR; lower is better for RAD and SVR.

Figure 9: Unsafe affordance suppression illustration with retained utility. Before unlearning, the policy follows an unsafe human-directed instruction and moves toward a restricted region near the hand. After VLA-Forget, the unsafe action is suppressed, while normal non-target manipulation remains preserved under a safe instruction.

Explore similar work

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering
Jun 10, 2026Hyun Joe Jeong, Gokul Swamy, Andrea BajcsyLanguage Model-Based Control Vision-Language-Action Models
Robotics Institute Carnegie Mellon University
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University
Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?
Jun 26, 2026Guoheng Sun, Kaixi Feng, Shwai He +8Model Compression Efficient VLA Models
University of Maryland, College Park · 2Cisco Research

Jun 10, 2026J/K move · Enter open · S save

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

Hyun Joe Jeong, Gokul Swamy, Andrea Bajcsy

Robotics Institute Carnegie Mellon University

Language Model-Based Control Vision-Language-Action Models VLMs for Robotics

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models provide a natural language interface to robot control, but the mapping from language to behavior is often brittle and unintuitive: semantically similar instructions can induce drastically different behaviors, while some capabilities may not be elicitable through prompting alone. As a result, both human instructions and zero-shot language models can fail to reliably steer VLAs toward successful task execution. In this work, we propose a framework that interactively searches for language sequences that improve closed-loop VLA task performance, distills these sequences into a test-time language feedback policy (LFP), and learns an improvement head that predicts when language steering will improve performance. We conformalize this improvement head to prevent harmful steering interventions, where the LFP decreases task performance relative to the original instruction on out-of-distribution scenarios. Crucially, our approach operates on arbitrary frozen pre-trained VLAs, requiring neither access to the original training distribution nor fine-tuning of the underlying model. On seen environments, our conformalized LFP improves base VLA performance by 24.7% in simulation and 65.0% in hardware. On visual and semantic perturbations, our conformalized LFP has strong harmlessness guarantees, and produces recovery behaviors not observed with open-loop prompting.

Abstract

Explore similar work

Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?