RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation

Zhuoran Li; Zhiyang Li; Kaijun Zhou; Jinyu Gu

cs.ROMar 25, 2026

RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation

Authors: Zhuoran Li, Zhiyang Li, Kaijun Zhou, Jinyu Gu

Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, China. · Shanghai Syslong Information Technology Co., Ltd.

Abstract

Despite the promise of Vision-Language-Action (VLA) models as generalist robotic controllers, their robustness against perceptual noise and environmental variations in out-of-distribution (OOD) tasks remains fundamentally limited by the absence of long-term memory, causal failure attribution, and dynamic intervention capability. To address this, we propose RoboHarness, a memory-augmented policy harness that upgrades frozen VLA policies for robust in-context adaptation without parameter fine-tuning. Specifically, RoboHarness operates through an online pipeline of contrastive Dual-Memory Retrieval-Augmented Generation (RAG), an attribution-driven vision-language orchestrator implemented with a multimodal large language model, and extensible Model Context Protocol (MCP) interventions, while an offline Memory Consolidation module continuously distills the execution traces into reliable priors. Experimental evaluations across three backbone models ( $π_0$ , $π_{0.5}$ , and SmolVLA) on LIBERO-PRO and our proposed LIBERO-RoboHarness benchmarks demonstrate that RoboHarness achieves an average absolute success rate gain of 56.6%. This includes a significant absolute improvement of 89.1% in long-horizon task chaining. The project page and source code are available at https://github.com/LZY-1021/RoboHarness.

Figures & tables

Panel of Figure 1 — Fig. 2 : Attention maps under varied inputs. Red/blue denote high/low attention; green circles mark interventions, not heatmap values. Baselines (red frames) show diffuse focus, while MCP interventions (green frames) improve object localization and boundary delineation via visual cues, prompt simplification, clutter filtering, and subtask decomposition.

Fig. 3 : RoboHarness Framework. Given the observation and instruction, Dual-Memory RAG retrieves relevant experiences from the dual-memory bank. The vision-language orchestrator selects MCP tools, estimates parameters $\theta$ , and synthesizes an intervention chain executed by extensible MCP tools. After execution, asynchronous memory consolidation updates the bank, closing the loop for iterative refinement.

Tool	Discrepancy $\Delta$ from $\mathcal{H}_{t}$	$\theta_{i}$
Paint	Feature distance from historical mean.	$C,\,M,\,A_{mask}$
Eraser	Novel geometry vs. background model.	$D,\,D_{masks}$
Prompt	Instruction mismatch with successful patterns	$\mathcal{S}_{origin}$
Chain	Sub-event-level alignment.	$(N_{e},N_{b})_{k}$
Encore	Similarity to historical failure trajectories.	$[s_{s},s_{e}],\,N_{w}$

TABLE I : Tool-specific Parameterization Mapping.

Fig. 4 : Visual Focus. Addressing visual shift in complex manipulation environments.

Fig. 5 : Clutter Removal. Erasing irrelevant distractors to mitigate causal confusion.

Fig. 6 : Noisy Prompt. Rewriting instructions that deviate from historical successful experience patterns into concise executable commands.

Fig. 7 : Encore. Recovering via key-step re-execution.

Level	Category	Task (Symbolic Form)	$\pi_{0.5}$ BASE	w/Our
Pos	Spat.	$P.(btw(p,r),pl.)$	2%	63.0%
		$P.(center,pl.)$	0%	97.0%
		$P.(drw_{top}(cab_{w}),pl.)$	0%	20.2%
		$P.(next\_to(p),pl.)$	0%	57.0%
		$P.(next\_to(r),pl.)$	12%	71.0%
	Obj.	$P.(soup,bskt.)$	0%	35.0%

TABLE III : Performance on LIBERO-PRO. Success rates under layout (Pos) and semantic (Task) shifts.

Fig. 9 : Ablation Study. Distribution of reasoning turns required to reach the target intervention plan.

RAG Config	Orchestrator Reasoning via RAG	MCP Tool	Action Result	Task Image	SR (%)
No-RAG	Ambiguity in target ID due to identical textures.	$o_{t}^{\prime}=\text{Paint-to-Action}(o_{t},\theta_{\text{dist}})$	Highlight bowl		19.2
	Potential detection errors from plate shadows.	none	—
	Clean background, no distractors found.	none	—
Limited-RAG	Edge proximity causes occlusion; highlight target.	$o_{t}^{\prime}=\text{Paint-to-Action}(o_{t},\theta_{\text{dist}})$	Highlight bowl		48.8
	1. Past failures suggest high mis-ID risk.	$o_{t}^{\prime\prime}=\text{Eraser}(o_{t}^{\prime},\theta_{\text{conf}})$	Remove 4 dist.
	2. Focus by removing identical distractors.

TABLE IV : Ablation Study. Comparison of reasoning logic and success rates across RAG configurations.

Fig. 10 : Ablation Study. Evolution of qualitative reasoning depth scores across configurations.

Robustness of VLA Models

Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.

See all 221 papers

RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Models
May 19, 2026Jingzhou Luo, Yifan Wen, Yongjie Bai +3VLM Robustness Language-Conditioned Robot Manipulation
Sun Yat-sen University, China · Peng Cheng Laboratory · Guangdong Key Laboratory of Big Data Analysis and Processing +1
Fast-VLA: You Only Think Once for Robot Manipulation
Oct 15, 2026Kehong Gong, Guanli Hou, Xiaoxu Zheng +6Vision-Language-Action Models Closed-Loop Control of VLA Models
Huawei Technologies Co., Ltd · Huawei Central Research Institute
Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation
Feb 22, 2026Zaijing Li, Bing Hu, Rui Shao +5Efficient VLA Model Inference Memory-Augmented VLMs
Harbin Institute of Technology, Shenzhen · PengCheng Laboratory · Shenzhen Loop Area Institute +1

May 19, 2026J/K move · Enter open · S save

RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Models

Jingzhou Luo, Yifan Wen, Yongjie Bai, Xinshuai Song+2

Sun Yat-sen University, China · Peng Cheng Laboratory · Guangdong Key Laboratory of Big Data Analysis and Processing+1

VLM Robustness Language-Conditioned Robot Manipulation Vision-Language-Action Models Consistency Regularization Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have shown strong performance on embodied manipulation, yet they remain brittle under visual observation changes, paraphrased language instructions, and compounded perturbations. This limitation suggests that existing methods still rely heavily on shallow correlations in the training distribution, rather than learning stable couplings among task semantics, environment states, and action generation. Although recent efforts improve robustness through larger-scale training, post-training adaptation, or enhanced predictive modeling, they rarely enforce invariance-oriented consistency within the end-to-end policy itself. To address this issue, we propose RoVLA, a robust vision-language-action framework with multi-consistency constraints. RoVLA enforces consistency under three complementary transformations: instruction semantics, trajectory evolution, and observation perturbation. Specifically, Instructional Consistency (IC) promotes stable grounding under semantically equivalent instruction rewrites, Evolutionary Consistency (EC) preserves coherent action intent throughout the generation process, and Observational Consistency (OC) improves robustness to visual and proprioceptive perturbations by enforcing consistent predictions before and after targeted disturbances. By explicitly modeling these invariances during training, RoVLA reduces reliance on superficial correlations and improves robustness and generalization. Experiments on LIBERO-Plus, RoboTwin 2.0, and real-world manipulation tasks show that RoVLA consistently outperforms strong baseline methods and exhibits superior robustness under diverse task and observation shifts. These results demonstrate the effectiveness of multi-consistency learning for robust embodied control. Codes will be available at https://github.com/HCPLab-SYSU/RoVLA.

Abstract

Robustness of VLA Models

Explore similar work

RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Models

Fast-VLA: You Only Think Once for Robot Manipulation

Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation