RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation
Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, China. · Shanghai Syslong Information Technology Co., Ltd.
Abstract
Despite the promise of Vision-Language-Action (VLA) models as generalist robotic controllers, their robustness against perceptual noise and environmental variations in out-of-distribution (OOD) tasks remains fundamentally limited by the absence of long-term memory, causal failure attribution, and dynamic intervention capability. To address this, we propose RoboHarness, a memory-augmented policy harness that upgrades frozen VLA policies for robust in-context adaptation without parameter fine-tuning. Specifically, RoboHarness operates through an online pipeline of contrastive Dual-Memory Retrieval-Augmented Generation (RAG), an attribution-driven vision-language orchestrator implemented with a multimodal large language model, and extensible Model Context Protocol (MCP) interventions, while an offline Memory Consolidation module continuously distills the execution traces into reliable priors. Experimental evaluations across three backbone models (, , and SmolVLA) on LIBERO-PRO and our proposed LIBERO-RoboHarness benchmarks demonstrate that RoboHarness achieves an average absolute success rate gain of 56.6%. This includes a significant absolute improvement of 89.1% in long-horizon task chaining. The project page and source code are available at https://github.com/LZY-1021/RoboHarness.
Figures & tables
| Tool | Discrepancy from | |
|---|---|---|
| Paint | Feature distance from historical mean. | |
| Eraser | Novel geometry vs. background model. | |
| Prompt | Instruction mismatch with successful patterns | |
| Chain | Sub-event-level alignment. | |
| Encore | Similarity to historical failure trajectories. |
| Level | Category | Task (Symbolic Form) | BASE | w/Our |
| Pos | Spat. | 2% | 63.0% | |
| 0% | 97.0% | |||
| 0% | 20.2% | |||
| 0% | 57.0% | |||
| 12% | 71.0% | |||
| Obj. | 0% | 35.0% |
| RAG Config | Orchestrator Reasoning via RAG | MCP Tool | Action Result | Task Image | SR (%) |
| No-RAG | Ambiguity in target ID due to identical textures. | Highlight bowl | 19.2 | ||
| Potential detection errors from plate shadows. | none | — | |||
| Clean background, no distractors found. | none | — | |||
| Limited-RAG | Edge proximity causes occlusion; highlight target. | Highlight bowl | 48.8 | ||
| 1. Past failures suggest high mis-ID risk. | Remove 4 dist. | ||||
| 2. Focus by removing identical distractors. |
Robustness of VLA Models
Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.
See all 221 papers