While generalist vision-language-action (VLA) models have enabled diverse open-world manipulation, real-world robotic assistance requires two critical capabilities they currently lack: personalization and reliable execution. First, executing an instruction like "make me a sandwich" requires grounding subtask decomposition in persistent user preferences rather than relying on a single generic solution. Second, reliable deployment demands explainable safety assurance that proactively prevents hazards and gracefully recovers from occasional physical failures. To address these gaps, we present CarryOn, a dual-loop harness designed to augment a frozen VLA policy. The planning loop maintains persistent user-level and scene-level context that is continually grounded against new observations and revised through interaction, enabling personalized task selection and execution grounding across conversations and changing scenes. The execution loop actively supervises policy rollouts, shielding actions from predicted hazardous contact. When execution fails, the system invokes reusable control-based recovery skills that restore task progress and are refined through reflection on earlier failures. On a real robot and in simulation, CarryOn dispatches preference-consistent subtasks across users and scenes, reduces the violation rate of an already safety-finetuned VLA from 52.6% to 6.1%, and increases recovery success from 13% to 80% when refined skills are reused.
Figures & tables
Figure 1: Serving food to a vegetarian user. (1) The user asks only, “Serve me food.” (2) CarryOn retrieves the context from memory. (3) The planner dispatches a sequence of subtasks to a frozen VLA policy: open the pink cover and, if the carrot is underneath, place it in the bowl. (4) During execution, a proactive shield steers the gripper and the carried carrot around an obstacle, and reactive recovery re-grasps the carrot to resume the task.
Figure 2: Two loops, one task context. The planning loop grounds a request in persistent context and dispatches supported subtasks. The execution loop retains the VLA for manipulation, with proactive shielding before hazardous contact and recovery after failure. A shared visual judge advances the plan on success or triggers local recovery on failure. Recovery attempts to repair the physical state so the VLA can continue the interrupted subtask. Post-episode reflection uses failure evidence to revise recovery skills, retaining successive versions in the skill library.
Figure 3: (a) Schematic of the perturbation, varying object displacement relative to the initial lifted position. (b) Policy-only response to object displacement on LIBERO-Object. Blue bars show task success, and the upper line shows the decrease from undisturbed success in percentage points.
Figure 4: Simulation settings. (a)–(d) RoboLab scenes for personalized planning and memory revision; (e)–(f) LIBERO-Safety tasks for proactive shielding; (g)–(h) controlled perturbations in LIBERO and recorded failure states in LIBERO-Recovery.
Figure 5: Real-robot settings. A Franka Research 3 with external and wrist-mounted cameras. (a) Covered-ingredient retrieval before and after lid opening; (b) sandwich preparation; (c) activity-dependent packing; (d) safe execution; (e) failure recovery.
Query
Correct / 88
Time (s)
Text
77
28.6
Image
77
34.6
Text + memory
88
7.0
Image + memory
88
7.1
Table 2: RoboLab planning without physical execution. (a) Static scenes: 88 requests per condition; latency is the median time to dispatch. (b) Relocated targets: 88 request pairs per query type. Memory corrected requires both writing the new location and removing the stale association.
TSA
FSHOA
Overall
Method
L0
L1
L2
L0
L1
L2
All
VLAct [ 40 ]
0.0 / 93.3
20.7 / 30.7
0.0 / 85.3
0.0 / 76.0
5.3 / 47.3
0.0 / 83.3
4.3 / 69.3
VLAct + VLSA
17.3 / 66.7
20.0 / 20.0
10.7 / 58.7
1.3 / 52.0
15.3 / 24.0
13.3 / 47.3
13.0 / 44.8
VLAct + Ours
20.0 / 25.3
20.0 / 24.7
20.0 / 28.0
11.3 / 26.0
10.7 / 15.3
1.3 / 37.3
13.9 / 26.1
OpenVLA-OFT [ 39 ]
0.0 / 67.3
20.7 / 38.7
6.0 / 61.3
0.0 / 78.7
14.7 / 52.0
0.0 / 70.7
6.9 / 61.4
OpenVLA-OFT + VLSA
12.7 / 47.3
19.3 / 22.7
6.7 / 45.3
0.7 / 46.7
18.0 / 46.0
16.7 / 44.0
12.3 / 42.0
Table 3: Physical safety by suite and difficulty on LIBERO-Safety. Each entry is VF ↑ / Viol. ↓ (%): violation-free task success and episodes with at least one constraint violation. Each suite–level cell contains 150 episodes.
Figure 6: Integrated food-serving runs. (a)–(d) Memory-guided lid opening and shielded transport. (e)–(h) Recovery restores a lost grasp before task completion. (i)–(l) Revealing meat triggers a scene-memory update and refusal under the vegetarian preference.
Table 10
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
User memory Mu
Scene memory Ms
Unit
One file per user; one section per category
One file; one section per place, named by the planner
Content
Facts in profile:* (identity, habits, preferences), profile:avoid:* (standing prohibitions), entity:* (owned objects, including reference_image )
- objects: and - description: lines
Example
## profile:eating_habit / - diet: vegetarian; does not eat meat
## frying_pan_under_pink_lid / - objects: carrot
Evidence
Dialogue
Observations
Writers
Summarizer; planner via update_user_memory
Planner via update_scene only
Updates
Latest value per key; retraction by delete
Section replaced; a stale entry is retracted by rewriting it
Appendix
Table 6: Memory schemas.
Parameter
Value
Controller
Operational-space pose control, kp=150 , damping ratio 1, control period 0.05 s; ωn=150≈12.25 s -1
Prediction horizon H
5 actions
Intervention threshold δ
6.67 cm (1.51 cm margin + largest displacement over 5 steps)
Appendix
Table 7: Shield parameters on LIBERO-Safety.
Figure 7: Geometric action filtering and retreat. Each example shows a geometric side view (left), the observed scene (upper right), and predicted gripper–obstacle distances (lower right); gray spheres depict current robot geometry and purple outlines depict predicted gripper geometry. (a) Actions before the first violation of the dashed distance threshold are released; (b) if the first action is blocked, the robot retreats and rechecks. Open triangles mark obstacle points nearest the predicted gripper surface for filtering, whereas the filled triangle n is nearest the current end-effector reference point and determines retreat. The two thin dashed arrows in (b), arranged head-to-tail, show the component away from n and the added upward bias; the thick solid arrow shows their resulting retreat direction.
Role
LIBERO perturbation / LIBERO-Recovery
Real robot / RoboLab
Planner
Qwen3-VL-32B-Instruct
Qwen3-VL-32B-Instruct
Visual Judge
Qwen3-VL-32B-Instruct
Qwen3-VL-32B-Instruct
Reflectors
Qwen3-VL-235B-A22B-Instruct
Qwen3-VL-235B-A22B-Instruct
VLA
StarVLA Qwen3-VL-OFT (LIBERO checkpoint)
StarVLA Qwen3-VL-OFT (task-specific fine-tunes)
Appendix
Table 8: Models per role. Language and vision-language models are called through an API; the VLA runs locally.
Module
Input
Output
Planning loop
Planner
User request, user and scene memory, camera images, results of earlier subtasks
A plan (goal, reply to the user, subtasks with endpoint, instruction, and dependencies), including memory updates
Background summarizer
Dialogue of a finished plan, current user memory
Durable user facts (Appendix B.2 )
Execution loop
Visual Judge
Current subtask, camera images, earlier observations
Keep waiting, advance, or invoke recovery; what an observation step revealed
S reflector
Reference image, scene image, selected mask and crop of each failed recovery
Enable re-grounding for the object, or skip
Appendix
Table 9: Inputs and outputs of the language-model modules.
Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China