CarryOn: A Dual-Loop Harness for Personalized and Reliable Robot Assistance
Organizations: HKUST · CUHK · SmartMore
Abstract
While generalist vision-language-action (VLA) models have enabled diverse open-world manipulation, real-world robotic assistance requires two critical capabilities they currently lack: personalization and reliable execution. First, executing an instruction like "make me a sandwich" requires grounding subtask decomposition in persistent user preferences rather than relying on a single generic solution. Second, reliable deployment demands explainable safety assurance that proactively prevents hazards and gracefully recovers from occasional physical failures. To address these gaps, we present CarryOn, a dual-loop harness designed to augment a frozen VLA policy. The planning loop maintains persistent user-level and scene-level context that is continually grounded against new observations and revised through interaction, enabling personalized task selection and execution grounding across conversations and changing scenes. The execution loop actively supervises policy rollouts, shielding actions from predicted hazardous contact. When execution fails, the system invokes reusable control-based recovery skills that restore task progress and are refined through reflection on earlier failures. On a real robot and in simulation, CarryOn dispatches preference-consistent subtasks across users and scenes, reduces the violation rate of an already safety-finetuned VLA from 52.6% to 6.1%, and increases recovery success from 13% to 80% when refined skills are reused.
Figures & tables
| Study | Condition | Result |
| Sandwich Preparation | Textual | 24 / 24 |
| Memory | 24 / 24 | |
| Preference update | 20 / 24 | |
| Activity-dependent Packing | Memory | 16 / 16 |
| No-habit | 16 / 20 | |
| Isolation | 16 / 18 |
| Query | Correct / 88 | Time (s) |
| Text | 77 | 28.6 |
| Image | 77 | 34.6 |
| Text + memory | 88 | 7.0 |
| Image + memory | 88 | 7.1 |
| TSA | FSHOA | Overall | |||||
| Method | L0 | L1 | L2 | L0 | L1 | L2 | All |
| VLAct [ 40 ] | 0.0 / 93.3 | 20.7 / 30.7 | 0.0 / 85.3 | 0.0 / 76.0 | 5.3 / 47.3 | 0.0 / 83.3 | 4.3 / 69.3 |
| VLAct + VLSA | 17.3 / 66.7 | 20.0 / 20.0 | 10.7 / 58.7 | 1.3 / 52.0 | 15.3 / 24.0 | 13.3 / 47.3 | 13.0 / 44.8 |
| VLAct + Ours | 20.0 / 25.3 | 20.0 / 24.7 | 20.0 / 28.0 | 11.3 / 26.0 | 10.7 / 15.3 | 1.3 / 37.3 | 13.9 / 26.1 |
| OpenVLA-OFT [ 39 ] | 0.0 / 67.3 | 20.7 / 38.7 | 6.0 / 61.3 | 0.0 / 78.7 | 14.7 / 52.0 | 0.0 / 70.7 | 6.9 / 61.4 |
| OpenVLA-OFT + VLSA | 12.7 / 47.3 | 19.3 / 22.7 | 6.7 / 45.3 | 0.7 / 46.7 | 18.0 / 46.0 | 16.7 / 44.0 | 12.3 / 42.0 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| User memory | Scene memory | |
| Unit | One file per user; one section per category | One file; one section per place, named by the planner |
| Content | Facts in profile:* (identity, habits, preferences), profile:avoid:* (standing prohibitions), entity:* (owned objects, including reference_image ) | - objects: and - description: lines |
| Example | ## profile:eating_habit / - diet: vegetarian; does not eat meat | ## frying_pan_under_pink_lid / - objects: carrot |
| Evidence | Dialogue | Observations |
| Writers | Summarizer; planner via update_user_memory | Planner via update_scene only |
| Updates | Latest value per key; retraction by delete | Section replaced; a stale entry is retracted by rewriting it |
| Parameter | Value |
| Controller | Operational-space pose control, , damping ratio 1, control period 0.05 s; s -1 |
| Prediction horizon | 5 actions |
| Intervention threshold | 6.67 cm (1.51 cm margin largest displacement over 5 steps) |
| Role | LIBERO perturbation / LIBERO-Recovery | Real robot / RoboLab |
| Planner | Qwen3-VL-32B-Instruct | Qwen3-VL-32B-Instruct |
| Visual Judge | Qwen3-VL-32B-Instruct | Qwen3-VL-32B-Instruct |
| Reflectors | Qwen3-VL-235B-A22B-Instruct | Qwen3-VL-235B-A22B-Instruct |
| VLA | StarVLA Qwen3-VL-OFT (LIBERO checkpoint) | StarVLA Qwen3-VL-OFT (task-specific fine-tunes) |
| Module | Input | Output |
| Planning loop | ||
| Planner | User request, user and scene memory, camera images, results of earlier subtasks | A plan (goal, reply to the user, subtasks with endpoint, instruction, and dependencies), including memory updates |
| Background summarizer | Dialogue of a finished plan, current user memory | Durable user facts (Appendix B.2 ) |
| Execution loop | ||
| Visual Judge | Current subtask, camera images, earlier observations | Keep waiting, advance, or invoke recovery; what an observation step revealed |
| S reflector | Reference image, scene image, selected mask and crop of each failed recovery | Enable re-grounding for the object, or skip |
Assistive Robotics
Assistive robotics develops robots and wearable devices that support mobility, daily activities, rehabilitation, and care.
See all 207 papers