cs.ROSep 26, 2026

Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter

Authors: Mengxue Fu, Ethan Xu, Sam Iyer-Singh, Yinlong Dai, Michael Hagenow, Dylan P. Losey

Organizations: Collaborative Robotics Lab (Collab), Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061 · RT2 Lab, Dept. of Computer Science, UW-Madison

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A common approach for shared autonomy blends human inputs with autonomous assistance based on the human's likely goal. However, most existing approaches assume that a static set of possible goals is known a priori, which limits the use of such methods in unstructured assistive settings. We instead investigate how to enable shared autonomy with open-ended and dynamically changing goals. We formulate goal-oriented shared autonomy as a particle filter in which particles represent candidate human goals. Unlike conventional approaches with a fixed goal set, our transition model dynamically proposes new candidate goals as the interaction evolves, and human actions update the belief over these goals in real time. We instantiate this framework with foundation models (e.g., vision grounding and large language models) that propose context-relevant semantic goals, generate goal-conditioned assistance from low-level skill primitives, and refine those skills from human corrections. We assess our approach through a user study where 12 participants perform a variety of tabletop manipulation tasks with our method and state-of-the-art shared autonomy baselines. The results show that our particle filter-based approach reduces the amount of time users spend teleoperating the system and improves user satisfaction. User study videos: https://youtu.be/Ii26XuRqm9c

Figures & tables

Figure 1: We frame goal-oriented shared autonomy as a particle filter . Given the semantic state, an LLM iteratively proposes a set of goals (i.e., particles). The particle weights are then updated based on the human’s teleoperation inputs. By dynamically changing the goal set, and then generating code to assist for those goals, we enable more open-ended shared autonomy.
Figure 2: Our proposed approach for goal-oriented shared autonomy. (A) Given an image of the scene, we extract objects and fill in a natural language state description. This state description is then provided to an LLM, which functions as the transition model and proposes semantic goals Gt\mathcal{G}_{t} . The number of proposed goals is adapted based on coverage. (B) The weight of each goal in Gt\mathcal{G}_{t} is updated using Bayesian inference based on our observation model , the cosine alignment between the goal-specific assistance and the human’s joystick inputs. (C) The robot provides assistance for the most likely goal. To find a policy that assists for that goal, an LLM dynamically generates skills (i.e., code blocks) which are executed by the robot. When the human intervenes during shared autonomy and corrects the robot’s behavior, we store that corrected trajectory and use it to condition future skill generation.
Figure 3: User study tasks and scenes. In all scenes the goals were open-ended — the robot was never told what tasks the human might want to perform. Scenes 1 and 2 were in-distribution : SAPS was fine-tuned on the tasks, and CASPER and Ours were given a library of goal-relevant skills. Scene 3 was out-of-distribution , with no fine-tuning or skill library. Users completed Scene 3 three times to assess how Ours generated and corrected assistance.
Figure 4: (Left) Objective results across in-distribution Scenes 1 and 2. (Right) Objective results for out-of-distribution Scene 3. For Time , the solid bar shows the total time to complete a task, and the shaded region (with the listed number) is the amount of time the human spent teleoperating the robot. To examine the effect of assistance type, we conducted repeated measures ANOVAs followed by post hoc comparisons using Bonferroni-corrected t-tests. This same analysis procedure was followed in Figure 5 . Error bars show standard error, and an ∗* denotes statistical significance ( p<.05p<.05 ).
Figure 5: Subjective results. Error bars show standard error, and an ∗* denotes statistical significance ( p<.05p<.05 ). When asked to select their preferred method, 11/12 users chose Ours , and the remaining user selected Teleop .

Explore similar work

CardsList

  1. Legible Shared Autonomy: Implicit Communication of Robot Belief through Motion

    Jun 29, 2026Jinwei Liu, Pengfei Li, Shaofeng Chen +2Robotic ManipulationShared Autonomy

    Department of Automation, University of Science and Technology of China, Hefei 230027, China · Department of Automation, Hefei University of Technology, Hefei 230009, China · National Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China

  2. What Is My Robot Thinking? Design Considerations for Transparent and Trustworthy Shared Autonomy

    Jun 5, 2026Atharv Belsare, Zohre Karimi, Connor Mattson +2Shared AutonomyAssistive Robotics

    Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA. · Robotics Center, University of Utah, Salt Lake City, UT, USA.

  3. Proximal State Nudging: Reducing Skill Atrophy from AI Assistance

    May 19, 2026Megha Srivastava, Jonathan Ouyang, Eric Zhou +6Human-in-the-Loop AIShared Autonomy

    Stanford University, Stanford, CA, USA. · University of California Los Angeles, Los Angeles, CA, USA. · Toyota Research Institute, Los Altos, CA, USA.