From Proposal to Execution: Modeling Interaction Semantics in Shared Autonomy

Shashank Shekhar; Laurent Jeanpierre; Abdel-Illah Mouaddib

cs.AIOct 18, 2026

From Proposal to Execution: Modeling Interaction Semantics in Shared Autonomy

Authors: Shashank Shekhar, Laurent Jeanpierre, Abdel-Illah Mouaddib

Organizations: Université de Caen Normandie, GREYC-CNRS UMR 6072, Caen, France

Abstract

In many human-AI systems, candidate actions are not executed directly but are first evaluated by another decision-making entity under a potentially different objective or decision model. Standard decision-theoretic frameworks do not require this interaction to be represented explicitly, as they define transition dynamics directly over state-action pairs, allowing evaluation to be absorbed within the transition model. We introduce a semantics-aware decision framework in which a candidate action is proposed, evaluated under an interaction context, and then resolved into an executed outcome. We formalize this through a joint action language and an interpretation function that maps each joint action and context to a semantic outcome. Transition and reward dynamics are induced through these semantic outcomes, thereby introducing an explicit representational layer for interaction semantics. Semantic outcomes provide explicit units for organizing interaction principles, which can be instantiated through application-specific evaluation mechanisms and reused across distinct interaction instances. This provides a structured interface for specifying context-dependent evaluation separately from the resulting execution dynamics, without requiring changes to the underlying planning algorithm. We instantiate the framework in shared human-AI autonomy, capturing interaction patterns such as advisory and authorization, and illustrating how interaction context influences decision making. The resulting formulation remains compatible with standard Markov decision process (MDP) planning.

Figures & tables

$att$	$\rho(s,a_{c})$	$V^{*}(s)$	$best\_act$	$\Delta Q$	$Q(s,noop)$
very low	0.504	1.876	right	1.188	0.688
low	0.509	3.546	right	1.355	2.192
medium	0.519	5.059	right	1.506	3.553
high	0.536	6.231	right	1.623	4.608
very high	0.562	7.313	right	1.731	5.582

Table 3: Start state

(0,0)

. The reference action (

a_{c}

= right ) remains the optimal advice across all attention levels. Advisory consistently outperforms silence despite only modest growth in acceptance.

$att$	$\rho(s,a_{c})$	$V^{*}(s)$	$best\_act$	$\Delta Q$	$Q(s,noop)$
very low	0.559	5.357	down	1.536	3.821
low	0.717	8.161	down	1.816	6.345
medium	0.855	10.112	down	2.011	8.101
high	0.922	11.586	down	2.159	9.427
very high	0.916	12.773	down	2.277	10.496

Table 4: State above bridge B1

(2,2)

. The reference action (

a_{c}

= down ) is consistently optimal, while acceptance increases strongly with attention, reflecting increasing agreement between the robot’s advice and the human’s evaluation.

$att$	$\rho(s,a_{c})$	$V^{*}(s)$	$best\_act$	$\Delta Q$	$Q(s,noop)$
very low	0.469	3.468	left	0.979	2.121
low	0.567	5.409	left	1.261	3.868
medium	0.701	7.087	left	1.684	5.378
high	0.819	10.353	down	2.035	8.317
very high	0.856	14.043	down	2.404	11.638

Table 5: State above bridge B2

(4,2)

. The reference action (

a_{c}

= down ). A phase transition occurs: the optimal advice changes from left to down as attention increases, illustrating context-dependent evaluation and execution.

$att$	$\rho(s,a_{c})$	$V^{*}(s)$	$best\_act$	$\Delta Q$	$Q(s,noop)$
very low	0.540	11.541	down	2.154	9.387
low	0.587	13.980	down	2.398	11.582
medium	0.637	16.665	down	2.667	13.999
high	0.677	19.573	down	2.957	16.616
very high	0.687	22.643	down	3.264	19.379

Table 6: State in the B2 corridor

(4,4)

. The reference action (

a_{c}

= down ). The optimal advice remains down , while both the advisory advantage over silence and the resulting state value increase with attention.

Human-AI Decision Making

Human-AI decision making studies how people and AI systems share, support, or delegate decisions across tasks and institutions.

See all 175 papers

Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning
Aug 23, 2026Juntao Cheng, Jingkai Wang, Yijun Shen +2World Model-Based Planning Action-Conditioned World Models
Beijing Academy of Artificial Intelligence (BAAI) · Shanghai Jiao Tong University
When Should an AI Act? A Human-Centered Model of Scene, Context, and Behavior for Agentic AI Design
Feb 26, 2026Soyoung Jung, Daehoo Yoon, Sung Gyu Koh +3Agentic AI Human-AI Interaction
Taejae Human-Centric AI Center (HCAC), Taejae University, Seoul, Republic of Korea · LG Electronics, Seoul, Republic of Korea
Rationalize: Shared Semantic Reasoning for Human-AI Alignment
May 28, 2026Aritra Dasgupta, Naga Datha Saikiran Battula, Avina Nakarmi +3AI Alignment Human-AI Interaction
New Jersey Institute of Technology Newark, USA

Aug 23, 2026J/K move · Enter open · S save

Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning

Juntao Cheng, Jingkai Wang, Yijun Shen, Xiansheng Chen+1

Beijing Academy of Artificial Intelligence (BAAI) · Shanghai Jiao Tong University

World Model-Based Planning Action-Conditioned World Models Object-Centric World Models Robotic Manipulation

Open paper AskPDF arXiv

World models for physical interaction are typically trained to predict future observations or latent features; however, a planning-oriented model must answer a fundamentally different question: whether a candidate action produces a task consistent future while preserving essential relations. Monolithic state representations obscure the underlying entities, while standard instance-level object slots merely identify what is present without specifying what role each entity plays in the task context. To bridge this gap, we present the Semantically Rich World Model (SR-WM), a task-conditioned world model structured around five functional roles: gripper, target, goal, relation, and phase. Within SR-WM, a visual entity encoder extracts soft entity hypotheses from pretrained patch features, allowing segmentation masks to serve as optional proposal priors without mandating them as required state representations or inference inputs. A role binder subsequently maps these hypotheses to task-specific roles, while an action conditioned dynamics model predicts role transitions alongside fine-grained semantics, including grasp/contact, predicate establishment, relation preservation, fixture state, and phase change. Crucially, this unified role state grounds downstream multi-candidate action generation, stage-aware reranking, and violation-aware suffix resampling. Our comprehensive evaluation protocol spans all four LIBERO simulation suites, cross-suite transfer, perception diagnostics, and action sensitivity analysis. Ultimately, this formulation transforms object-centric prediction into a semantic interface linking visual dynamics with planning-oriented decision making

Abstract

Human-AI Decision Making

Explore similar work

Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning

When Should an AI Act? A Human-Centered Model of Scene, Context, and Behavior for Agentic AI Design

Rationalize: Shared Semantic Reasoning for Human-AI Alignment