Generative Embodied Multiple Behavior Control Systems for Human-like Agents

Chongyu Bao; Haokai Yang; Yuhan Wang; Zhaochong An; Kunpeng Liu; Xiaolan Liu

cs.AISep 19, 2026

Generative Embodied Multiple Behavior Control Systems for Human-like Agents

Authors: Chongyu Bao, Haokai Yang, Yuhan Wang, Zhaochong An, Kunpeng Liu, Xiaolan Liu

Organizations: University of Bristol · University of Copenhagen · Clemson University

Abstract

Building human-like agents that reproduce human behavior in realistic 3D environments has been a longstanding objective in AI. Existing human-like agent frameworks primarily focus on modeling goal-directed behavior. However, cognitive neuroscience commonly believes that human behaviors are more likely controlled by multiple control systems, including goal-directed and habitual behavior control systems. Habitual behavior has been largely overlooked though it plays a crucial role in human daily life. In this paper, we address this gap by proposing a multiple control systems setup that jointly models goal-directed and habitual behaviors. Building on this setup, we propose GEMS, in which the Habitual Controller retrieves habitual actions from habit memory in response to relevant environmental stimuli, while the Goal-directed Controller proposes goal-directed actions and estimates their values. The Arbiter dynamically governs the relative influence of each controller and selects the final action. To construct diverse human-level behavior instructions in 3D environments, we further develop a keyframe-guided motion generation module. Extensive quantitative evaluations, human and ablation studies demonstrate that human-likeness performance is substantially improved by GEMS. The efficacy of GEMS indicates the benefits of leveraging habitual behavior and multiple behavior control system coordination for believable embodied human-like agents. The code is available at \href{https://anonymous.4open.science/r/review-video-82f4/demo.mp4}{this link}.

Figures & tables

Figure 2: Overall framework of the proposed GEMS. The cognitive module takes the agent’s goal and its current state as input: the Habitual Controller retrieves habitual actions based on cues triggered by the environment; the Goal-directed Controller proposes goal-directed actions and estimates the values of both habitual and goal-directed actions based on outcomes imagined by the world model; and the Arbiter selects an action based on both controller’s output, individual bias, and psychological states. The selected action is then constructed by the motion generation module.

Figure 3: The motion generation module consists of two components. Given the action description in natural language from the cognitive module and the current scene information, the high-level planner produces an executable sub-action plan; sub-actions are aligned with keyframes where needed. Using this as a condition, the low-level generator uses T2M models to generate motion sequences.

Method	Cognitive module				Motion generation module					Action type
Method	Nat. $\uparrow$	Coh. $\uparrow$	PA $\uparrow$	Mean. $\uparrow$	SSR $\uparrow$	GSR $\uparrow$	DTG $\downarrow$	Qual. $\uparrow$	Act. $\uparrow$	Action type
X-VirtualHome ( Leng et al., 2026 )	42.26	83.81	77.52	67.86	56.67	53.33	19.95	1.27	2.86	Atomic
AGA ( Yu et al., 2024 )	56.98	53.74	69.17	59.96	51.33	26.67	19.96	1.21	1.68	Atomic
D2A ( Wang et al., 2025 )	63.74	76.90	75.43	72.02	–	–	–	–	–	–
ASVO ( Lin et al., 2026 )	42.01	46.83	27.11	38.65	–	–	–	–	–	–
ACTOR ( Liang et al., 2025 )	47.77	86.18	77.62	70.52	59.79	46.67	98.42	3.75	3.59	Motion clips

Table 1: Quantitative comparison with previous human-like agent frameworks on controlled daily-life goals with the same initialization. Mean denotes the average of Nat., Coh., and PA. Besides, Qual. and Act. have a maximum score of 5. The best results are highlighted in bold .

Figure 4: Qualitative example generated by GEMS for the goal “prepare dinner”.

Figure 5: Key human preference results. Each entry compares the win rate of the method on the vertical axis against that on the horizontal axis, with diagonal values fixed at 0.5. Full results are provided in Appendix B .

Table 2: Ablation study. (a) Ablation of the key components in the cognitive module. (b) Ablation of

F(s_{t}^{\mathrm{psy}})

in the Arbiter under different initializations of

s_{t}^{\mathrm{psy}}

. Initial states

s_{0}^{\mathrm{psy}}

include (stress, depletion, cognitive load): Normal uses

(3,3,3)

; Adverse denotes negative psychological states and it averages results across three settings:

(7,3,3)

(3,7,3)

, and

(3,3,7)

. (c) Comparison of alternative motion-generation methods. (d) Ablation of keyframe guidance in the motion generation module. (e) Comparison of different habit retrieval methods. (f) Effect of the coefficient

\rho

. Habit Ratio measures the proportion of habitual actions, while TE measures task execution efficiency.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Cognitive module				Motion generation module					Action type
Method	Nat. $\uparrow$	Coh. $\uparrow$	PA $\uparrow$	Mean. $\uparrow$	SSR $\uparrow$	GSR $\uparrow$	DTG $\downarrow$	Qual. $\uparrow$	Act. $\uparrow$	Action type
X-VirtualHome ( Leng et al., 2026 )	45.18	84.62	78.10	69.30	58.11	54.22	19.73	1.31	2.91	Atomic
AGA ( Yu et al., 2024 )	59.42	55.87	70.31	61.87	52.40	28.11	19.81	1.24	1.71	Atomic
ACTOR ( Liang et al., 2025 )	53.84	87.21	79.05	73.37	61.08	47.15	96.31	3.78	3.62	Motion clips
ACTOR † ( Liang et al., 2025 )	57.12	87.66	80.13	74.97	64.02	50.21	91.76	3.70	3.73	Motion clips
Ours	87.19	88.34	85.01	86.85	79.11	70.24	21.87	4.15	4.25	Generative

Appendix

Table 3: Quantitative comparison with previous human-like agent frameworks under long-horizon daily simulation.

Table 4: Additional ablation studies. (a) Comparison of different planning algorithms. (b) Comparison of different LLMs, with GPT-5.5 used by default. (c) Effect of

F(s_{t}^{\mathrm{psy}})

under different psychological states with

\rho=0.5

Figure 6: Qualitative comparison under the goal “find something to eat in the fridge.” We compare (a) Ours, (b) ACTOR, and (c) X-VirtualHome. While (b)–(c) are largely task-oriented, ours exhibits richer habitual behaviors and appears more human-like. Red frames indicate skeletal drift.

Figure 7: Human preference results from the subjective evaluation. The figure shows pairwise human-likeness comparisons for the Cognitive Behavior Simulation and 3D Motion Generation settings. Our full multiple controller system achieves preference scores close to real human behavior and consistently outperforms the Goal-only variant.

Figure 8: Diverse behavioral trajectories generated under the same goal. All three agents are assigned the goal of “deep clean the house” and are initialized at the same spatial location, while differing in their personas, habit memories, and psychological states. The spatial trajectories correspond to Adrian (colored green with $\rho=0.5$ ), Elena (colored blue with $\rho=0.6$ ), and Michael (colored pink with $\rho=0.4$ ). For visual clarity, only the first six actions of each agent and their corresponding key action frames are shown. The examples also illustrate cooperation and competition between habitual and goal-directed control.

Habit initialization	Nat. $\uparrow$	Coh. $\uparrow$	PA $\uparrow$	Mean $\uparrow$
LLM-based	87.5	88.2	85.8	87.2
Human-annotated	86.3	89.1	87.7	87.7

Appendix

Table 5: Comparison between LLM-based and human-annotated habit initialization for Alice. Mean denotes the average of Nat., Coh., and PA.

	Ours			Sim. $\times$ HS			Sim.-only
Agent	Val.	$\kappa^{*}$	Test	Val.	$\kappa^{*}$	Test	Val.	$\kappa^{*}$	Test
Michael	82.8	0.55	81.2	71.0	0.80	71.6	73.0	0.85	72.1
Adrian	85.4	0.55	82.1	68.7	0.70	66.9	71.1	0.90	70.4
Elena	93.1	0.45	90.7	77.1	0.75	75.8	78.6	0.85	77.3
Average	87.1		84.7	72.3		71.4	74.2		73.3

Appendix

Table 6: Validation-selected retrieval thresholds and held-out test performance. For each agent and each retrieval strategy,

\kappa

is selected on the validation set and fixed for test evaluation.

Factor	Habit:Goal (Normal $\rightarrow$ Elevated)	$\beta_{k}$
Stress	$38{:}62\rightarrow 66{:}34$	$0.45$
Depletion	$16{:}84\rightarrow 48{:}52$	$0.38$
Cognitive load	$55{:}45\rightarrow 73{:}27$	$0.40$

Appendix

Table 7: Empirical effect-size references for parameterizing the psychological state gate. Habit:Goal ratios are behavioral or computational proxies.

Figure 9: The framework of the context-based world model

Panel of Figure 10 — Figure 10: Ablation task: Sitting on the sofa while watching television

Panel of Figure 11 — Figure 11: Ablation task: standing up to cheer

Panel of Figure 12 — Figure 12: Ablation task: Sitting in front of the kitchen table

Panel of Figure 13 — Figure 13: Ablation task: Performing jumping jacks in a suitable open area in the living room

Panel of Figure 14 — Figure 14: Ablation task: Walking toward a toilet while holding the abdomen in pain

Panel of Figure 15 — Figure 15: Ablation task: Sitting on the toilet

Explore similar work

ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control
Apr 30, 2026Yanghao Zhou, Jingyu Ma, Yibo Peng +3Full-Body Humanoid Control Human Motion Generation
Beijing Academy of Artificial Intelligence (BAAI)
MIND: Multi-Scale Intent Diffusion for Text-Driven Physics-Based Humanoid Control
May 25, 2026Bin Li, Ruichi Zhang, Han Liang +4Full-Body Humanoid Control Human Motion Generation
ShanghaiTech University, China · University of Pennsylvania, USA · Bytedance Seed, China +3
EMoG: Emotion-Modulated Gait Generation for Expressive Humanoid Locomotion
Sep 13, 2026Yi Lu, Tianhao Jiang, Honglong Tian +6Gait Dynamics Full-Body Humanoid Control
School of Electronic Science and Engineering, Nanjing University, Nanjing 210023, China. · School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China. · School of Artiﬁcial Intelligence, Nanjing University, Nanjing 210023, China. +1

Apr 30, 2026, cs.ROJ/K move · Enter open · S save

ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control

Yanghao Zhou, Jingyu Ma, Yibo Peng, Zhenguo Sun+2

Beijing Academy of Artificial Intelligence (BAAI)

Full-Body Humanoid Control Human Motion Generation Camera-Conditioned Video Generation Generative Video Models Egocentric Video Motion Control+1

Open paper AskPDF arXiv

Humanoid control systems have made significant progress in recent years, yet modeling fluent interaction-rich behavior between a robot, its surrounding environment, and task-relevant objects remains a fundamental challenge. This difficulty arises from the need to jointly capture spatial context, temporal dynamics, robot actions, and task intent at scale, which is a poor match to conventional supervision. We propose ExoActor, a novel framework that leverages the generalization capabilities of large-scale video generation models to address this problem. The key insight in ExoActor is to use third-person video generation as a unified interface for modeling interaction dynamics. Given a task instruction and scene context, ExoActor synthesizes plausible execution processes that implicitly encode coordinated interactions between robot, environment, and objects. Such video output is then transformed into executable humanoid behaviors through a pipeline that estimates human motion and executes it via a general motion controller, yielding a task-conditioned behavior sequence. To validate the proposed framework, we implement it as an end-to-end system and demonstrate its generalization to new scenarios without additional real-world data collection. Furthermore, we conclude by discussing limitations of the current implementation and outlining promising directions for future research, illustrating how ExoActor provides a scalable approach to modeling interaction-rich humanoid behaviors, potentially opening a new avenue for generative models to advance general-purpose humanoid intelligence.

Abstract

Explore similar work

ExoActor: Exocentric Video Generation as Generalizable Interactive Humanoid Control

MIND: Multi-Scale Intent Diffusion for Text-Driven Physics-Based Humanoid Control

EMoG: Emotion-Modulated Gait Generation for Expressive Humanoid Locomotion