Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination

Rakshit Trivedi; Kartik Sharma; David C Parkes

cs.AIFeb 24, 2026

Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination

Authors: Rakshit Trivedi, Kartik Sharma, David C Parkes

Organizations: Massachusetts Institute of Technology · Georgia Institute of Technology · Harvard University

Abstract

Effective human-AI coordination requires artificial agents capable of exhibiting and responding to human-like behaviors while adapting to changing contexts. Imitation learning has emerged as one of the prominent approaches to build such agents by training them to mimic human-demonstrated behaviors. However, current methods struggle to capture the inherent diversity and non-Markovian nature of human behavior and lack the ability to steer behavior at inference time. Drawing inspiration from the theory of human cognitive processes, where inner speech guides action selection before execution, we propose MIMIC (Modeling Inner Motivations for Imitation and Control), a framework that uses language as an internal representation of behavioral intent. MIMIC employs the novel use of vision-language models as linguistic scaffolding to train a conditional variational autoencoder capable of generating inner speech from observations. A diffusion-based behavior cloning policy then selects actions conditioned on current observations and the generated inner speech. MIMIC enables fine-grained steering of behavior at inference time by conditioning the agent on behavior-specific speech. Experiments across robotic manipulation tasks and human-AI collaboration games demonstrate that MIMIC significantly enhances both behavior diversity and fidelity to human demonstrations while enabling nuanced behavioral steering without training on additional demonstrations. We open source our code and provide pre-trained MIMIC agents and qualitative demos at: https://mimic-research.github.io.

Figures & tables

Panel of Figure 1 — Figure 1: Paradigm comparison. (a) direct state-to-action mapping. (b) inner speech $m_{t}$ mediates between perception and action. Extended discussion available in Appendix B.1 .

Figure 2: Overview of MIMIC : Agent inner speech is scaffolded by using a pre-trained VLM to discriminate different behaviors in human demonstrations. Next, we train a DDPM-T (diffusion policy with transformer architecture) behavior cloner conditioned on this inner speech and a VAE-based inner speech generator conditioned on the history of states. During simulation, the inner speech is periodically generated to influence the behavior cloner given the actions it generated in the past.

Environment	Model	Success rate $\uparrow$	Distance $\downarrow$	Entropy $\uparrow$	State-wass $\downarrow$	Time-wass $\downarrow$
Aligning	BC	0.6645	0.1105	0.4743	0.6961	59.034
	MIMIC-S	0.8021	0.0664	0.4184	0.0459	50.569
	MIMIC-E	0.7229	0.0847	0.6148	0.0492	45.397
Aligning-Vision	BC	0.1833	0.1875	0.0895	-	-
	MIMIC-S	0.2229	0.1885	0.0849	-	-
	MIMIC-E	0.2083	0.1849	0.1473	-	-

Table 1: Comparison of MIMIC against BC with the DDPM-T architecture on the D3IL benchmark. ‘-wass’ denotes Wasserstein Distance metrics for the non-vision environments (infeasible in case of vision) where state Wasserstein distance is calculated using 5 random rollouts.

Panel of Figure 4 — Figure 4: Effect on Success Rate and Entropy performance of MIMIC on changing various components on the Aligning dataset: (a) removing inner speech during simulation, (b) changing the embedding model, (c-d) using different VLMs for training and scaffolding.

Panel of Figure 5 — Figure 5: Conditional behavior generation. Figures in the right panel show generated behaviors for the specified condition as inner speech with no subsequent updates.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6

Approach	Behavioral Diversity	Designer Control	Speech Type	Language Grounding	Language Annotations Required	Latent Space	Conditioning Type
Behavior Transformer [ 39 ]	✓ (discrete modes)	$\times$	N/A	$\times$	No	$\times$	Unconditional
Diffusion BC [ 33 ]	✓ (continuous)	$\times$	N/A	$\times$	No	✓ (implicit)	Unconditional
BESO [ 36 ]	✓ (partial) 1	✓ (goals only)	External	✓ (goals)	Yes (goals)	✓	Goal-conditioned
Thought Cloning [ 18 ]	$\times$ 2	$\times$	External 3	✓ (thoughts)	Yes (per-step)	$\times$	Unconditional
MIMIC (Ours)	✓ (stochastic)	✓ (general)	Internal	✓ (inner speech)	No 4	✓ (CVAE)	General linguistic

Appendix

Table 4: Comparative analysis of IL approaches across key dimensions of behavioral modeling and control. Legend: ✓ = Full support;

\times

= No support. Annotations: 1 Generates diversity through diffusion but only within goal-constrained trajectories. 2 Conditions on fixed human-provided thoughts, limiting emergent diversity. 3 Uses linguistic signals that remain external annotations rather than internally generated mediators. 4 Bootstraps from visual observations using VLM-generated captions, eliminating the need for human language annotations.

Panel of Figure 7 — Figure 7: Environments used in our experiments. (a-c) : D3IL environments Aligning, Stacking and Sorting environments, respectively. (d-f) : Overcooked map layouts Cramped-room, Coorindation ring, and Asymmetric Advantage, respectively.

Panel of Figure 8 — Figure 8: TSNE visualization of CLIP-encoded inner speech generated using GPT-4o for the environments used in our experiments. (a-f) : Top and inhand cameras in D3IL: Aligning, Stacking and Sorting environments, respectively. (g-l) : Blue and green hat agents in overcooked map layouts: Cramped-room, Coorindation ring, and Asymmetric Advantages, respectively.

Environment	Model	$p_{drop}$	$t_{0}$	$W$	Success rate $\uparrow$	Distance $\downarrow$	Entropy $\uparrow$
Aligning	BC				0.6645	0.1105	0.4743
	MIMIC-S	0.1	50	50	0.8021	0.0664	0.4184
	MIMIC-E	0.1	12	50	0.7229	0.0847	0.6148
Aligning-Vision	BC				0.1833	0.1875	0.0895
	MIMIC-S	0.1	1	20	0.2229	0.1885	0.0849
	MIMIC-E	0.0	1	20	0.2083	0.1849	0.1473

Appendix

Table 5: Comparison of MIMIC against BC with the DDPM-T architecture on the D3IL benchmark.

Environment	Model	$p_{drop}$	$t_{0}$	$W$	Collective reward	Action Wasserstein
Cramped room	PPO ${}_{H_{proxy}}^{*}$				$\sim 155-160$	-
	BC				115.8 $\pm$ 3.86	0.24
	MIMIC	0.1	10	100	151.8 $\pm$ 2.45	0.25
	MIMIC-MA	0.1	1	50	148.4 $\pm$ 2.17	0.25
Cramped room-Vision	BC				73.6 $\pm$ 6.18	-
	MIMIC	0.0	1	50	108.8 $\pm$ 4.84	-

Appendix

Table 6: Comparison of MIMIC against BC with DDPM-T on the Overcooked environments. ‘-’ denotes “action Wasserstein" is not feasible or not available. ∗ denotes values taken directly from [ 5 ] . Note that “state Wasserstein" is infeasible due to a large dimension (96) of state features.

Panel of Figure 9 — Figure 9: Hyperparameter sensitivity of MIMIC on the D3IL benchmark.

Panel of Figure 10 — Figure 10: Sensitivity on overcooked Cramped-room

Aligning					Overcooked cramped room
	Model	Success rate	Distance	Entropy		Model	Collective Reward
	BeT	0.51667	0.12949	0.40475		BeT	47.2 $\pm$ 4.64
	BeSO	0.85417	0.04954	0.6141		BESO	67.8 $\pm$ 4.55
	MIMIC-S	0.88125	0.04234	0.7215		MIMIC	120.2 $\pm$ 2.86
	MIMIC-E	0.86875	0.04759	0.7706		MIMIC-MA	141.2 $\pm$ 3.68

Appendix

Table 8: Comparison with other imitation learning models. Here, we use BESO as the base diffusion policy network in MIMIC instead of DDPM-T for fairness.

Panel of Figure 11 — Figure 11: Conditional generation on the Aligning dataset. The color gradient (Plasma) shows the simulation time going from 0 (dark purple) to the end (bright yellow), going through red and orange. Inner speech updates are marked as yellow circles with corresponding times inside; the first inner speech is equal to the specified condition.

Panel of Figure 12 — Figure 12: Conditional generation on the Sorting dataset. The color gradient (Plasma) shows the simulation time going from 0 (dark purple) to the end (bright yellow) going through red and orange. Inner speech updates are marked as yellow circles with corresponding times inside; the first inner speech is equal to the specified condition.

Timestep	Closest description	Similarity
t=49	I push the box directly without rotation, aiming for a straightforward alignment.	0.9183
t=99	I approach directly, then rotate in place to align perfectly.	0.9614
t=149	I approach directly, then rotate in place to align perfectly.	0.9614
t=249	Starting from a slightly rotated angle, I need a mid-action adjustment to align.	0.9616

Appendix

Table 18

Timestep	Closest description	Similarity
t=49	I approach the box from the side, rotating slightly to align smoothly with the fixed box.	0.9471
t=99	I approach directly, then rotate in place to align perfectly.	0.9614
t=149	I approach directly, then rotate in place to align perfectly.	0.9619
t=249	I approach with a direct path but make a last-second adjustment to align perfectly.	0.9611

Appendix

Table 19

Timestep	Closest description	Similarity
t=49	I execute a direct push from behind, minimizing lateral movement.	0.9477
t=99	I carefully approach from the left, ensuring alignment from a diagonal perspective.	0.9613
t=149	Starting from a slightly rotated angle, I need a mid-action adjustment to align.	0.9613
t=249	I approach with a direct path but make a last-second adjustment to align perfectly.	0.9613

Appendix

Table 20

Timestep	Closest description	Similarity
t=0	I start with a straight approach from a central position, requiring minimal rotation for alignment.	0.9693
t=50	I start with a straight approach from a central position, requiring minimal rotation for alignment.	0.9691
t=100	I start with a straight approach from a central position, requiring minimal rotation for alignment.	0.9696

Appendix

Table 21

Timestep	Closest description	Similarity
t=100	I focus on sorting all blocks of one color first before switching to the other.	0.9607
t=300	I focus on sorting all blocks of one color first before switching to the other.	0.9604

Appendix

Table 22

Timestep	Closest description	Similarity
t=100	I quickly grab onions from the pile and place them in the pot, prioritizing speed over precision.	0.9082
t=200	I quickly grab onions from the pile and place them in the pot, prioritizing speed over precision.	0.9096
t=300	I quickly grab onions from the pile and place them in the pot, prioritizing speed over precision.	0.9062
t=400	I quickly grab onions from the pile and place them in the pot, prioritizing speed over precision.	0.9086

Appendix

Table 23

Timestep	Closest description	Similarity
t=60	I adjust its movement pattern to avoid congestion, optimizing its task execution efficiency.	0.9521
t=110	I adjust its movement pattern to avoid congestion, optimizing its task execution efficiency.	0.9405
t=160	I adjust its movement pattern to avoid congestion, optimizing its task execution efficiency.	0.9466
t=210	I adjust its movement pattern to avoid congestion, optimizing its task execution efficiency.	0.9407
t=260	I adjust its movement pattern to avoid congestion, optimizing its task execution efficiency.	0.9395

Appendix

Table 24

Timestep	Closest description	Similarity
t=50	I maneuver around obstacles effectively.	0.8391
t=100	I balance interactions with other agents and time efficiency.	0.8416
t=150	I balance interactions with other agents and time efficiency.	0.8463
t=200	I maneuver around obstacles effectively.	0.8443
t=250	I balance interactions with other agents and time efficiency.	0.8401

Appendix

Table 25

Explore similar work

The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction
Aug 23, 2026Xunzhe Zhou, Yiyang Cai, Fengyi Wang +9Imitation Learning Robot Imitation Learning
The University of Hong Kong · TranscEngram · Fudan University +1
MIMESIS: Learning User Simulators as Training Environments for Interactive Agents
Oct 7, 2026Hoang Phan, Dat Huynh, Andrey Zhmoginov +7User Simulation LLM Agent Training
Meta Superintelligence Labs · New York University · University of Wisconsin - Madison
MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies
Sep 17, 2025Dayi Dong, Maulik Bhatt, Seoyeon Choi +1Multi-Agent Coordination Diffusion Policy
Department of Mechanical Engineering, University of California Berkeley, Berkeley, CA 94709, USA

Aug 23, 2026J/K move · Enter open · S save

The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

Xunzhe Zhou, Yiyang Cai, Fengyi Wang, Ran Ju+8

The University of Hong Kong · TranscEngram · Fudan University+1

Imitation Learning Robot Imitation Learning Learning from Demonstration Cross-Embodiment Robot Learning

Open paper AskPDF arXiv

Humans imitate at the level of intent: given a demonstration, we infer its goal and carry it out with whatever tools, objects, and layouts are at hand. Current robot policies instead learn observation-to-action mappings from visual inputs and language instructions, without explicitly inferring the demonstrated task. Learning from human video thus remains largely trajectory-level: models can replay motions in near-identical scenes, but still struggle to imitate what the demonstrator intends rather than merely what they do. We introduce The Imitator Game, a four-level benchmark (L0-L3) that progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice and task understanding becomes necessary. We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date and the only one instantiated across all four levels in both real and simulated settings (20,000+ paired episodes, 50+ tasks, 6 domains), and Imitator Arena, an open platform for blind A/B human evaluation. Across nine state-of-the-art models, performance is stable from L0 to L2 but collapses at L3, identifying functional substitution - achieving the same intent through a different object affordance - as the decisive barrier to intent-level imitation. Human-video-conditioned models outperform caption-conditioned ones, yet every model falls below 13% zero-shot success on unseen tasks; fine-tuning IG-10K-pretrained models with only

10

paired human-robot demonstrations yields large gains that grow with pretraining scale. The project website and access to Imitator Arena are available at https://imitator-game.github.io.

Abstract

Explore similar work

The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

MIMESIS: Learning User Simulators as Training Environments for Interactive Agents

MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies