AgentDreamer: Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation

Mijeong Kim; Suho Park; Junha Park; Bohyung Han

cs.CVOct 22, 2026

AgentDreamer: Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation

Authors: Mijeong Kim, Suho Park, Junha Park, Bohyung Han

Abstract

We propose AgentDreamer, an agent-guided optimization framework that aligns text instructions with simulated motion through explicit 3D motion planning. Given a text instruction and scene geometry, an off-the-shelf agent translates the instruction into a structured semantic motion prior, specifying what should move, in which direction, and in what temporal order. The agent represents this prior using parametric functions over a series of temporal chunks, yielding dense 3D trajectories. We then optimize the parameters of a differentiable MPM simulator such that its rollout best matches the prior while remaining constrained by the scene dynamics. This allows the agent to provide semantic guidance from language, while the simulator grounds it in physically feasible motion. We further introduce Motion-IG, an information-theoretic metric for text-motion alignment based on masked text recovery. By comparing the likelihood of recovering masked motion-related content with and without the simulated motion, Motion-IG measures the information contributed by the motion beyond that available from the remaining text and scene context. Experiments show that AgentDreamer improves text alignment over video-based guidance, producing simulations that more closely follow the requested physical behavior.

Figures & tables

Figure 1: 3D motion planning as physical reasoning. Given a text instruction and scene geometry, the agent reasons about the physical behavior required by the instruction and constructs a corresponding parametric motion plan. The plan produces dense 3D trajectories that guide physics optimization toward the requested motion.

Figure 2: Overview of AgentDreamer. Given a text instruction and scene geometry, an off-the-shelf agent constructs a structured semantic motion prior using parametric functions over temporal chunks. The resulting dense 3D trajectories guide optimization of a differentiable MPM simulator under the scene dynamics.

Figure 3: Effect of parametric motion planning. Direct prediction of future point positions produces irregular trajectories and distorts the object structure (top). Parametric functions yield coherent trajectories that better preserve spatial relationships between points (bottom). Colors identify corresponding points across time.

Sequence	Instruction
$X$	The soft fox sways from side to side while remaining attached at the rear.
$X_{m}$	The [MASK] fox [MASK] from [MASK] to side while [MASK] [MASK] at the rear.
$X$	The basketball rolls forward in a straight line , rotating naturally without slipping .
$X_{m}$	The basketball [MASK] [MASK] in a [MASK] [MASK], [MASK] [MASK] [MASK] [MASK].
$X$	The rigid playdoh ball falls straight down and bounces several times without deforming .
$X_{m}$	The [MASK] playdoh ball [MASK] [MASK] [MASK] and [MASK] [MASK] [MASK] without [MASK].

Table 1: Examples of word-level masking for Motion-IG. Bold words in the original instruction

X

are replaced by [MASK] in the masked instruction

X_{m}

Figure 4: Qualitative comparison of text-controllable physics simulation. AgentDreamer more closely follows the requested motion than PhysFlow and MotionPhysics in the shown examples. The shown examples contrast the requested motion with the behaviors produced by video-guided optimization and explicit 3D planning.

Table 2: Comparison of text–motion alignment using Motion-IG with (a) token likelihoods from open-weight evaluators and (b) verbalized probabilities from black-box API evaluators (higher is better). Best in counts the evaluator–instruction pairs in which each method ranks first.

Figure 5: Pairwise human preference for text–motion alignment. Win rate is the percentage of trials in which one method is ranked above the other. AgentDreamer is preferred over PhysFlow in 82.6% of trials and over MotionPhysics in 85.6% of trials.

Motion cue	Function family	Representative expression	Matched / Total
Roll	Coupled translation and rotation	$d(\tau)=v\tau,\quad\theta(\tau)=(v/R)\tau$	2/2
Spin, slow down, stop	Quadratic	$\theta(\tau)=\omega_{0}\tau+\frac{1}{2}\alpha\tau^{2},\quad\alpha<0$	2/2
Fall, bounce, jump	Quadratic	$z(\tau)=z_{0}+v_{z,0}\tau-\frac{1}{2}g\tau^{2}$	4/4
Sway, oscillate, rock	Sinusoid	$\theta(\tau)=Ae^{-\lambda\tau}\sin(\omega\tau)$	8/8
Pull, bend, squash, deform	Smoothstep interpolation	$s(u)=3u^{2}-2u^{3}$	4/4
$\tau$ : elapsed time within a chunk; $u=\tau/(t_{j}-t_{j-1})$ : normalized time.

Table 3: Correspondence between agent-generated motion functions and expected forms across

20

instructions. The expected forms follow classical mechanics for rolling, spin decay, falling, and oscillation, while deformation uses smoothstep interpolation. Expressions show representative components, and Matched / Total counts plans that use the expected form in each category.

Figure 8: Temporal decomposition of an agent-generated motion plan. The agent uses separate parametric functions to describe the ball movement, compression on impact, and partial shape recovery. Dashed lines mark temporal chunks, and colored points show target positions over time.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Additional qualitative comparisons with PhysFlow ( Liu et al., 2025 ) and MotionPhysics ( Wang et al., 2026 ) under the same text instructions. The left panels show frames from the simulated videos, while the right panels show the corresponding sampled-point trajectories. In the shown examples, AgentDreamer produces simulated motion that more closely follows the requested behavior.

Figure 11: Additional qualitative comparisons with PhysFlow ( Liu et al., 2025 ) and MotionPhysics ( Wang et al., 2026 ) under the same text instructions. The left panels show frames from the simulated videos, while the right panels show the corresponding sampled-point trajectories.

		Open-weight (avg.)			Black-box (avg.)
Category	$\bm{n}$	PhysFlow	MotionPhysics	Ours	PhysFlow	MotionPhysics	Ours
Fall and bounce	4	$+0.18$	$+0.36$	$\mathbf{+0.63}$	$-1.46$	$-0.87$	$\mathbf{+0.77}$
Roll	3	$-0.22$	$-0.24$	$\mathbf{+0.46}$	$-1.23$	$-2.01$	$\mathbf{+0.92}$
Spin decay	2	$+0.29$	$+0.20$	$\mathbf{+0.55}$	$-0.15$	$-0.23$	$\mathbf{+0.62}$
Oscillation	9	$+0.77$	$+0.75$	$\mathbf{+0.83}$	$+0.33$	$-0.50$	$\mathbf{+0.35}$
Bend and deform	5	$+0.67$	$+0.70$	$\mathbf{+0.74}$	$+0.34$	$-0.12$	$\mathbf{+0.39}$

Appendix

Table 5: Text–motion alignment per motion category measured by Motion-IG (higher is better), averaged over four open-weight and five black-box evaluators.

Method	1st (%) $\uparrow$	2nd (%)	3rd (%) $\downarrow$
PhysFlow	14.0	45.8	40.2
MotionPhysics	12.4	33.2	54.4
Ours	73.6	21.0	5.4

Appendix

Table 6: Rank distribution in the human evaluation (25 participants

\times

20 instructions

=

500 trials). Each entry is the percentage of trials in which the method receives the rank.

Figure 12: Examples of annotated images used for automatic semantic injection. Control points sampled from the initial scene geometry are marked with numerical identifiers in two views of a fox (top) and a soccer ball (bottom).

Figure 13: Prompt for agent-based motion planning. The agent divides the time horizon into temporal chunks and specifies parametric motion functions for groups of sampled points.

Figure 14: Prompt for translating the motion plan into deterministic Python code. The code implements the specified motion functions to compute target point positions.

Figure 15: Prompt for the open-weight evaluators. The masked words are scored by teacher forcing, and the same prompt without the motion gives the reference term.

Figure 16: Prompt for the black-box evaluators. Each masked word receives a stated probability, collected in one JSON response per query.

Figure 17: Example question in the human evaluation of text–motion alignment. Given a text instruction and three simulation results for the same object, participants rank the motions by how closely they follow the requested behavior.

Text-to-Motion Generation

Text-to-motion generation converts natural-language descriptions into semantically aligned, temporally coherent 3D human movement.

See all 96 papers

Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Training
Sep 22, 2026Raphael Memmesheimer, Sven BehnkeHuman Motion Generation Robot Motion Generation
Autonomous Intelligent Systems, Computer Science Institute VI, Lamarr Institute for Machine Learning and Artificial Intelligence, and Center for Robotics, University of Bonn, Bonn, Germany
In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
Jun 25, 2026Xiaomeng Fu, Junfan Lin, Yang Liu +4Text-to-Motion Generation Human Motion Generation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise
Sep 29, 2026Xiaoyu Xiong, Tsun-Hsuan Wang, Yi-Ling Qiao +2Physics-Based Simulation
Tsinghua University · Carnegie Mellon University · Genesis AI +1

Sep 22, 2026J/K move · Enter open · S save

Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Training

Raphael Memmesheimer, Sven Behnke

Autonomous Intelligent Systems, Computer Science Institute VI, Lamarr Institute for Machine Learning and Artificial Intelligence, and Center for Robotics, University of Bonn, Bonn, Germany

Human Motion Generation Robot Motion Generation Humanoid Motion Tracking

Open paper AskPDF arXiv

Text-to-motion models generate plausible human motion but do not model a robot's dynamics; whole-body tracking controllers execute robot references reliably but cannot replan an infeasible one. Recent language-to-humanoid systems bridge this gap by training. We measure how much of the gap closes with no training at all, by putting the deployment controller itself in the loop. Sample-simulate-select (S

^3

) draws

N

motions per prompt from a frozen text-to-motion model, retargets each to a Unitree G1 by direction-matching inverse kinematics, rolls all of them out under full rigid-body dynamics with the pretrained SONIC tracking policy, and keeps the candidate the policy executed best. Because the verifier is the deterministic simulator itself, S

^3

attains the any-of-

N

ceiling by construction; what we measure is where that ceiling lies and what falls short of it. On 200 stratified HumanML3D test prompts with

N=8

, upright execution rises from 83.5% to 89.5% and hardware-gate passes from 33 to 85; on the complete test split (4,184 prompts) it rises from 80.5% to 89.5%. A kinematic verifier that predicts falls well (AUROC 0.90) recovers only a quarter of this gain: ranking a prompt's own candidates is harder than classifying the population. What selection cannot fix is one class, prompts that lower the pelvis, which a generator trained on retargeted robot data does execute. We further score the semantic fidelity of the executed motion with the standard text-motion evaluator, with a real-mocap control that attributes the loss to the robot projection, ablate the retargeter against GMR (complementary failures: the any-of-8 ceiling rises to 95.0% over both), and execute all 177 gate-selected clips on the real G1: every one completes standing, with hardware tracking error matching simulation (

r=0.94

Abstract

Text-to-Motion Generation

Explore similar work

Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Training

In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise