cs.ROApr 9, 2026
SaveGenPHRI: Agentic Generative Simulation for Physical Human-Robot Interaction
Organizations: Carnegie Mellon University, Pittsburgh, PA, USA
Abstract
Real-world physical human-robot interaction (pHRI) development is costly and slow, while simulation requires substantial task-specific scene and motion authoring. We present GenPHRI, a generative simulation framework for producing deployment-ready pHRI scenarios entirely from natural-language task descriptions using vision-language model (VLM) agents. Generated scenarios contain a room with furniture, a task-appropriate human pose and placement, and a robot executing the requested interaction. Generator and critic agents iterate on each component, and an orchestrator agent controls cross-stage decisions. With GenPHRI, we generate 50 different assistive scenarios in simulation with no manual intervention and deploy two tasks in a 12-participant user study. Vision policies trained on these scenarios achieve zero-shot task completion rates of ~80% in real-world trials against ~90% in simulation, while producing behaviors that participants rate as consistent with textual task prompts. Further, we demonstrate direct execution of generated motion as a complementary, training-free deployment pathway that enables rapid real-world inspection of generated trajectories at the cost of contact coverage. We release GenPHRI as a platform for generating, training, and deploying pHRI tasks from textual prompts. Additional information may be found on our project website: https://rchi-lab.github.io/gen_phri/
Figures & tables
Figure 1. Overview of GenPHRI, our proposed agentic generative simulation framework for synthesizing physical human-robot interaction (pHRI) scenarios. From a natural-language task prompt, GenPHRI generates complete scenarios in simulation, which can be transferred zero-shot to real-world robot behaviors by directly executing the motion or training a policy.
Figure 2. Flowchart of GenPHRI, an agentic generative simulation system for physical human-robot interaction. An orchestrator agent takes in a task prompt and sends detailed prompts to four generation stages that compose a scenario, including a posed human and robot within a scene and task-completing robot motion. The resulting scenario can be either transferred directly into the real world through mesh fitting and motion planning, or used in training a policy with zero-shot sim-to-real capabilities.
Figure 3. Robot placement before (left) and after (right) a critic iteration. The first candidate places the robot intersecting the furniture, while the revised placement is valid.
Figure 4. Robot motion generation. Left: excerpt from task prompt and generated trajectory code. Right: visualization of the resulting motion trajectory.
| Stage | Scenarios with 1 critic rejection | Mean critic iterations | Orchestrator decisions |
| 1 | 22/50 | 2.02 | 5 |
| 2 | 40/50 | 3.22 | 7 |
| 3 | 14/50 | 4.16 | 25 |
| 4 | 37/50 | 6.62 | 18 |
| Total | 50/50 | 16.0 | 55 |
Table 1. Critic and orchestrator activity in generating the 50 scenarios. Stages are: 1. Human generation, 2. Human placement, 3. Robot placement, and 4. Robot motion generation.
| Configuration | All waypoints reachable (%) | Tracking excursions 2 cm (%) | P95 final pos. error (mm) |
| Full pipeline | 100.0 | 0.0 | 1.8 |
| Critics only | 57.0 | 4.0 | 152.3 |
| Critics only, minus: | |||
| Human Generation | 0.0 | 100.0 | 155.9 |
| Human Placement | 37.0 | 50.0 | 77.4 |
| Robot Placement | 1.0 | 81.8 | 69.9 |
Table 2. Reachability and target-tracking metrics for the pipeline ablation.
Figure 5. Real-world configurations for bathing and scratching, with visualizations of robot motion, final outcome, and human arm movement in scratching.
| Bathing | Scratching | |||
| Policy | Direct | Policy | Policy+Motion | |
| Real | ||||
| Sim | — | |||
Table 3. Mean target completion (% SD across humans) in physical and simulated evaluations. Physical targets require removal or contact. Simulated targets use a 2 cm proximity criterion.
Figure 6. Participant ratings of prompt adherence. “ns” denotes a nonsignificant difference in the two-sided paired comparison.
| 1 Human generation | 2 Human placement | 3 Robot placement | 4 Motion generation | |
| Generator input | • Two-stage prompt: qualitative pose description, then joint angles via few-shot description–pose pairs • Body-shape seed | • Top-down render of the empty furniture • Furniture-frame coordinate rules (normalized ; head and facing directions) | • API: get_joint_pos over 11 body landmarks • Constraints: -axis toward the workspace, camera head-on, yaw-only, base outside the person | • API: get_joint_pos over 12 landmarks; surface projection of a landmark onto the body point cloud; per-point normals; camera pose • Waypoint schema and contact constraints |
| Generator output | • SMPL-X parameters for body shape • SMPL-X angles for hips, knees, neck, shoulders, elbows • Posture class (sitting / lying) | • in the furniture frame • head direction • facing direction | • Python get_robot_placement(human) returning base position and yaw | • Python generate_trajectory(…) returning waypoints • Each waypoint: position, orientation, velocity, force , planner RRT, Cartesian |
| Critic input | • Top-down and axonometric renders • Current joint angles • History | • Top-down render (primary); isometric render to confirm or dismiss intersections • Current placement parameters • History | • Top-down and isometric renders of scene, human and robot • Current code • Tagged metrics • History | • Top-down and isometric trajectory renders over the body; rollout video • Current code • Tagged metrics incl. per-waypoint reachability • History |
| Critic checklist | • Implausible bending or twisting • Broken limb configuration • Any self-intersection • Pose-task mismatch | • Body-furniture relation matches the task • No body-furniture intersection (limbs may hang off when lying; near the edge when seated) • Near the edge of wide furniture so a robot can reach • Whether , , head or facing should change | • Proximity: not too far to reach the workspace • Orientation: faces the interaction area head-on • Task suitability | • Contact velocity safe • Force only on segments that start and end in contact • RRT in free space, Cartesian in contact • Every waypoint reachable • Contact waypoints cover the body region the task names |
| Report: gates | • None | • None | • G: base tilt after settling | • G: schema assertions (force, planner, contact velocity m/s, first waypoint force-free) • G: all waypoints reachable |
| Report: advisory flags | • F: self-collision pairs • F: max penetration cm | • None | • None | • F: contact waypoints off target • F: surface-normal angle • F: contact segment cm from its landmark • F: goal-observation distance mean cm / max cm |
Table 4. Generator input/output, critic input/criteria, and report gates/flags throughout the four generation stages.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Gates | Advisory flags |
| Data collection | No nonfinite values in training data. | • Dropped-episode fraction • Episode-length ratio outside • Mean tracking error cm or maximum cm |
| Policy training | • Planned epochs completed • No training or validation batches with nonfinite loss | • Final validation loss the best epoch’s loss • Reduction from initial to best validation loss • Worst held-out episode loss the median episode loss |
| Simulated evaluation | • Every configured human produces an evaluation episode • Success rates on training and held-out bodies each meet the configured minimum | • Training-body success rate minus held-out success rate • Mean goal error cm • Stalled-episode fraction • Worst subgoal completion • Success-rate spread across humans • Visual-numerical success agreement |
Table 5. Stage reports for the policy-learning branch. Gates specify required conditions; advisory flags identify concerns for orchestrator review. The same training criteria apply separately to the high- and low-level policies.
| Activity Family | Count | Primary Target Body Regions |
| Back percussion | 7 | Upper back, mid back, lower back |
| Itch relief | 7 | Mid/lower back, scalp, shoulder, arm, leg |
| Sponge bathing | 6 | Head/neck, foot, hand, back, leg, arm |
| Lotion application | 5 | Arm/hand, back, leg, shoulder/neck, hips |
| Skin assessment | 5 | Heel, hips, spine, shoulder, elbow |
| Hand warming | 5 | Palm, back of hand, knuckles, wrist |
Table 6. Summary of the 50 generated caregiving scenarios grouped by activity family, detailing scenario counts and primary target body regions.
Figure 7. Illustrations of Fast SAM3D Body fitting results, compared against the actual human point cloud.
| Task | High-level (h) | Low-level (h) | Total (h) |
| Bathing | 17.5 | 2.3 | 19.8 |
| Scratching | 17.7 | 1.2 | 19.0 |
Table 7. Approximate policy training time on four NVIDIA RTX PRO 5000 Blackwell GPUs.
More in this topic
Human-Robot Interaction
Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.
See all 452 papers