Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback
Organizations: Aalto University
Abstract
Recent work on human-scene interaction (HSI) generation moves from single interactions to long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction. However, existing agents struggle in challenging everyday scenarios (e.g., picking up a toy and throwing it into a basket) because they lack spatial control over the motion and quantitative feedback on its execution, and they often fail to complete the interaction sequence. To address these gaps, we present Inhabitant, a training-free agent framework in which the agent specifies each motion through positions, paths and poses, measures how the executed interaction deviates from its intent, and revises the control specification accordingly. For an extensive evaluation in such scenarios, we build Inhabit, a benchmark of 75 household tasks in 31 furniture layouts of an apartment, in which every task chains one to three goals on movable objects and is carried out by the whole body, excluding fingers. On Inhabit, Inhabitant completes 98.7% of the tasks with Claude Opus 5.5, and 93.3% and 82.7% with Claude Sonnet 5.5 and Gemini 4 Argon, respectively. Ablations confirm that without spatial control the agent fails most tasks, and without quantitative feedback its motion becomes less precise. Code is available on the project page.
Figures & tables
| Qualitative feedback , after every try | |
|---|---|
| Check | whether the success rule is satisfied by the sequence extended by the candidate segment |
| Frames | eight frames of the candidate segment, each from two third-person cameras: one follows the body, one views the body and the target of the motion from the side |
| Quantitative feedback , on a query | |
| Body | (i) position of a body part; (ii) distance from a body part to an object, a surface or a handle; (iii) contacts of the body; (iv) pose that the body executed at a frame |
| Scene | (v) position of an object; (vi) distance between two objects or between an object and a region; (vii) extent of an object or a piece of furniture |
| Success rate (%) | Motion quality | Cost | |||||||||||
| Method | Language model | All | low | mid | high | posture | Sub-goal | Collisions | Disturbed | Skate (%) | Tries | Calls | Tokens |
| InteractAgent † [ 60 ] | Claude Opus 5.5 | 26.7 | 53.3 | 10.0 | 0.0 | 20.0 | 34.5 | 105.0 | 2.4 | 0.9 | 52.8 | 133 | 123k |
| HumanCLAW † [ 37 ] | Claude Opus 5.5 | 20.0 | 33.3 | 15.0 | 6.7 | 10.0 | 26.8 | 42.1 | 0.8 | 3.6 | 11.9 | 12 | 91k |
| GEMS † [ 1 ] | Claude Opus 5.5 | 6.7 | 13.3 | 5.0 | 0.0 | 0.0 | 13.5 | 11.9 | 1.0 | 5.3 | 3.8 | 7 | 98k |
| Inhabitant | GLM-5.3-Flash | 44.0 | 80.0 | 10.0 | 0.0 | 70.0 | 53.2 | 36.8 | 1.7 | 5.9 | 38.2 | 112 | 221k |
| gpt-6-luna | 30.7 | 40.0 | 25.0 | 6.7 | 50.0 | 41.4 | 45.3 | 1.5 | 5.4 | 46.4 | 122 | 143k | |
| Control specification | Success | Collisions | Missed grasps | Tries | Calls |
|---|---|---|---|---|---|
| Atomic skills | 30.0 | 8.4 | 68 | 12.6 | 32 |
| Text prompts only | 20.0 | 7.2 | 73 | 61.0 | 134 |
| + root waypoints | 90.0 | 9.6 | 62 | 27.9 | 69 |
| + end-effector targets | 95.0 | 7.7 | 45 | 17.4 | 51 |
| + full-body keyframes (Inhabitant) | 95.0 | 7.8 | 40 | 14.8 | 47 |
| # | Check | Frames | Body | Scene | Success | Collisions | Missed grasps | Tries | Calls |
|---|---|---|---|---|---|---|---|---|---|
| 1 | ✓ | ✓ | 85.0 | 14.6 | 58 | 22.7 | 54 | ||
| 2 | ✓ | ✓ | ✓ | 95.0 | 10.9 | 47 | 19.3 | 64 | |
| 3 | ✓ | ✓ | ✓ | 95.0 | 12.4 | 61 | 22.0 | 59 | |
| 4 | ✓ | ✓ | ✓ | ✓ | 95.0 | 7.8 | 40 | 14.8 | 47 |
| 5 | ✓ | ✓ | ✓ | 85.0 | 9.5 | 47 | 19.8 | 60 | |
| 6 | ✓ | 70.0 | 11.1 | 53 | 21.6 | 54 |
| Setting | Success | Collisions | Tries | Calls |
|---|---|---|---|---|
| Inhabitant | 95.0 | 7.8 | 14.8 | 47 |
| Single-Try | 75.0 | 30.4 | 13.2 | 36 |
| Open-Loop | 0.0 | 10.9 | 4.2 | 2 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Call | Tool and arguments | Result |
|---|---|---|
| generate_segment: text prompts, durations, constraints, seed, post-processing, guidance weights ( ) and interaction events ( ) | success check and eight rendered frames of the try | |
| inspect: a list of queries | one value or time series per query | |
| commit | commit | state description within 2 m of the body |
| retry_from: index of a committed segment | state description | |
| – | get_state | full state description |
| Query | Kind | Returns |
|---|---|---|
| where | body, scene | position of a body part or an object, now, at a time of the try or every 0.1 s; for an object also the surface or hand it is on, for a handle whether it is open |
| distance | body, scene | distance between two entities (body parts, objects, surfaces, furniture) every 0.1 s, with the direction at the closest moment |
| contacts | body | contacts of the body with the scene during the try |
| pose | body | pose that the body executed at a time of the try, in the format of a full-body keyframe |
| extent | scene | corners and heights of an object, a surface or a piece of furniture |
| fk | – | joint positions of a pose written as a full-body keyframe, before it is used in a try (Sec. 3.3 ) |
| Generator | Root (m) | End effector (m) | Within 10 cm | Heading ( ∘ ) | Time (s) |
|---|---|---|---|---|---|
| Kimodo, full control specification | 0.035 | 0.005 | 100% | 1.5 | 3.7 |
| OmniControl | 0.085 | 0.057 | 81% | 39.9 | 95.4 |
| Kimodo, text prompts only | 1.886 | 0.708 | 0% | 52.9 | 3.6 |
| Tier | Tasks | with throw | by foot | EB original |
|---|---|---|---|---|
| low (1 goal) | 30 | 5 | 5 | 3 |
| mid (2 goals) | 20 | 3 | 4 | 3 |
| high (3 goals) | 15 | 0 | 3 | 2 |
| posture | 10 | – | – | 0 |
| total | 75 | 8 | 12 | 8 |
| Condition | Holds if |
|---|---|
| on | the object rests inside the region of the target surface and is not held |
| in | the centre of the object is inside the target box or basket, wherever that container is now, and the object is not held |
| inside | the object is inside the fridge and is not held |
| open | the fridge door is opened by more than 1.07 rad, or the drawer is pulled out by more than 0.35 m |
| closed | the joint of the door or drawer is within 0.15 of its closed position |
| throw | the object ends on or in the target, and the palm was at least 1 m (horizontally) from the target when the object left the hand |
| Task | Instruction | Conditions |
|---|---|---|
| low_01 | Could you tidy up the toy airplane? It goes in the cardboard box by the TV stand. | in |
| low_02 | Put the lego in the cardboard box next to the black table. | in |
| low_03 | Put the toy airplane in the cardboard box next to the black table. | in |
| low_04 | The towel is on the floor. Put it back on the sofa. | on |
| low_05 | There’s a towel lying on the floor - could you put it back on the sofa? | on |
| low_06 | Pick up the remote control from the floor and put it on the TV stand. | on |
| Task | Instruction | Conditions |
|---|---|---|
| mid_01 | Could you tidy the toy airplane and the rubriks cube into the basket by the black table? | 2 in |
| mid_02 | Could you tidy the lego and the toy airplane into the cardboard box by the sofa? | 2 in |
| mid_03 | Tidy up: the rubriks cube goes in the basket next to the sofa; throw the toy airplane in from a metre or more. | in, throw |
| mid_04 | The fridge is open. Put the banana inside and close the door. | inside, closed (ordered) |
| mid_05 | The fridge is open. Put the peach inside and close the door. | inside, closed (ordered) |
| mid_06 ∗ | Take both the coffee mug and the serving bowl on the black table to the sink in one go - one in each hand. | 2 on, together |
| Task | Instruction | Conditions |
|---|---|---|
| high_01 | Clear the sofa: all the things on it go on the top shelf of the tall shelf next to the black table. | 3 on |
| high_02 | Clear the sofa: all the things on it go on the top shelf of the tall shelf next to the brown table. | 3 on |
| high_03 ∗ | Take everything out of the fridge, put it on the black table, and close the fridge again. | 2 on, closed |
| high_04 | Empty the fridge onto the black table, then shut the fridge door. | 2 on, closed |
| high_05 | Put all the things on the black table into the cardboard box beside it. | 3 in |
| high_06 | Clear everything off the black table into the cardboard box next to it. | 3 in |
| Failed | Budget | Stopped | Plan | Missed | |
| Baseline | episodes | used up | by the agent | executed | grasps |
| InteractAgent † | 55 | 46 | 9 | – | 83% |
| HumanCLAW † | 60 | 0 | 60 | – | 75% |
| GEMS † | 70 | – | – | 70 | 68% |
| Tokens per episode | Cost (USD) | |||||
| Method | Language model | Input | Cached | Output | Episode | Solved task |
| InteractAgent † | Claude Opus 5.5 | 6.3M | 98% | 39k | 2.56 | 9.62 |
| HumanCLAW † | Claude Opus 5.5 | 1.5M | 95% | 16k | 0.96 | 4.80 |
| GEMS † | Claude Opus 5.5 | 1.3M | 92% | 32k | 1.41 | 21.08 |
| Inhabitant | GLM-5.3-Flash | 15.4M | 0% | 106k | – | – |
| gpt-6-luna | 10.5M | 87% | 51k | 0.25 | 0.82 | |
| Task | IA † | HC † | GEMS † | GLM | luna | Gemini | Sonnet | Opus | Task | IA † | HC † | GEMS † | GLM | luna | Gemini | Sonnet | Opus | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| low_01 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | mid_09 | ✓ | ✓ | ✓ | ✓ | |||||||
| low_02 | ✓ | ✓ | ✓ | ✓ | ✓ | mid_10 | ✓ | ✓ | ✓ | |||||||||
| low_03 | ✓ | ✓ | ✓ | ✓ | mid_11 | ✓ | ✓ | ✓ | ||||||||||
| low_04 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | mid_12 | ✓ | ✓ | |||||||
| low_05 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | mid_13 | ✓ | ✓ | ✓ | ✓ | ||||||
| low_06 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | mid_14 | ✓ | ✓ | ✓ |
| Opus 5.5 | Sonnet 5.5 | Gemini | GLM | luna | InteractAgent † | |
| Segments with a spatial constraint | 97% | 96% | 100% | 90% | 90% | – |
| with an end-effector target or a full-body keyframe | 37% | 26% | 49% | 17% | 4% | – |
| Committed at the first try | 46% | 52% | 48% | 46% | 36% | 32% |
| Tries followed by a measurement | 93% | 99% | 98% | 80% | 95% | – |
| Revisions that change only the seed | 6% | 7% | 0% | 1% | 1% | – |
| Missed grasps | 48% | 66% | 23% | 73% | 79% | 83% |
| Task | Skills | Text | + root | + end eff. | + keyfr. |
| low_07 | ✓ | ✓ | ✓ | ||
| low_11 | ✓ | ✓ | ✓ | ||
| low_12 | ✓ | ✓ | ✓ | ||
| low_16 | ✓ | ✓ | ✓ | ✓ | ✓ |
| low_18 | ✓ | ✓ | ✓ | ||
| low_21 | ✓ | ✓ |
| Task | 1 | 2 | 3 | 4 | 5 | 6 |
| low_07 | ✓ | ✓ | ✓ | ✓ | ||
| low_11 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| low_12 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| low_16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| low_18 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| low_21 | ✓ | ✓ | ✓ | ✓ | ✓ |
| Success check | All | low | mid | high | posture | Sub-goal | Collisions | Disturbed | Calls | Tokens |
|---|---|---|---|---|---|---|---|---|---|---|
| with | 98.7 | 100.0 | 100.0 | 100.0 | 90.0 | 99.6 | 9.3 | 1.0 | 51 | 99k |
| without | 86.7 | 96.7 | 85.0 | 80.0 | 70.0 | 92.0 | 8.2 | 0.7 | 58 | 128k |
| Control specification | Full loop | Single-Try | Open-Loop |
|---|---|---|---|
| Full (Inhabitant) | 19 (7.8) | 15 (30.4) | 0 (10.9) |
| Text prompts only | 4 (7.2) | 8 (178.3) | 0 (23.2) |
| HumanCLAW skills | 6 (8.4) | 6 (23.9) | 1 (20.4) |
| Full, no measurements | 17 (14.6) | ||
| Windows (frames covered) | 171 (73%) |
|---|---|
| Tracked without failure | 161 (94%) |
| Global MPJPE | 53.7 mm |
| Root-relative MPJPE | 40.5 mm |
| Acceleration error | 6.0 mm/frame 2 |
| Global MPJPE, walking | 52.8 mm |
| Global MPJPE, low postures | 110.8 mm |
Long-Horizon LLM Agents
Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.
See all 491 papers