Recent progress in embodied robotics is bringing robots closer to deployment in homes, as vision-language models (VLMs) are demonstrating significant advances in reasoning, making VLM-driven robots promising home assistant agents. These agents will often be introduced into homes that already have cameras, Internet of Things (IoT) devices, and residents with wearables. To fulfill their potential, they need to integrate with existing smart infrastructure and reason through ambiguous scenarios: a smoke alarm may come from a pan about to ignite, a pan already on fire, a resident cooking or a faulty sensor, and each calls for a different response. To that end, we introduce HomeLoop, a simulation framework and benchmark built on Isaac Sim that combines robot bodies, networked devices and wearables, simulated residents and visitors, faulty sensors, and hazards that evolve over time. Our benchmark is composed of 20 tasks across five families grounded in real-world household hazards and needs, and designed so that the same alert can have different causes. A runtime monitor scores each episode with temporal-logic checks on the final outcome and the actions along the way, and every action carries a predefined cost. We evaluate six open-weight and proprietary VLMs. The best achieves a family-balanced score of 61.9%, and across models we see a pattern of over-reaction: failed episodes involving prohibited interventions outnumber those involving missed required interventions by more than two to one, with models often shutting off the stove before investigating the alarm's cause. HomeLoop is an extensible foundation intended for use in research on robot-IoT coordination, long-term memory, proactive assistance, and privacy-safety trade-offs. HomeLoop will be released with 2,100 home layouts that can be automatically furnished in diverse configurations, five robot bodies and 20 networked device types.
Figures & tables
Figure 1. The house used in the main experiments, with situations from different tasks drawn together: a pan fire, a leak, a fall and a delivery. Each task stages one of them. Icons mark networked devices, and the frames on the right are images from the robot’s camera and three house cameras. A cutaway render of a furnished single-story house seen from above, combining situations that occur in separate tasks. In the kitchen a pan is on fire and water spreads across the floor; a humanoid robot on a wheeled base stands nearby. A person lies on the floor of a bedroom, another sleeps in a second bedroom, a resident sits on the living room sofa and a courier in a safety vest waits on the front porch. Four inset frames on the right show the views of the robot's camera and of house cameras in the kitchen, living room and porch.
Robot body
IoT sensing
IoT control
House cameras
Evolving hazards
False alarms
Physical time
Smart-home agents without robots
HomeBench ( Li et al., 2025 )
SimuHome ( Seo et al., 2026 )
SMH-Bench ( Li et al., 2026b )
PersonalHomeBench ( Bharadwaj et al., 2026 )
Embodied agents without networked devices
Table 1. Capabilities of closely related work as evaluated in each paper, the work has the capability, it has a reduced form, it does not. Appendix E defines each criterion and its reduced form
Figure 2. HomeLoop composes an episode from independent parts and runs it as a loop. Left: the parts in the release. Right: the files that define one episode, the loop between the agent and the simulated home. A two-column diagram. The left column shows five robot bodies, sample furniture, articulated, decor and graspable objects with their counts, icons for 20 networked device types, four furnished homes, people in three situations and three hazards. The right column shows the home, body, scenario, seed and task of one episode, a loop from the agent through skill calls to the simulated home and back as results and events, a note that the agent gets the rooms and the wake event but not the world state, the named checks used for scoring, and the conditions that end an episode.
Switch off a heating pan before it burns; call emergency services only for a fire; leave a resident’s cooking alone; shut the water main for a leak but not for a spill or a faulty sensor; see the cause and tell the resident where a check requires it.
Bring the parcel indoors and lock up; bring the glasses or the red pills within reach; withhold the wrong medicine and report a missing one; move the mug to the coffee table; water the dry plant.
Bring the parcel in and lock the door; check the porch and keep the door locked when no one is there; see an intruder, call emergency services and tell the resident.
Close the window and switch off the lamp, door still locked; move the laundry into the basket.
Table 2. The 20 core tasks, each with its shortened name used elsewhere in the paper. Horizons start when the agent is woken, and budgets are in budget units.
Figure 3. Each square is one episode run for each model in Appendix A ; a filled square means the check in that row passed.
Model
Passed (seed 0, 1)
Balanced %
Hazards %
Requests %
Entry %
Wellbeing %
Chores %
Alert groups %
Skill cost %
USD (total)
Astra
12, 12
61.9
43
83
33
100
50
25
26.3
27.84
Gemini Robotics-ER 2
11, 11
51.4
57
58
67
75
0
38
34.3
22.44
Sonnet 5
10, 12
51.2
64
50
67
75
0
13
33.4
15.50
Gemma 4 31B
11, 10
42.9
64
67
33
50
0
0
20.4
0.96
Luna 6
8, 10
39.0
29
75
67
25
0
0
24.1
0.18
Qwen3.5-9B
5, 6
24.0
29
33
33
25
0
0
47.0
3.20
Table 3. Results on 20 tasks at two seeds (40 episodes per model). Section 4.3 defines the scores.
Figure 4. Successes out of two seeds for each model and task. Tasks follow the order in Table 2 . A heat map with six model rows and twenty task columns grouped into hazards, requests, entry, wellbeing and chores. Cells read 0, 1 or 2. The close-up chore is zero for every model; the dryer is two only for Astra.
Model
Failed
Over
Under
Evidence
Physical
Report
Limit
Astra
16
8
2
0
4
2
2
Gemini Robotics-ER 2
18
6
3
1
9
0
4
Sonnet 5
18
5
1
0
10
2
6
Gemma 4 31B
19
0
8
1
12
4
2
Luna 6
22
11
0
5
9
1
0
Qwen3.5-9B
29
12
2
4
15
1
6
Table 4. Failures by check type; an episode may appear in several columns. Over: forbidden intervention. Under: missed required intervention. Evidence: missing required capture. Physical: incorrect object or house state. Report: missing required message. Limit: terminated by a request, budget, horizon, or malformed-reply limit.
Model
Real
Benign or faulty
Within 1 s
Median (s)
Latency (s)
Astra
6
8
14
0.5
5.0
Gemini Robotics-ER 2
6
6
12
0.7
10.3
Sonnet 5
6
5
0
22.7
39.4
Gemma 4 31B
4
1
0
16.9
17.1
Luna 6
6
8
13
0.6
10.7
Qwen3.5-9B
6
6
0
24.8
88.7
Table 5. Model intervention to hazard alarms. Real and Benign or faulty : counts episodes with intervention out of 6 real and 8 benign alarms. Median: time from alarm to intervention. Latency: median real time spent by model to intervene.
Astra
Gemma 4 31B
Service
Task
with
without
with
without
Hub, wearables
smoke_while_cooking
0
0
2
2
smoke_sensor_fault
0
0
2
2
watering_can_spill
0
2
2
2
House cameras
doorbell_false_alarm
2
0
2
0
answer_delivery
0
0
0
0
Table 6. Task success with and without each service. Entries count successful seeds out of two.
Model
Costs
Passed (of 20)
Cost (%)
Astra
hidden
12
27.6
shown
10
25.2
Gemma 4 31B
hidden
11
24.1
shown
10
20.9
Table 7. Results with skill costs hidden or shown at seed 0. Hidden is the main-run condition.
Model
Images
Passed (of 20)
Cost (%)
Astra
on
12
27.6
off
9
34.6
Gemma 4 31B
on
11
24.1
off
8
29.9
Table 8. Results with images on or off
Study
Task
Astra
Gemma 4 31B
Priorities
groceries during a pan fire
2
0
groceries during a faulty alarm
0
0
Shared households
real fall
2
0
false fall alert
2
2
Table 9. Extension tasks: successes out of two seeds.
Family
Tasks
Pairs
House %
Homes %
Hazards
7
68
70.6
47.1
Requests
6
60
66.7
40.0
Entry
3
22
36.4
36.4
Wellbeing
2
18
50.0
50.0
Chores
2
6
0.0
0.0
All
20
174
60.3
42.0
Table 10. Gemma in ten additional homes. Pairs counts evaluated home–task pairs. House gives main-house success reweighted to match the task frequencies in these pairs; Homes gives success in the additional homes.
Table 14. The named checks of each core and extension task and when the task is scored.
Criterion
Capability ( )
Reduced form ( )
3D simulation
A rendered 3D environment the agent acts in
Pre-rendered or static imagery, or a simulator used only in an add-on evaluation
Physical test
The system runs on hardware or in a physical testbed
–
Robot body
An embodied robot or agent body that moves and acts
An abstract agent or avatar with high-level actions and no modeled body, or a robot that appears only in footage or demonstrations
IoT sensing
Networked sensors whose readings reach the agent
Device states given once as text
IoT control
Networked devices the agent can command without touching them
Commands that are not executed or not exercised in the evaluation, or a single command to a wearable
House cameras
Fixed cameras the agent can query, separate from the robot’s own view
Fixed-camera output handed to the agent as recorded clips, pushed views or text, with no choice of camera, or cameras the evaluation does not use
Appendix
Table 15. The 11 criteria of Table 16 . has the capability, has the reduced form.
Work
3D sim.
Phys. test
Robot body
IoT sens.
IoT ctrl.
House cam.
Anim. humans
Evolv. hazards
False alarms
Phys. time
Privacy cost
Smart-home agent benchmarks
HomeBench ( Li et al., 2025 )
SimuHome ( Seo et al., 2026 )
SMH-Bench ( Li et al., 2026b )
PersonalHomeBench ( Bharadwaj et al., 2026 )
HomeFlow ( Gu et al., 2026 )
Appendix
Table 16. All 37 compared works under the 11 criteria defined in this appendix, each as evaluated in its paper. has the capability, has a reduced form, does not, n/d not documented. Every cell was checked against the full text of the version cited.
Figure 5. The ten further homes at one scale, with indoor rooms, floor area and the tasks Gemma passed. Ten apartment floor plans in two rows of five, drawn as dark walls over light grey rooms, labelled Home A to Home J. Each has a line giving rooms and floor area, from 8 rooms and 91 square meters to 15 rooms and 291 square meters, and a line giving tasks passed, from 4 of 17 in home E to 10 of 19 in home I.
Figure 6. Gemma’s outcome per home and task. Filled: passed. Outlined: failed. Dash: applicable but not evaluated. Blank: does not apply. The house row is the main run. A grid with eleven rows, the house and homes A to J, and twenty task columns grouped into hazards, requests, entry, wellbeing and chores. Filled squares mark passes. The smoke-fault column is filled in every row; the parcel and intruder columns are empty in every row.
Figure 7. Gemma passing water_the_plants in four further homes. Left: the robot’s start, its position after each call, the can and the plant. The route turns solid once the robot holds the can. Middle: the frames Gemma picked and poured from. Four rows, one per home. Each row has a floor plan drawn from above with a robot icon, a water-drop icon in the kitchen and a sprout icon in the office joined by straight route lines, then two robot camera frames: one with a blue box around a watering can on a kitchen counter, one with a green box around a potted plant on a desk. A column on the right gives 15, 44, 25 and 19 skill calls, 59, 140, 112 and 85 seconds, and 39, 98, 71 and 51 of 200 budget units.
Time (s)
Skill
Argument
Result
Units
0.1
list_devices
83 devices
1
0.2
list_cameras
7 cameras
1
0.3
get_state
office plant soil
moisture 0.1
1
0.4
get_state
living-room plant soil
moisture 0.7
1
0.5
status
in the living room, holding nothing
1
15.1
navigate_to
kitchen
in the kitchen
5
Appendix
Table 18. Every skill call of home A’s episode in Figure 7 , with results abridged. Time is simulated time when the call returned.
Figure 8. Frames from two failed put_away episodes. (a) Home A’s dark office, where perception dropped the mug (dashed). (b) The office after Gemma switched the light on. (c) The detection home D’s mug was placed on. Three robot camera frames. The first is almost black, with a small dashed amber box around a mug on a desk. The second shows the same desk lit, with a blue box around a yellow mug. The third shows a living room with a seated resident and an orange box around a dining table in the foreground.
Home
Failed check
End
Record
A
mug_on_table
stopped
Ran detect_objects 31 times in the dark office, then switched the light on. Placed the mug on the dining table.
D
mug_on_table
stopped
Placed the mug on the dining table.
E
mug_on_table
request limit
Left the office light off. Perception dropped the mug in all 16 captures.
F
plant_watered
stopped
No capture showed the can. Reported finding no container.
Appendix
Table 19. Gemma’s failures of put_away and water_the_plants in the further homes.
Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions. However, existing smart-home benchmarks often focus on static instruction-to-API mapping or limited simulations, failing to evaluate whether LLMs can reason, interact, and act reliably in realistic household scenarios. To address these limitations, we introduce SMH-Bench, a comprehensive benchmark for evaluating LLMs in smart-home environments. Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories. It further stratifies tasks across simple, medium and complex homes, ranging from small apartments to dense multi-room environments with 135 devices. Experiments show that although frontier LLMs achieve strong performance on explicit control and query tasks, they still exhibit significant weaknesses in automation task scheduling, ambiguity handling and personalized reasoning, especially as home complexity increases. We hope SMH-Bench will facilitate the development of more reliable, context-aware, and practically deployable smart-home agents.
Kuan Li, Shuo Zhang, Huacan Wang +12
Midea Group · Beijing University of Posts and Telecommunications · Donghua University +2
Large language model agents are moving beyond text-only interaction toward physical-world control, with smart homes as a representative domain. Real domestic interaction requires understanding ambiguous intents, operating in dynamic environments, and performing multi-turn reasoning. However, existing methods struggle to generate high-quality training data for smart home agents. We propose HomeFlow, a verifiable data flywheel for this domain. HomeFlow uses HomeEnv as a unified simulation environment and HomeMaker to procedurally generate diverse home settings. Subsequently, Blueprint compiles open-ended user intents into executable state-based success conditions, while MCTS-Flow synthesizes diverse, verifiable multi-turn trajectories through environment-guided tree search. We then optimize the agents via supervised fine-tuning and step-wise RLVE, which facilitates iterative improvement through authentic physical feedback. We further construct SmartHome-Bench to evaluate the agent across various smart home tasks. On this benchmark, HomeFlow-RL-4B and HomeFlow-RL-8B achieve task success rates of 84.60% and 87.03%. It is worth noting that HomeFlow-RL-8B even surpasses the leading GPT-5.5 by 1.23 percentage points.
Yi Gu, Huacan Wang, Shuo Zhang +10
1Midea Group · 2Beijing University of Posts and Telecommunications · 3Donghua University +1
The rapid evolution of embodied agents has accelerated the deployment of household robots in real-world environments. However, unlike structured industrial settings, household spaces introduce unpredictable safety risks, where system limitations such as perception latency and lack of common sense knowledge can lead to dangerous errors. Current safety evaluations, often restricted to static images, text, or general hazards, fail to adequately benchmark dynamic unsafe action detection in these specific contexts. To bridge this gap, we introduce HomeSafe-Bench, a challenging benchmark designed to evaluate Vision-Language Models (VLMs) on unsafe action detection in household scenarios. HomeSafe-Bench is contrusted via a hybrid pipeline combining physical simulation with advanced video generation and features 438 diverse cases across six functional areas with fine-grained multidimensional annotations. Beyond benchmarking, we propose Hierarchical Dual-Brain Guard for Household Safety (HD-Guard), a hierarchical streaming architecture for real-time safety monitoring. HD-Guard coordinates a lightweight FastBrain for continuous high-frequency screening with an asynchronous large-scale SlowBrain for deep multimodal reasoning, effectively balancing inference efficiency with detection accuracy. Evaluations demonstrate that HD-Guard achieves a superior trade-off between latency and performance, while our analysis identifies critical bottlenecks in current VLM-based safety detection.
Jiayue Pu, Zhongxiang Sun, Zilu Zhang +2
Gaoling School of Artificial Intelligence, Renmin University of China · University of Chinese Academy of Sciences · Beijing University of Posts and Telecommunications