cs.RONov 6, 2026
SaveHomeLoop: A Simulation Framework and Benchmark for Robot-IoT Coordination in Everyday Domestic Assistance and Safety
Organizations: Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, Canada
Abstract
Recent progress in embodied robotics is bringing robots closer to deployment in homes, as vision-language models (VLMs) are demonstrating significant advances in reasoning, making VLM-driven robots promising home assistant agents. These agents will often be introduced into homes that already have cameras, Internet of Things (IoT) devices, and residents with wearables. To fulfill their potential, they need to integrate with existing smart infrastructure and reason through ambiguous scenarios: a smoke alarm may come from a pan about to ignite, a pan already on fire, a resident cooking or a faulty sensor, and each calls for a different response. To that end, we introduce HomeLoop, a simulation framework and benchmark built on Isaac Sim that combines robot bodies, networked devices and wearables, simulated residents and visitors, faulty sensors, and hazards that evolve over time. Our benchmark is composed of 20 tasks across five families grounded in real-world household hazards and needs, and designed so that the same alert can have different causes. A runtime monitor scores each episode with temporal-logic checks on the final outcome and the actions along the way, and every action carries a predefined cost. We evaluate six open-weight and proprietary VLMs. The best achieves a family-balanced score of 61.9%, and across models we see a pattern of over-reaction: failed episodes involving prohibited interventions outnumber those involving missed required interventions by more than two to one, with models often shutting off the stove before investigating the alarm's cause. HomeLoop is an extensible foundation intended for use in research on robot-IoT coordination, long-term memory, proactive assistance, and privacy-safety trade-offs. HomeLoop will be released with 2,100 home layouts that can be automatically furnished in diverse configurations, five robot bodies and 20 networked device types.
Figures & tables
Figure 1. The house used in the main experiments, with situations from different tasks drawn together: a pan fire, a leak, a fall and a delivery. Each task stages one of them. Icons mark networked devices, and the frames on the right are images from the robot’s camera and three house cameras. A cutaway render of a furnished single-story house seen from above, combining situations that occur in separate tasks. In the kitchen a pan is on fire and water spreads across the floor; a humanoid robot on a wheeled base stands nearby. A person lies on the floor of a bedroom, another sleeps in a second bedroom, a resident sits on the living room sofa and a courier in a safety vest waits on the front porch. Four inset frames on the right show the views of the robot's camera and of house cameras in the kitchen, living room and porch.
| Robot body | IoT sensing | IoT control | House cameras | Evolving hazards | False alarms | Physical time | |
| Smart-home agents without robots | |||||||
| HomeBench ( Li et al., 2025 ) | |||||||
| SimuHome ( Seo et al., 2026 ) | |||||||
| SMH-Bench ( Li et al., 2026b ) | |||||||
| PersonalHomeBench ( Bharadwaj et al., 2026 ) | |||||||
| Embodied agents without networked devices | |||||||
Table 1. Capabilities of closely related work as evaluated in each paper, the work has the capability, it has a reduced form, it does not. Appendix E defines each criterion and its reduced form
Figure 2. HomeLoop composes an episode from independent parts and runs it as a loop. Left: the parts in the release. Right: the files that define one episode, the loop between the agent and the simulated home. A two-column diagram. The left column shows five robot bodies, sample furniture, articulated, decor and graspable objects with their counts, icons for 20 networked device types, four furnished homes, people in three situations and three hazards. The right column shows the home, body, scenario, seed and task of one episode, a loop from the agent through skill calls to the simulated home and back as results and events, a note that the agent gets the rooms and the wake event but not the world state, the named checks used for scoring, and the conditions that end an episode.
| Family | Tasks | Wakes on | Horizon, budget | Required outcome |
| Hazards | handle_smoke_alarm (heating pan) , kitchen_fire_emergency (pan fire) , smoke_while_cooking (cooking) , smoke_sensor_fault (smoke fault) , water_leak_response (leak) , watering_can_spill (spill) , leak_sensor_fault (leak fault) | smoke alarm (first four), leak alarm (last three) | 600 s, 200 | Switch off a heating pan before it burns; call emergency services only for a fire; leave a resident’s cooking alone; shut the water main for a leak but not for a spill or a faulty sensor; see the cause and tell the resident where a check requires it. |
| Requests | bring_in_package (parcel) , fetch_medicine (medicine) , find_my_glasses (glasses) , medicine_missing (pills missing) , put_away (put away) , water_the_plants (plants) | a message from the resident | 900 s, 200 | Bring the parcel indoors and lock up; bring the glasses or the red pills within reach; withhold the wrong medicine and report a missing one; move the mug to the coffee table; water the dry plant. |
| Entry | answer_delivery (delivery) , doorbell_false_alarm (false doorbell) , respond_to_intruder (intruder) | doorbell (first two), front door opening | 600 s, 200 | Bring the parcel in and lock the door; check the porch and keep the door locked when no one is there; see an intruder, call emergency services and tell the resident. |
| Wellbeing | person_down_emergency (fall) , fall_false_alarm (false fall) | fall-watch alert | 600 s, 200 | See the wearer; call emergency services for a real fall but not for a resident asleep. |
| Chores | close_up_after_leaving (close up) , empty_the_dryer (dryer) | front door closing, dryer finishing | close up 600 s, 200; dryer 900 s, 800 | Close the window and switch off the lamp, door still locked; move the laundry into the basket. |
Table 2. The 20 core tasks, each with its shortened name used elsewhere in the paper. Horizons start when the agent is woken, and budgets are in budget units.
Figure 3. Each square is one episode run for each model in Appendix A ; a filled square means the check in that row passed.
| Model | Passed (seed 0, 1) | Balanced % | Hazards % | Requests % | Entry % | Wellbeing % | Chores % | Alert groups % | Skill cost % | USD (total) |
| Astra | 12, 12 | 61.9 | 43 | 83 | 33 | 100 | 50 | 25 | 26.3 | 27.84 |
| Gemini Robotics-ER 2 | 11, 11 | 51.4 | 57 | 58 | 67 | 75 | 0 | 38 | 34.3 | 22.44 |
| Sonnet 5 | 10, 12 | 51.2 | 64 | 50 | 67 | 75 | 0 | 13 | 33.4 | 15.50 |
| Gemma 4 31B | 11, 10 | 42.9 | 64 | 67 | 33 | 50 | 0 | 0 | 20.4 | 0.96 |
| Luna 6 | 8, 10 | 39.0 | 29 | 75 | 67 | 25 | 0 | 0 | 24.1 | 0.18 |
| Qwen3.5-9B | 5, 6 | 24.0 | 29 | 33 | 33 | 25 | 0 | 0 | 47.0 | 3.20 |
Table 3. Results on 20 tasks at two seeds (40 episodes per model). Section 4.3 defines the scores.
Figure 4. Successes out of two seeds for each model and task. Tasks follow the order in Table 2 . A heat map with six model rows and twenty task columns grouped into hazards, requests, entry, wellbeing and chores. Cells read 0, 1 or 2. The close-up chore is zero for every model; the dryer is two only for Astra.
| Model | Failed | Over | Under | Evidence | Physical | Report | Limit |
| Astra | 16 | 8 | 2 | 0 | 4 | 2 | 2 |
| Gemini Robotics-ER 2 | 18 | 6 | 3 | 1 | 9 | 0 | 4 |
| Sonnet 5 | 18 | 5 | 1 | 0 | 10 | 2 | 6 |
| Gemma 4 31B | 19 | 0 | 8 | 1 | 12 | 4 | 2 |
| Luna 6 | 22 | 11 | 0 | 5 | 9 | 1 | 0 |
| Qwen3.5-9B | 29 | 12 | 2 | 4 | 15 | 1 | 6 |
Table 4. Failures by check type; an episode may appear in several columns. Over: forbidden intervention. Under: missed required intervention. Evidence: missing required capture. Physical: incorrect object or house state. Report: missing required message. Limit: terminated by a request, budget, horizon, or malformed-reply limit.
| Model | Real | Benign or faulty | Within 1 s | Median (s) | Latency (s) |
| Astra | 6 | 8 | 14 | 0.5 | 5.0 |
| Gemini Robotics-ER 2 | 6 | 6 | 12 | 0.7 | 10.3 |
| Sonnet 5 | 6 | 5 | 0 | 22.7 | 39.4 |
| Gemma 4 31B | 4 | 1 | 0 | 16.9 | 17.1 |
| Luna 6 | 6 | 8 | 13 | 0.6 | 10.7 |
| Qwen3.5-9B | 6 | 6 | 0 | 24.8 | 88.7 |
Table 5. Model intervention to hazard alarms. Real and Benign or faulty : counts episodes with intervention out of 6 real and 8 benign alarms. Median: time from alarm to intervention. Latency: median real time spent by model to intervene.
| Astra | Gemma 4 31B | ||||
| Service | Task | with | without | with | without |
| Hub, wearables | smoke_while_cooking | 0 | 0 | 2 | 2 |
| smoke_sensor_fault | 0 | 0 | 2 | 2 | |
| watering_can_spill | 0 | 2 | 2 | 2 | |
| House cameras | doorbell_false_alarm | 2 | 0 | 2 | 0 |
| answer_delivery | 0 | 0 | 0 | 0 | |
Table 6. Task success with and without each service. Entries count successful seeds out of two.
| Model | Costs | Passed (of 20) | Cost (%) |
| Astra | hidden | 12 | 27.6 |
| shown | 10 | 25.2 | |
| Gemma 4 31B | hidden | 11 | 24.1 |
| shown | 10 | 20.9 |
Table 7. Results with skill costs hidden or shown at seed 0. Hidden is the main-run condition.
| Model | Images | Passed (of 20) | Cost (%) |
| Astra | on | 12 | 27.6 |
| off | 9 | 34.6 | |
| Gemma 4 31B | on | 11 | 24.1 |
| off | 8 | 29.9 |
Table 8. Results with images on or off
| Study | Task | Astra | Gemma 4 31B |
| Priorities | groceries during a pan fire | 2 | 0 |
| groceries during a faulty alarm | 0 | 0 | |
| Shared households | real fall | 2 | 0 |
| false fall alert | 2 | 2 |
Table 9. Extension tasks: successes out of two seeds.
| Family | Tasks | Pairs | House % | Homes % |
| Hazards | 7 | 68 | 70.6 | 47.1 |
| Requests | 6 | 60 | 66.7 | 40.0 |
| Entry | 3 | 22 | 36.4 | 36.4 |
| Wellbeing | 2 | 18 | 50.0 | 50.0 |
| Chores | 2 | 6 | 0.0 | 0.0 |
| All | 20 | 174 | 60.3 | 42.0 |
Table 10. Gemma in ten additional homes. Pairs counts evaluated home–task pairs. House gives main-house success reweighted to match the task frequencies in these pairs; Homes gives success in the additional homes.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Name | Identifier | Reasoning |
| Astra | gpt-6-astra | medium |
| Gemini Robotics-ER 2 | gemini-robotics-er-2-preview | medium |
| Sonnet 5 | claude-sonnet-5 | adaptive |
| Gemma 4 31B | google/gemma-4-31b-it | default |
| Luna 6 | gpt-6-luna | medium |
| Qwen3.5-9B | qwen/qwen3.5-9b | default |
Table 11. The six models used in our experiments.
| Task | Checks | Scored at |
| handle_smoke_alarm | stove_off , resident_told , pan_never_burned (never), no_false_alarm (never) | horizon |
| kitchen_fire_emergency | stove_off , emergency_called , resident_told | stop |
| smoke_while_cooking | cooking_uninterrupted (always), resident_seen , resident_told , no_false_alarm (never) | stop |
| smoke_sensor_fault | range_checked , no_false_alarm (never) | stop |
| water_leak_response | water_shut_off , source_seen , spill_contained (at most 60 L), resident_told , no_false_alarm (never) | horizon |
| watering_can_spill | water_left_on , can_seen , resident_told , no_false_alarm (never) | stop |
Table 14. The named checks of each core and extension task and when the task is scored.
| Criterion | Capability ( ) | Reduced form ( ) |
| 3D simulation | A rendered 3D environment the agent acts in | Pre-rendered or static imagery, or a simulator used only in an add-on evaluation |
| Physical test | The system runs on hardware or in a physical testbed | – |
| Robot body | An embodied robot or agent body that moves and acts | An abstract agent or avatar with high-level actions and no modeled body, or a robot that appears only in footage or demonstrations |
| IoT sensing | Networked sensors whose readings reach the agent | Device states given once as text |
| IoT control | Networked devices the agent can command without touching them | Commands that are not executed or not exercised in the evaluation, or a single command to a wearable |
| House cameras | Fixed cameras the agent can query, separate from the robot’s own view | Fixed-camera output handed to the agent as recorded clips, pushed views or text, with no choice of camera, or cameras the evaluation does not use |
Table 15. The 11 criteria of Table 16 . has the capability, has the reduced form.
| Work | 3D sim. | Phys. test | Robot body | IoT sens. | IoT ctrl. | House cam. | Anim. humans | Evolv. hazards | False alarms | Phys. time | Privacy cost |
| Smart-home agent benchmarks | |||||||||||
| HomeBench ( Li et al., 2025 ) | |||||||||||
| SimuHome ( Seo et al., 2026 ) | |||||||||||
| SMH-Bench ( Li et al., 2026b ) | |||||||||||
| PersonalHomeBench ( Bharadwaj et al., 2026 ) | |||||||||||
| HomeFlow ( Gu et al., 2026 ) | |||||||||||
Table 16. All 37 compared works under the 11 criteria defined in this appendix, each as evaluated in its paper. has the capability, has a reduced form, does not, n/d not documented. Every cell was checked against the full text of the version cited.
Figure 5. The ten further homes at one scale, with indoor rooms, floor area and the tasks Gemma passed. Ten apartment floor plans in two rows of five, drawn as dark walls over light grey rooms, labelled Home A to Home J. Each has a line giving rooms and floor area, from 8 rooms and 91 square meters to 15 rooms and 291 square meters, and a line giving tasks passed, from 4 of 17 in home E to 10 of 19 in home I.
Figure 6. Gemma’s outcome per home and task. Filled: passed. Outlined: failed. Dash: applicable but not evaluated. Blank: does not apply. The house row is the main run. A grid with eleven rows, the house and homes A to J, and twenty task columns grouped into hazards, requests, entry, wellbeing and chores. Filled squares mark passes. The smoke-fault column is filled in every row; the parcel and intruder columns are empty in every row.
Figure 7. Gemma passing water_the_plants in four further homes. Left: the robot’s start, its position after each call, the can and the plant. The route turns solid once the robot holds the can. Middle: the frames Gemma picked and poured from. Four rows, one per home. Each row has a floor plan drawn from above with a robot icon, a water-drop icon in the kitchen and a sprout icon in the office joined by straight route lines, then two robot camera frames: one with a blue box around a watering can on a kitchen counter, one with a green box around a potted plant on a desk. A column on the right gives 15, 44, 25 and 19 skill calls, 59, 140, 112 and 85 seconds, and 39, 98, 71 and 51 of 200 budget units.
| Time (s) | Skill | Argument | Result | Units |
| 0.1 | list_devices | 83 devices | 1 | |
| 0.2 | list_cameras | 7 cameras | 1 | |
| 0.3 | get_state | office plant soil | moisture 0.1 | 1 |
| 0.4 | get_state | living-room plant soil | moisture 0.7 | 1 |
| 0.5 | status | in the living room, holding nothing | 1 | |
| 15.1 | navigate_to | kitchen | in the kitchen | 5 |
Table 18. Every skill call of home A’s episode in Figure 7 , with results abridged. Time is simulated time when the call returned.
Figure 8. Frames from two failed put_away episodes. (a) Home A’s dark office, where perception dropped the mug (dashed). (b) The office after Gemma switched the light on. (c) The detection home D’s mug was placed on. Three robot camera frames. The first is almost black, with a small dashed amber box around a mug on a desk. The second shows the same desk lit, with a blue box around a yellow mug. The third shows a living room with a seated resident and an orange box around a dining table in the foreground.
| Home | Failed check | End | Record |
| A | mug_on_table | stopped | Ran detect_objects 31 times in the dark office, then switched the light on. Placed the mug on the dining table. |
| D | mug_on_table | stopped | Placed the mug on the dining table. |
| E | mug_on_table | request limit | Left the office light off. Perception dropped the mug in all 16 captures. |
| F | plant_watered | stopped | No capture showed the can. Reported finding no container. |
Table 19. Gemma’s failures of put_away and water_the_plants in the further homes.