Increasing Resilience of Smart Home Agents

Christopher Terrazas; Eduardo Cotilla-Sanchez

cs.MAOct 4, 2026

Increasing Resilience of Smart Home Agents

Authors: Christopher Terrazas, Eduardo Cotilla-Sanchez

Organizations: School of Electrical Engineering and Computer Science Oregon State University Corvallis, Oregon, USA

Abstract

Smart homes and smart devices are becoming more prevalent across millions of homes around the world. With the rise of AI, the smart home industry is quickly increasing its integration to manage common smart home tasks. However, existing work in large language models (LLMs) as agents within smart homes have shown minimal resilience due to limited environment scenarios or poor performance in complex tasks. We explore several strategies for LLMs as agents within the popular open-source software (OSS) smart home automation framework HomeAssistant to increase overall smart home resilience. Our approach combines traditional supervised learning techniques and optimized prompting as the core learning process. We use a diverse set of LLMs covering different levels of reasoning and costs in our optimization pipeline and evaluate their performance on a subset of the hardest tasks in a smart home benchmark. We include ReAct and Reflexion agentic paradigms and reveal how both provide marginal return on investment compared to fine-tuned LLMs for multi-device control within HomeAssistant but show promise in resilience tasks such as failure response.

Figures & tables

Panel of Figure 1 — Fig. 1: Reflexion architecture. The Actor (ReAct) interacts with HomeAssistant and writes to a shared trajectory $\mathcal{H}$ . After each action, the Evaluator scores $\mathcal{H}$ ; on a zero reward, the Self-Reflection module produces a verbal plan $\sigma_{i}$ that is appended to a bounded FIFO long-term memory ( $|M|\leq 3$ ). The contents of $M$ are appended back into the trajectory at every step, making prior reflections visible to the Actor on subsequent decisions within the same episode. All four components are implemented as custom DsPy modules.

Model	LLM Strategy
Name	Base	Opt	ReAct	Reflexion
gpt-oss-120b	86.7	88.6	85.4	85.0
gpt-5-mini	86.2	86.2	85.3	84.7
grok-4-1	86.3	89.9	88.0	87.8
deepseek-r1	79.0	85.4	77.7	76.6
qwen-2.5-72b	46.4	52.4	45.4	44.6

TABLE I: F1 Evaluation on HomeBench MultiDevice Tasks (%)

Model	LLM Strategy
Name	Base	Opt	ReAct	Reflexion
gpt-oss-120b	43.2 $\pm$ 8.2	45.2 $\pm$ 8.8	42.3 $\pm$ 8.7	47.1 $\pm$ 10.2
gpt-5-mini	43.8 $\pm$ 8.6	45.0 $\pm$ 8.6	44.7 $\pm$ 7.9	44.4 $\pm$ 8.8
grok-4-1	47.0 $\pm$ 9.6	47.7 $\pm$ 9.3	48.6 $\pm$ 10.0	46.1 $\pm$ 9.4
deepseek-r1	43.7 $\pm$ 8.4	45.0 $\pm$ 9.1	46.6 $\pm$ 9.1	41.1 $\pm$ 8.3
qwen-2.5-72b	41.2 $\pm$ 8.0	41.2 $\pm$ 8.0	41.3 $\pm$ 7.9	41.6 $\pm$ 8.4

TABLE II: F1 Evaluation on Resilience Dataset (%)

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 832 papers

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
Jun 1, 2026Kuan Li, Shuo Zhang, Huacan Wang +12LLM Agent Evaluation Internet of Things
Midea Group · Beijing University of Posts and Telecommunications · Donghua University +2
HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation
May 31, 2026Yi Gu, Huacan Wang, Shuo Zhang +10AI Agent Benchmarks LLM Agent Training
1Midea Group · 2Beijing University of Posts and Telecommunications · 3Donghua University +1
HearthNet: Edge Multi-Agent Orchestration for Smart Homes
Mar 16, 2026Zhonghao Zhan, Krinos Li, Yefan Zhang +1LLM Agent Orchestration Internet of Things
Imperial College London, London, UK · Independent Researcher, Seattle, USA

Jun 1, 2026J/K move · Enter open · S save

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

Kuan Li, Shuo Zhang, Huacan Wang, Fangzhou Yu+11

Midea Group · Beijing University of Posts and Telecommunications · Donghua University+2

LLM Agent Evaluation Internet of Things AI Agent Benchmarks

Open paper AskPDF arXiv

Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions. However, existing smart-home benchmarks often focus on static instruction-to-API mapping or limited simulations, failing to evaluate whether LLMs can reason, interact, and act reliably in realistic household scenarios. To address these limitations, we introduce SMH-Bench, a comprehensive benchmark for evaluating LLMs in smart-home environments. Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories. It further stratifies tasks across simple, medium and complex homes, ranging from small apartments to dense multi-room environments with 135 devices. Experiments show that although frontier LLMs achieve strong performance on explicit control and query tasks, they still exhibit significant weaknesses in automation task scheduling, ambiguity handling and personalized reasoning, especially as home complexity increases. We hope SMH-Bench will facilitate the development of more reliable, context-aware, and practically deployable smart-home agents.

Abstract

LLM Agent Reliability

Explore similar work

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation

HearthNet: Edge Multi-Agent Orchestration for Smart Homes