cs.LGFeb 24, 2026

SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts

Authors: Qingsong Zou, Zhi Yan, Zhiyao Xu, Kuofeng Gao, Jingyu Xiao, Yong Jiang

Organizations: Tsinghua Shenzhen International Graduate School Shenzhen, China · Jilin University Jilin, China · The Chinese University of Hong Kong Hong Kong, China · Pengcheng Laboratory Shenzhen, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Due to the strong context-awareness capabilities demonstrated by large language models (LLMs), recent research has begun exploring their integration into smart home assistants to help users manage and adjust their living environments. While LLMs have been shown to effectively understand user needs and provide appropriate responses, most existing studies primarily focus on interpreting and executing user behaviors or instructions. However, a critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state. This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions. To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench, which is the first smart home dataset designed for LLMs, containing both normal and anomalous device states as well as normal and anomalous device state transition contexts. We evaluate 13 mainstream LLMs on this benchmark. The experimental results show that most state-of-the-art models cannot achieve good anomaly detection performance. For example, Claude-Sonnet-4.5 achieves only 66.1% detection accuracy on context-independent anomaly categories, and performs even worse on context-dependent anomalies, with an accuracy of only 57.8%. More experimental results suggest that next-generation LLM-based smart home assistants are still far from being able to effectively detect and handle anomalous conditions in the smart home environment. Our dataset is publicly available at https://github.com/horizonsinzqs/SmartBench.

Figures & tables

Figure 1. Examples of anomaly conditions in a smart home.
Figure 2. Dataset collection process of SmartBench .
Figure 3. Anomaly types and percentages of SmartBench .
StatisticsDescriptionStatisticsDescription
Time of Daymorning, midday, afternoon, nightRoom typesbasement, bathroom, bedroom, entrance, garage, hallway, kitchen, living_room
Seasonspring, summer, fall, winterEnvironment Parametertemperature, humidity, light, motion, weather, smoke, gas, moisture day_of_week, is_weekend
#Occupancy29#Duration15
#Device155#State274
#CI Normal1000#CD Normal1200
#CI Anom1000#CD Anom1200
Table 1. General statistics of SmartBench .
 Context-IndenpendentContext-Dependent
PrecisionRecallF1 scoreFPRAL ScorePrecsionRecallF1 scoreFPRAL Score
gemini-374.2%85.2%79.3%29.7%0.49157.4%79.8%66.8%59.2%0.347
gemini-2.564.5%85.6%73.5%47.2%0.39753.8%91.0%67.6%78.2%0.365
claude-4.563.9%74.0%68.6%41.8%0.31959.6%59.0%59.3%40.0%0.257
claude-473.8%50.7%60.1%18.0%0.23267.3%44.5%53.6%21.7%0.247
deepseek-r175.8%68.5%72.0%21.9%0.36552.2%83.7%64.3%76.5%0.261
Table 2. The main results of different LLMs on SmartBench . Bold indicates the best performance.
Figure 4. Performance of LLMs on different anomaly tasks.
Figure 5. Anomaly attribution capability of LLMs.
 Context-IndependentContext-Dependent
PRF1PRF1
qwen-8b34.3%24.0%28.2%47.6%41.6%44.4%
qwen-32b48.8%80.0%60.6%45.2%79.1%57.5%
llama-8b30.8%8.0%12.7%50.0%16.7%25.0%
llama-70b42.9%18.0%25.4%0.0%0.0%0.0%
llama-405b57.1%32.0%41.0%75.0%37.5%50.0%
Table 3. Performance across models of different sizes.
 AccuracyF1 scoreToken
RawCCRawCCRawCC
claude-4.560.0%60.0%66.7%66.7%5.58M2.18M
claude-462.0%76.0%64.2%77.8%5.58M2.17M
deepseek-r164.0%68.0%59.1%63.6%5.40M2.16M
deepseek-v346.0%44.0%62.9%53.3%5.33M2.12M
gemini-346.0%50.0%62.0%64.8%10.90M3.12M
Table 4. The performance of LLMs on compressed samples.
Figure 6. The effects of ICL on LLMs’ performance.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7. An example of the context-independent sample.
Figure 8. An example of the context-dependent sample.
Figure 9. The prompt used to evaluate the explanations for anomalies.
Figure 10. The Prompt Used for Sample Compliance Validation.
#DeviceState#DeviceState
1‘alarm_panel‘ARMED, ARMED_AWAY, DISARMED, TRIGGERED79‘kitchen_smart_plug‘OFF, ON
2‘alarm_panel_connectivity‘OFFLINE80‘kitchen_stove‘OFF, ON
3‘backyard_light_connectivity‘OFFLINE, ONLINE81‘kitchen_stove_state_log‘PERSISTED:ON
4‘basement_circuit_breaker‘CHECK82‘kitchen_toaster‘OFF, ON
5‘basement_hvac‘COOLING, HEATING, OFF83‘kitchen_under_cabinet_light‘OFF, ON
6‘basement_light‘OFF, ON84‘living_room_blinds‘OPEN
Appendix
Table 5. Devices and device states in SmartBench .
Anomaly TypeDescriptionSubtype
device_malfunctionDevice or sensor malfunction, offline status, or abnormal reporting, or inconsistencies between “control/display status” and telemetry readings, physical location, or other supporting evidence.actuator_position_anomaly, alarm_false_trigger, camera_offline, configuration_conflict, invalid_categorical_state, leak_sensor_false_positive, leak_sensor_suspicious, lock_door_state_inconsistent, sensor_unavailable, telemetry_state_conflict
safety_hazardHigh-risk safety events or protection failures related to smoke, gas, water leakage, door/window security, or surveillance, which may lead to fire, explosion, intrusion, or property loss.alarm_sensor_state_conflict, garage_access_risk, gas_detected, open_entry_while_away, security_disabled_with_activity, security_system_disabled, security_system_disabled_with_activity, smoke_detected, smoke_false_alarm_or_sensor_fault, surveillance_disabled, surveillance_disabled_with_activity, unlocked_entry_while_away, unlocked_entry_with_activity, water_leak_detected, window_left_open_in_bad_weather
physical_violationOut-of-range, missing, or improperly formatted values, or values that contradict basic physical laws or environmental facts (e.g., humidity readings inconsistent with signs of leakage, or abnormal temperature differences between adjacent rooms).humidity_invalid_value, humidity_out_of_range, humidity_physics_conflict, temperature_invalid_value, temperature_out_of_range, temperature_spatial_inconsistency, temperature_unrealistic_indoor, thermal_distribution_anomaly
compound_anomalyMultiple high-risk signals occur concurrently within the same time window, forming compound risks such as fire, intrusion, or utility-related hazards (with overlapping risks that are harder to interpret).fire_signature, intrusion_signature, multiple_safety_alerts, security_posture_failure, water_electric_risk_signature
environmental_conflictThe on/off status, mode, or target temperature of the HVAC system is inconsistent with ambient conditions such as indoor temperature or open-window ventilation, leading to conflicting control objectives or unreasonable adjustment directions.cooling_target_conflict, cooling_temperature_conflict, heating_target_conflict, heating_temperature_conflict, hvac_not_responding, hvac_running_while_off, hvac_status_mode_mismatch, hvac_window_conflict
occupancy_mismatchOccupancy profiling (e.g., at-home or away status) conflicts with activity evidence such as motion sensors, access control logs, or power consumption patterns, indicating potential labeling errors or suggesting abnormal behavior or intrusion risks.activity_signature_with_no_occupants, device_active_while_away, motion_with_no_occupants, multi_room_activity_with_no_occupants
Appendix
Table 6. Context-Independent Anomaly Types
Anomaly TypeDescriptionSubtype
device_malfunctionDevice or sensor failure, disconnection, or performance degradation, manifested as loss of controllability, lack of reporting, or reported readings/states that contradict other sensor data or power consumption patterns observed during the same time period.device_failure, parameter_drift, parameter_spike, sensor_stuck, sensor_drift, thermostat_failure, device_aging, component_cascade_failure, predictive_failure_risk
causal_temporalAnomalous causal and temporal relationships between events, such as expected events not occurring, events happening out of order, significantly delayed responses, or durations/frequencies deviating from normal patterns.causal_break, causal_delay, duration_anomaly, sequence_violation, timing_anomaly, periodicity_anomaly, frequency_anomaly, period_anomaly
statistical_anomalyThe data significantly deviates from historical patterns or similar scenarios in terms of overall level, fluctuation amplitude, or the synchronization between key indicators.correlation_anomaly, baseline_deviation, point_anomaly, contextual_anomaly, collective_anomaly, spurious_event, distribution_shift, apparent_anomaly_mostly_seasonal
safety_hazardHigh-risk safety incidents related to water leakage, intrusion, security breaches, device compromise, or electricity theft, which may pose direct threats to personal or property safety.water_leak, hidden_water_leak, collective_security_trigger, security_issue, unauthorized_entry, progressive_device_compromise, power_theft_upstream
climate_anomalyAnomalies in HVAC system cooling/heating/ventilation and zone control, such as malfunctions, overheating, ventilation issues, mode conflicts, or imbalanced zone regulation.hvac_failure, excess_heat, ventilation_anomaly, hvac_mode_conflict, hvac_zone_conflict, hvac_zone_imbalance
network_disconnectAnomalies in network links, gateways, or integration channels, leading to communication interruptions/fluctuations, state desynchronization, or cross-zone synchronization failures.network_failure, hub_offline, wireless_interference, network_topology_anomaly, state_sync_failure, zone_sync_failure, integration_failure, sensor_fusion_anomaly
Appendix
Table 7. Context-Dependent Anomaly Types
More in this topic

Anomaly Detection

Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.

See all 652 papers

Explore similar work

CardsList
Jun 1, 2026

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions. However, existing smart-home benchmarks often focus on static instruction-to-API mapping or limited simulations, failing to evaluate whether LLMs can reason, interact, and act reliably in realistic household scenarios. To address these limitations, we introduce SMH-Bench, a comprehensive benchmark for evaluating LLMs in smart-home environments. Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories. It further stratifies tasks across simple, medium and complex homes, ranging from small apartments to dense multi-room environments with 135 devices. Experiments show that although frontier LLMs achieve strong performance on explicit control and query tasks, they still exhibit significant weaknesses in automation task scheduling, ambiguity handling and personalized reasoning, especially as home complexity increases. We hope SMH-Bench will facilitate the development of more reliable, context-aware, and practically deployable smart-home agents.

Kuan Li, Shuo Zhang, Huacan Wang +12

Midea Group · Beijing University of Posts and Telecommunications · Donghua University +2

LLM Agent EvaluationInternet of ThingsAI Agent Benchmarks
AskPDF
Sep 26, 2026

OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents

Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life. In such interactions, users often rely on the surrounding multimodal context-pointing at objects or referring to what they see or hear, leaving their requests underspecified in language alone. Existing smart-home benchmarks, however, express user requests solely through language, leaving context-dependent real-world requests underexplored. To bridge this gap, we introduce OmniSmartHome, a multimodal smart-home benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests. OmniSmartHome comprises 1,360 synthetic and 272 real-world episodes. We evaluate 16 omnimodal large language models (Omni-LLMs) and reveal that, while they perform strongly when speech alone sufficiently conveys the user's intent, performance drops substantially when resolving it requires reasoning over multimodal contextual cues. As a simple agent baseline, we provide PROME (PROcedural Memory for multimodal Evidence gathering), which equips agents with specialized audio-visual perception tools and procedural memory for orchestrating their use. PROME generally improves performance across six Omni-LLMs. Demos and examples are available at https://omni-smart-home.github.io

Jihoo Jung, Suho Yoo, Jeongsoo Choi +5

Korea Advanced Institute of Science and Technology (KAIST) · Hankuk University of Foreign Studies

Multimodal AgentsMultimodal ReasoningAI Agent Benchmarks
AskPDF
Apr 11, 2026

Trust Your Memory: Verifiable Control of Smart Homes through Reinforcement Learning with Multi-dimensional Rewards

Large Language Models (LLMs) have become a key foundation for enabling personalized smart home experiences. While existing studies have explored how smart home assistants understand user queries to control devices in real time, their ability to perform memory-driven device control remains challenging from both evaluation and methodological perspectives. In terms of evaluation, existing benchmarks either focus on immediate device control or general open-domain memory retrieval tasks, and therefore cannot effectively evaluate a model's ability to perform memory-driven device control. Methodologically, while memory-driven device control can be approached using Reinforcement Learning, conventional RL methods generally rely on outcome-based supervision (i.e., whether the final task is achieved). This lack of intermediate feedback can lead to sub-optimal performance or local failures in fine-grained memory management tasks (adding, updating, deleting, and utilizing). To address these issues, we first release MemHomeLife, built from anonymized real-world long-term user interaction logs. To enable more fine-grained evaluation of different memory-related subtasks, we further construct MemHome, the first benchmark designed to systematically evaluate memory-driven device control in smart home scenarios.

Kai-Yuan Guo, Jiang Wang, Renjie Zhao +5

AI Research Center, Midea Group, Shanghai, China

Memory-Augmented Language Model AgentsInternet of ThingsRL for LLM Agents
AskPDF