SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts
Organizations: Tsinghua Shenzhen International Graduate School Shenzhen, China · Jilin University Jilin, China · The Chinese University of Hong Kong Hong Kong, China · Pengcheng Laboratory Shenzhen, China
Abstract
Due to the strong context-awareness capabilities demonstrated by large language models (LLMs), recent research has begun exploring their integration into smart home assistants to help users manage and adjust their living environments. While LLMs have been shown to effectively understand user needs and provide appropriate responses, most existing studies primarily focus on interpreting and executing user behaviors or instructions. However, a critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state. This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions. To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench, which is the first smart home dataset designed for LLMs, containing both normal and anomalous device states as well as normal and anomalous device state transition contexts. We evaluate 13 mainstream LLMs on this benchmark. The experimental results show that most state-of-the-art models cannot achieve good anomaly detection performance. For example, Claude-Sonnet-4.5 achieves only 66.1% detection accuracy on context-independent anomaly categories, and performs even worse on context-dependent anomalies, with an accuracy of only 57.8%. More experimental results suggest that next-generation LLM-based smart home assistants are still far from being able to effectively detect and handle anomalous conditions in the smart home environment. Our dataset is publicly available at https://github.com/horizonsinzqs/SmartBench.
Figures & tables
| Statistics | Description | Statistics | Description |
| Time of Day | morning, midday, afternoon, night | Room types | basement, bathroom, bedroom, entrance, garage, hallway, kitchen, living_room |
| Season | spring, summer, fall, winter | Environment Parameter | temperature, humidity, light, motion, weather, smoke, gas, moisture day_of_week, is_weekend |
| #Occupancy | 29 | #Duration | 15 |
| #Device | 155 | #State | 274 |
| #CI Normal | 1000 | #CD Normal | 1200 |
| #CI Anom | 1000 | #CD Anom | 1200 |
| Context-Indenpendent | Context-Dependent | |||||||||
| Precision | Recall | F1 score | FPR | AL Score | Precsion | Recall | F1 score | FPR | AL Score | |
| gemini-3 | 74.2% | 85.2% | 79.3% | 29.7% | 0.491 | 57.4% | 79.8% | 66.8% | 59.2% | 0.347 |
| gemini-2.5 | 64.5% | 85.6% | 73.5% | 47.2% | 0.397 | 53.8% | 91.0% | 67.6% | 78.2% | 0.365 |
| claude-4.5 | 63.9% | 74.0% | 68.6% | 41.8% | 0.319 | 59.6% | 59.0% | 59.3% | 40.0% | 0.257 |
| claude-4 | 73.8% | 50.7% | 60.1% | 18.0% | 0.232 | 67.3% | 44.5% | 53.6% | 21.7% | 0.247 |
| deepseek-r1 | 75.8% | 68.5% | 72.0% | 21.9% | 0.365 | 52.2% | 83.7% | 64.3% | 76.5% | 0.261 |
| Context-Independent | Context-Dependent | |||||
|---|---|---|---|---|---|---|
| P | R | F1 | P | R | F1 | |
| qwen-8b | 34.3% | 24.0% | 28.2% | 47.6% | 41.6% | 44.4% |
| qwen-32b | 48.8% | 80.0% | 60.6% | 45.2% | 79.1% | 57.5% |
| llama-8b | 30.8% | 8.0% | 12.7% | 50.0% | 16.7% | 25.0% |
| llama-70b | 42.9% | 18.0% | 25.4% | 0.0% | 0.0% | 0.0% |
| llama-405b | 57.1% | 32.0% | 41.0% | 75.0% | 37.5% | 50.0% |
| Accuracy | F1 score | Token | ||||
|---|---|---|---|---|---|---|
| Raw | CC | Raw | CC | Raw | CC | |
| claude-4.5 | 60.0% | 60.0% | 66.7% | 66.7% | 5.58M | 2.18M |
| claude-4 | 62.0% | 76.0% | 64.2% | 77.8% | 5.58M | 2.17M |
| deepseek-r1 | 64.0% | 68.0% | 59.1% | 63.6% | 5.40M | 2.16M |
| deepseek-v3 | 46.0% | 44.0% | 62.9% | 53.3% | 5.33M | 2.12M |
| gemini-3 | 46.0% | 50.0% | 62.0% | 64.8% | 10.90M | 3.12M |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| # | Device | State | # | Device | State |
| 1 | ‘alarm_panel‘ | ARMED, ARMED_AWAY, DISARMED, TRIGGERED | 79 | ‘kitchen_smart_plug‘ | OFF, ON |
| 2 | ‘alarm_panel_connectivity‘ | OFFLINE | 80 | ‘kitchen_stove‘ | OFF, ON |
| 3 | ‘backyard_light_connectivity‘ | OFFLINE, ONLINE | 81 | ‘kitchen_stove_state_log‘ | PERSISTED:ON |
| 4 | ‘basement_circuit_breaker‘ | CHECK | 82 | ‘kitchen_toaster‘ | OFF, ON |
| 5 | ‘basement_hvac‘ | COOLING, HEATING, OFF | 83 | ‘kitchen_under_cabinet_light‘ | OFF, ON |
| 6 | ‘basement_light‘ | OFF, ON | 84 | ‘living_room_blinds‘ | OPEN |
| Anomaly Type | Description | Subtype |
|---|---|---|
| device_malfunction | Device or sensor malfunction, offline status, or abnormal reporting, or inconsistencies between “control/display status” and telemetry readings, physical location, or other supporting evidence. | actuator_position_anomaly, alarm_false_trigger, camera_offline, configuration_conflict, invalid_categorical_state, leak_sensor_false_positive, leak_sensor_suspicious, lock_door_state_inconsistent, sensor_unavailable, telemetry_state_conflict |
| safety_hazard | High-risk safety events or protection failures related to smoke, gas, water leakage, door/window security, or surveillance, which may lead to fire, explosion, intrusion, or property loss. | alarm_sensor_state_conflict, garage_access_risk, gas_detected, open_entry_while_away, security_disabled_with_activity, security_system_disabled, security_system_disabled_with_activity, smoke_detected, smoke_false_alarm_or_sensor_fault, surveillance_disabled, surveillance_disabled_with_activity, unlocked_entry_while_away, unlocked_entry_with_activity, water_leak_detected, window_left_open_in_bad_weather |
| physical_violation | Out-of-range, missing, or improperly formatted values, or values that contradict basic physical laws or environmental facts (e.g., humidity readings inconsistent with signs of leakage, or abnormal temperature differences between adjacent rooms). | humidity_invalid_value, humidity_out_of_range, humidity_physics_conflict, temperature_invalid_value, temperature_out_of_range, temperature_spatial_inconsistency, temperature_unrealistic_indoor, thermal_distribution_anomaly |
| compound_anomaly | Multiple high-risk signals occur concurrently within the same time window, forming compound risks such as fire, intrusion, or utility-related hazards (with overlapping risks that are harder to interpret). | fire_signature, intrusion_signature, multiple_safety_alerts, security_posture_failure, water_electric_risk_signature |
| environmental_conflict | The on/off status, mode, or target temperature of the HVAC system is inconsistent with ambient conditions such as indoor temperature or open-window ventilation, leading to conflicting control objectives or unreasonable adjustment directions. | cooling_target_conflict, cooling_temperature_conflict, heating_target_conflict, heating_temperature_conflict, hvac_not_responding, hvac_running_while_off, hvac_status_mode_mismatch, hvac_window_conflict |
| occupancy_mismatch | Occupancy profiling (e.g., at-home or away status) conflicts with activity evidence such as motion sensors, access control logs, or power consumption patterns, indicating potential labeling errors or suggesting abnormal behavior or intrusion risks. | activity_signature_with_no_occupants, device_active_while_away, motion_with_no_occupants, multi_room_activity_with_no_occupants |
| Anomaly Type | Description | Subtype |
|---|---|---|
| device_malfunction | Device or sensor failure, disconnection, or performance degradation, manifested as loss of controllability, lack of reporting, or reported readings/states that contradict other sensor data or power consumption patterns observed during the same time period. | device_failure, parameter_drift, parameter_spike, sensor_stuck, sensor_drift, thermostat_failure, device_aging, component_cascade_failure, predictive_failure_risk |
| causal_temporal | Anomalous causal and temporal relationships between events, such as expected events not occurring, events happening out of order, significantly delayed responses, or durations/frequencies deviating from normal patterns. | causal_break, causal_delay, duration_anomaly, sequence_violation, timing_anomaly, periodicity_anomaly, frequency_anomaly, period_anomaly |
| statistical_anomaly | The data significantly deviates from historical patterns or similar scenarios in terms of overall level, fluctuation amplitude, or the synchronization between key indicators. | correlation_anomaly, baseline_deviation, point_anomaly, contextual_anomaly, collective_anomaly, spurious_event, distribution_shift, apparent_anomaly_mostly_seasonal |
| safety_hazard | High-risk safety incidents related to water leakage, intrusion, security breaches, device compromise, or electricity theft, which may pose direct threats to personal or property safety. | water_leak, hidden_water_leak, collective_security_trigger, security_issue, unauthorized_entry, progressive_device_compromise, power_theft_upstream |
| climate_anomaly | Anomalies in HVAC system cooling/heating/ventilation and zone control, such as malfunctions, overheating, ventilation issues, mode conflicts, or imbalanced zone regulation. | hvac_failure, excess_heat, ventilation_anomaly, hvac_mode_conflict, hvac_zone_conflict, hvac_zone_imbalance |
| network_disconnect | Anomalies in network links, gateways, or integration channels, leading to communication interruptions/fluctuations, state desynchronization, or cross-zone synchronization failures. | network_failure, hub_offline, wireless_interference, network_topology_anomaly, state_sync_failure, zone_sync_failure, integration_failure, sensor_fusion_anomaly |
Anomaly Detection
Anomaly detection finds rare or unexpected patterns in data, from defective products and system faults to fraud and security threats.
See all 652 papers