Trail: Accurate Local Smart Home Control Using Decision Models
Abstract
Natural-language driven smart home control remains challenging: cloud-hosted Large Language Models (LLMs) raise privacy concerns, while local Small Language Models (SLMs) can generate invalid actions or misinterpret requests. We introduce Trail (Typed Registry-Anchored Inference for Language commands in Smart Homes), a fully local framework that replaces open-ended command generation with hierarchical typed decisions. Trail exploits the natural structure of a home's action space, selecting a room, device, and operation from candidates derived from the runtime device registry. This ensures valid actions while enabling confidence-based abstention when requests cannot be reliably resolved. We also design a structured memory module that uses typed decisions to support user preferences, device references, and multi-turn interactions. To evaluate Trail, we extend HomeBench and generate three datasets: Trek-Base, which tests natural-language variation; Trek-Robust, which evaluates robustness to changes in the home and its description; and Trek-Memory, which assesses conversational memory across multi-turn interactions. Trail achieves 85.3% accuracy on Trek-Robust versus 69.8% for a finetuned Qwen3.5-4B baseline, while remaining competitive on Trek-Base requests. With memory, Trail resolves 75.6% of the requests that depend on earlier turns, compared with 42.5% for the finetuned baseline given conversation history.
Figures & tables
| Trek-Base | Trek-Robust | |
| Trail | 0.862 | 0.853 |
| Qwen3.5-4B, prompted | 0.495 | 0.451 |
| Qwen3.5-4B, finetuned | 0.897 | 0.698 |
| VCU-LLM | 0.849 | 0.505 |
| Domux | 0.517 | 0.467 |
| Sasha | 0.463 | 0.463 |
| Original | Explicit | Implied | Goal | All | |
| Trail | 0.994 | 0.885 | 0.817 | 0.703 | 0.862 |
| Qwen3.5-4B, prompted | 0.788 | 0.424 | 0.442 | 0.196 | 0.495 |
| Qwen3.5-4B, finetuned | 0.998 | 0.913 | 0.865 | 0.771 | 0.897 |
| VCU-LLM | 0.959 | 0.843 | 0.801 | 0.795 | 0.849 |
| Domux | 0.675 | 0.417 | 0.483 | 0.477 | 0.517 |
| Sasha | 0.620 | 0.342 | 0.446 | 0.406 | 0.463 |
| CF | UC | MI | NI | Rnd | NH | NV | Scale | SD | All | |
| Trail | 0.851 | 0.670 | 0.885 | 0.920 | 0.855 | 0.685 | 0.915 | 0.905 | 0.995 | 0.853 |
| Qwen3.5-4B, prompted | 0.551 | 0.375 | 0.125 | 0.570 | 0.450 | 0.555 | 0.305 | 0.550 | 0.475 | 0.451 |
| Qwen3.5-4B, finetuned | 0.799 | 0.620 | 0.245 | 0.915 | 0.910 | 0.670 | 0.600 | 0.950 | 0.470 | 0.698 |
| VCU-LLM | 0.630 | 0.250 | 0.310 | 0.405 | 0.860 | 0.115 | 0.875 | 0.815 | 0.155 | 0.505 |
| Domux | 0.479 | 0.380 | 0.690 | 0.510 | 0.580 | 0.265 | 0.460 | 0.670 | 0.155 | 0.467 |
| Sasha | 0.434 | 0.325 | 0.630 | 0.420 | 0.535 | 0.385 | 0.485 | 0.535 | 0.450 | 0.463 |
| # Homes Training | Trek-Base, 20 left-out | Trek-Robust, new home |
| 100 (Trail) | 0.859 | 0.685 |
| 80 | 0.860 | 0.695 |
| All | Original | Memory | |
| Trail, no memory | 0.760 | 0.872 | 0.277 |
| Trail + memory | 0.848 | 0.870 | 0.756 |
| Qwen3.5-4B, finetuned | 0.801 | 0.905 | 0.350 |
| Qwen3.5-4B, finetuned + history | 0.809 | 0.898 | 0.425 |
| Median (s) | P90 (s) | # Calls | |
| Trail | 0.166 | 0.801 | 5.4 |
| Qwen3.5-4B, prompted | 0.573 | 0.813 | 1.0 |
| Qwen3.5-4B, finetuned | 0.531 | 0.897 | 1.0 |
| VCU-LLM | 0.079 | 0.151 | 1.8 |
| Domux | 0.169 | 0.797 | 1.0 |
| Sasha | 1.803 | 4.542 | 2.9 |
| Median (s) | P90 (s) | # Calls | |
| Trail | 0.169 | 0.607 | 4.5 |
| Qwen3.5-4B, prompted | 0.647 | 1.578 | 1.0 |
| Qwen3.5-4B, finetuned | 0.643 | 1.684 | 1.0 |
| VCU-LLM | 0.094 | 0.160 | 1.4 |
| Domux | 0.169 | 0.345 | 1.0 |
| Sasha | 1.691 | 4.427 | 2.3 |
| Median (s) | P90 (s) | Calls | |
| Trail, no memory | 0.172 | 0.836 | 5.7 |
| Trail + memory | 0.215 | 1.070 | 7.5 |
| Qwen3.5-4B, finetuned | 0.486 | 0.821 | 1.0 |
| Qwen3.5-4B, finetuned + history | 0.727 | 1.673 | 1.0 |
| Decision | Question |
| Level 1 – Room | Which room does this command refer to? |
| Level 2 – Device | Which device in that room does this command control? |
| Level 3 – Operation | Which operation on that device does this command ask for? |
| Enumerated parameter | Which setting does the command ask for on this device? |
| Numeric parameter, first pass | What value between {lo} and {hi} does the command ask for? |
| Numeric parameter, refinement | Refine the value: which exact number between {lo} and {hi} does the command ask for? |
| Decision | Question | Options |
| Command Type | What is the user doing? | now : asking the home to do something now; remember : telling the assistant something to remember for the future; revise : correcting the last action or asking the assistant to forget something it remembers. |
| Kind of statement | What does the user want the assistant to remember or change? | preference : a lasting preference for how a device should usually be set; alias : a name or nickname for a device; forget : forgetting a remembered preference or name; correction : the previous action used the wrong device and this names the right one. |
| Preference operation | Which setting of the device is the user talking about? | The device’s set_* operations. |
| Preference situation | Which words describe the situation in which the user wants this setting? | Phrases from the message, plus “no particular situation – this is the usual setting”. |
| Alias | Which words are the name the user wants to use for the device? | Phrases from the message. |
| Correction | Which words in the earlier request referred to the device? | Phrases from the earlier request. |
Language Model-Based Control
Language model-based control uses language models to interpret goals, choose actions, and steer AI systems or physical processes.
See all 196 papers