Can You Check That? The Checkability Boundary for Local LLM Network Automation

Maleeha Masood; Momina Nofal

cs.NISep 25, 2026

Can You Check That? The Checkability Boundary for Local LLM Network Automation

Authors: Maleeha Masood, Momina Nofal

Organizations: University of Illinois Urbana-Champaign Urbana, IL, USA · Independent Researcher USA

Abstract

Sending every network-automation input to a third-party frontier LLM exports sensitive artifacts such as production configurations, topologies, and logs. Querying small language models (SLMs) locally avoids this egress, but SLM outputs can be error-prone for direct use. This work introduces checkability as a criterion for determining which tasks are suitable for local inference. A task is checkable when it exposes a cheap, deterministic test - an intrinsic check - that rejects outputs violating a necessary correctness condition. We instantiate this idea in Touchstone, a local-first pipeline that uses seven off-the-shelf SLMs (1-8B parameters) to generate candidates, uses task-specific intrinsic checks to reject responses, and escalates unresolved inputs to a frontier LLM. On conflict detection and intent translation tasks, Touchstone reaches 98.6% and 93.8% end-to-end accuracy while escalating only 16% and 17% of inputs, respectively. On TeleQnA, a knowledge-only control that has no task-specific intrinsic checks, Touchstone is unable to match the accuracy of the frontier baseline. Our results support a simple deployment rule: keep inference local when task semantics support precise, low-cost checks; escalate the rest.

Figures & tables

Panel of Figure 1 — Figure 1. Touchstone’s local-first workflow

Structured Tasks
Dataset	$n$	Best SLM	GPT-5.5	Touchstone	Egress
Conflict det.	500	0.840	1.000	0.986	16%
Intent transl.	500	0.740	0.992	0.938	17%
Log parse (SSH)	300	0.743	0.997	0.997	37%
Log parse (HDFS)	300	0.863	0.987	0.917	7%
Log parse (Prox.)	300	0.650	0.917	0.877	42%

Table 1. Touchstone accuracy and frontier egress across all tasks, compared with GPT-5.5 and Best Single SLM accuracy.

Check	Task	FAR
Consistency	Conflict detection	0.0%
Grounding + coverage	Intent translation	5.6%
Invertibility	Log (OpenSSH)	27.7%
Invertibility	Log (HDFS)	14.4%
Invertibility	Log (Proxifier)	30.0%

Table 2. False-accept rate (FAR) of each intrinsic check on the evaluation split.

LLM Answer Verification

Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.

See all 150 papers

HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment
Sep 29, 2026Kenan Alkiek, Moontae Lee, David Jurgens +1On-Device Language Model Inference Selective Prediction
School of Information University of Michigan Ann Arbor, MI 48109, USA · SNU-LG AI Research Center Seoul, South Korea
Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
Aug 17, 2026Xiangfan Wu, Zonghao Ying, Huiyu Wu +4LLM Auditing Black-Box Model Auditing
Tencent Zhuque Lab
Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs
Feb 19, 2026Arka Pal, Louai Zahran, William Gvozdjak +2Verifiable Outsourced Computation LLM Inference
Ritual · MIT · Columbia University

Sep 29, 2026J/K move · Enter open · S save

HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment

Kenan Alkiek, Moontae Lee, David Jurgens, V. G. Vinod Vydiswaran

School of Information University of Michigan Ann Arbor, MI 48109, USA · SNU-LG AI Research Center Seoul, South Korea

On-Device Language Model Inference Selective Prediction Efficient Language Model Reasoning LLM Uncertainty Estimation Efficient Language Model Inference Language Model Cascading

Open paper AskPDF arXiv

Running a language model locally offers advantages in privacy, latency, and cost, but local hardware fits only small models, which are less capable than frontier models. The usual remedy for a hard query, escalating it to a cloud model, gives up the privacy and cost advantages of running locally. A deployment that stays local faces two decisions for hard queries instead. First, it can spend more computation on a query, e.g., reasoning before answering, which raises accuracy at a cost in latency, so it must decide which queries are worth the extra computation (efficiency). Second, some queries are beyond the local model, and delivering a wrong answer is worse than deferring the query to a human in the loop, so it must decide which answers are safe to deliver (safety). We show that both decisions can be made from the model's own hidden states. The prefill state, computed before any token is generated, predicts whether the model will answer correctly, and the answer state, at the end of the generated answer, predicts whether that answer is correct. HARISSA fine-tunes the model so that both states predict correctness, then makes both decisions with one policy that cascades through the ways of answering from cheapest to most expensive, skipping a way the prefill state predicts will fail and deferring the query when the answer it stops with is predicted wrong. On a device running a single model, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency. On a server holding four sizes of one model, HARISSA is more accurate than the FrugalGPT and Self-REF cascades at the same latency, and at the same deferral rate the answer state leaves fewer wrong answers than the standard confidence signals in five of six task and setting pairs.

Abstract

LLM Answer Verification

Explore similar work

HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs