ADF-EA: A Unified Execution Assurance System for Agent Device Foundation
Organizations: Huawei
Abstract
Agents based on large language models (LLMs) can access heterogeneous devices through tools and APIs, but reliable execution must account for unmet effects, uncertain outcomes, and changing prerequisites. A command may be acknowledged without producing its intended effect, while missing feedback may obscure an action that has already succeeded. We present Agent Device Foundation--Execution Assurance (ADF-EA), an architecture that connects agent planning and device execution through shared capability contracts. Device Capability Contracts (DCCs) unify invocation conditions, intended effects, evidence requirements, and recovery rules across heterogeneous interfaces. Agents use these contracts to plan, while the runtime applies the same semantics to authorize actions, verify effects, and govern continuation and completion. Persistent execution state retains verified progress, unresolved outcomes, and remaining budgets across plan revisions, enabling observation-based recovery, authorized retries, and necessary state repair. We formalize the execution lifecycle and establish conditional soundness properties for completion and recovery authorization. Evaluations span multiple LLMs, five agent frameworks, and simulated process-control, household, and robotic manipulation domains. Compared with direct invocation and existing execution-checking approaches, ADF-EA reduces false completion and unnecessary repetition, supports necessary state repair, prevents calls to unavailable capabilities, and preserves permitted task completion and recovery. These results demonstrate DCCs as a reusable semantic foundation for agent autonomy across heterogeneous devices, unifying capability-based planning, evidence-grounded execution, and authorized recovery within one architecture.
Figures & tables
| System | Contracts and checks | Execution state and recovery | Assurance focus |
|---|---|---|---|
| ToolGate [ 18 ] | Hoare-style preconditions and postconditions | Typed symbolic state; state updates committed after verification | Admissible invocation and verified state evolution |
| ROSClaw (Cardenas et al.) [ 5 ] | Capability discovery; pre-execution action validation | Normalized observations; structured audit logs | Mediated ROS 2 execution and behavioral auditing |
| ROSClaw runtime repository [ 28 ] | Action authorization; effect verification | Durable receipts; unknown interrupted REAL outcomes; no automatic retry; operator review | Evidence-backed execution records and restart gating |
| SafeGate [ 23 ] | Guards, invariants, aborts; pre/post-action checks | Symbolic state; violation records; halt on violation | Safety-contract compliance during execution |
| SINT [ 30 ] | Capability tokens; policy and physical constraints | Request lifecycle; evidence ledger; failure/rollback transitions | Authorized requests and auditable execution records |
| RTA: SOTER [ 7 ] | Safety specifications; runtime monitoring | Controller state; safe-controller fallback | Safety preservation under module assumptions |
| Instance | Invocation / effect | Evidence realization | Interruption example |
|---|---|---|---|
| Heating: heat_to | Target input range 20–100 ∘ C; the studied target effect is 79–81 ∘ C | Three consecutive observations from a sensor provider distinct from the actuator, with identity-qualified windows, 1-logical-second spacing, maximum age of 20 logical seconds, and sample matching | Additional observation after interrupted verification; restricted rebinding subject to its declared state constraint |
| AI2-THOR: opening an object | Retained-step ordering; native object-open condition | Object-specific native state; projected sequencing and task-completion predicates | Unestablished effect or stale observation must be resolved before dependent progress |
| Meta-World: bounded manipulation segment | Sequence-ready prerequisite; successor readiness or final goal.complete | Native evaluate_state success for final completion, with separately projected state observations | Effect suppression and stale-observation conditions exercise verification and continuation |
| Metric | ADF-EA | ToolGate | VTC | Direct |
| Goal-confirmed completion (permitted conditions) | 14/14 | 9/14 | 14/14 | 13/14 |
| Evidence-qualified completion (permitted conditions) | 14/14 | 9/14 | 12/14 | 11/14 |
| Accepted completion with physical goal unmet (runs) | 0/17 | 0/17 | 1/17 | 2/17 |
| Accepted completion: goal met, evidence unqualified (runs) | 0/17 | 0/17 | 3/17 | 3/17 |
| Violating provider dispatches | 0 | 10 | 14 | 14 |
| Runs containing a violating dispatch | 0/17 | 8/17 | 11/17 | 11/17 |
| Condition | Terminal outcome | Dispatches | Repairs | Improper repeats | Model calls |
|---|---|---|---|---|---|
| Container state invalidated | 3/3 qualified, authorized completions | 15 | 3 | 0 | 27 |
| Put evidence unavailable | 3/3 explicit agent stops; goal unmet | 9 | 0 | 0 | 27 |
| Condition | Native goal | Qualified completion | Provider calls | Repeated calls | Control steps |
|---|---|---|---|---|---|
| Normal | Met | Yes | 3 | 0 | 52 |
| Persistent evidence insufficiency | Met | No | 3 | 0 | 52 |
| Adaptive study | Outcome | Direct | ADF-EA |
| Room-exit: Flash | Feasible-task completion | 9/9 | 9/9 |
| False completion | 6/18 | 0/18 | |
| Unavailable dispatches | 7 | 0 | |
| Room-exit: GLM MAX | Feasible-task completion | 9/9 | 9/9 |
| False completion | 6/18 | 0/16 | |
| Unavailable dispatches | 8 | 0 |
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | ADF-EA | ToolGate | VTC | Direct |
| No external fault: qualified completion | 3/3 | 3/3 | 3/3 | 3/3 |
| Evidence-recovery conditions: qualified completion | 7/7 | 4/7 | 5/7 | 5/7 |
| Stale-evidence subset: qualified completion | 1/1 | 1/1 | 1/1 | 1/1 |
| Retry-permitted conditions: qualified completion | 3/3 | 0/3 | 2/3 | 1/3 |
| State-repair condition: qualified completion | 1/1 | 0/1 | 0/1 | 0/1 |
| Necessary state-repair dispatches | 1 | 1 | 1 | 1 |
| Metric | ADF-EA | ToolGate | VTC | Direct |
| Final physical goal met; no completion declaration required | ||||
| Completion-designated conditions | 14/14 | 10/14 | 14/14 | 13/14 |
| Stopping probes (descriptive) | 1/3 | 1/3 | 2/3 | 2/3 |
| All conditions (descriptive aggregate) | 15/17 | 11/17 | 16/17 | 15/17 |
| Violation-free completion (permitted conditions) | 14/14 | 3/14 | 4/14 | 4/14 |
| Violating physical requests actually executed | 0/14 | 10/10 | 14/14 | 14/14 |
| Metric | ADF-EA | ToolGate | VTC | Direct |
|---|---|---|---|---|
| Physical-action requests | 67 | 52 | 63 | 63 |
| Provider calls / native Boolean actions | 53 | 52 | 65 | 63 |
| Observations (including algorithm-internal calls) | 7 | 4 | 8 | 5 |
| State reads | 199 | 38 | 22 | 22 |
| All issued requests | 104 | 69 | 93 | 93 |
| Original requests not issued after termination | 0 | 31 | 7 | 7 |
| Evaluation focus | Design | Sample composition | Results |
|---|---|---|---|
| Published-algorithm comparison Boolean Device | Controlled requests; independent event scoring | 17 conditions on 3 task IDs 4 methods = 68 runs; 16 pilot runs excluded | Table 3 |
| Frozen-contract composition reuse | Six new combinations; frozen runtime | 6 cases 4 methods = 24 records; separate six-case development regression | Section 4.2 |
| Completion and safe termination Room-exit | Adaptive, Flash and GLM MAX | 6 configurations 3 repetitions 2 arms 2 models = 72 scheduled runs | Table 11 |
| Permitted recovery AI2-THOR ReAct | Adaptive reevaluation | 15 tasks, each normal and faulted, 2 arms = 60 runs | Table 12 |
| Admission and verification | Fixed-proposal replay | 43 original and 31 replication tasks, each normal and faulted; separate 49-case lifecycle set | Table 13 |
| ADF continuation mechanisms | Internal component ablation | 43-task faulted population; ADF component removals and Direct reference | Table 24 |
| DeepSeek V4.1 Flash | GLM-5.3 MAX | |||
| Outcome | Direct | ADF-EA | Direct | ADF-EA |
| Valid / scheduled runs | 18/18 | 18/18 | 18/18 | 16/18 |
| Transport-interrupted runs | 0 | 0 | 0 | 2 |
| Feasible runs: goal-confirmed completion | 9/9 | 9/9 | 9/9 | 9/9 |
| Infeasible runs: safe termination | 3/9 | 9/9 | 3/9 | 7/7 |
| False completion | 6/18 | 0/18 | 6/18 | 0/16 |
| Outcome | Direct | ADF-EA |
| Normal: qualified completion | 15/15 | 15/15 |
| Allowed recovery: qualified completion | 5/5 | 5/5 |
| Stale evidence: qualified completion | 5/5 | 5/5 |
| No continuation: explicit agent stop | 5/5 | 5/5 |
| No continuation: runs with violating dispatch | 5/5 | 0/5 |
| Policy-violating provider dispatches | 10 | 0 |
| Original population | Fresh replication | Unified lifecycle | ||||
|---|---|---|---|---|---|---|
| Outcome | Direct | ADF-EA | Direct | ADF-EA | Direct | ADF-EA |
| Normal goal-confirmed completion | 43/43 | 43/43 | 31/31 | 31/31 | 8/8 | 8/8 |
| Faulted false completion | 15/43 | 0/43 | 8/31 | 0/31 | 8/41 | 0/41 |
| Effect rejection | 0/15 | 15/15 | 0/8 | 8/8 | 0/4 | 4/4 |
| Appropriate no-continuation termination | 0/14 | 14/14 | 0/12 | 12/12 | 0/7 | 7/7 |
| Faulted unsafe/unavailable dispatch | 14/43 | 0/43 | 12/31 | 0/31 | 33/41 | 0/41 |
| Outcome | Policy-informed Direct | ADF-EA |
| Qualified completion | ||
| Normal execution | 2/2 | 2/2 |
| Evidence reacquisition | 2/2 | 2/2 |
| Retry permitted | 2/2 | 2/2 |
| Forbidden retry requests dispatched | 2/2 | 0/2 |
| Accepted completion with physical goal unmet | 2 | 0 |
| Allowed retry: qualified completion | Stale evidence: qualified completion | No continuation: runs with violating dispatch | |||||
| Framework | Version | Direct | ADF-EA | Direct | ADF-EA | Direct | ADF-EA |
| OpenAI Agents SDK | 0.19.4 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
| LangGraph | 1.2.9 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
| Google ADK | 2.5.0 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
| Microsoft Agent Framework | 1.12.0 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
| smolagents | 1.26.0 | 5/5 | 5/5 | 5/5 | 5/5 | 5/5 | 0/5 |
| Domain | Changed integration | Observed mechanism behavior | Scope |
|---|---|---|---|
| Heating | Process model; heater, measurements, recorder/readback | Scenario completion 2/4 to 4/4; both applicable provider recoveries succeed | Four curated software cases; scenario results without generic terminal TaskResult |
| AI2-THOR | Grounded household actions and native goal predicates | Controlled replay, held-out replication, and adaptive-agent comparisons | Qualified tasks under structured observations and injected faults |
| Meta-World | Bounded scripted skills and native state evaluation | Normal goal-confirmed completion 8/8 in both arms; all eight missing effects rejected; all six unavailable successors blocked by ADF-EA | Four tasks, two seeds; no low-level control or real-sensor evaluation |
| ReAct: 30 runs per arm | Five SDKs: 75 runs per arm | |||
|---|---|---|---|---|
| Count | Direct | ADF-EA | Direct | ADF-EA |
| Model calls | 129 | 122 | 351 | 297 |
| Provider dispatches | 93 | 83 | 245 | 195 |
| Provider retries | 10 | 5 | 50 | 25 |
| Native action executions | 78 | 78 | 170 | 170 |
| Repeated native action executions | 0 | 0 | 0 | 0 |
| Action and observation | Verification result | Progress and obligations | Permitted next step |
|---|---|---|---|
| 1. Heating acknowledged Authorized heating returns an ACK; temperature evidence is stale. | Insufficient evidence. | Heating remains unverified; its evidence obligation stays open. | Reacquire temperature observations. The ACK does not authorize completion or reheating. |
| 2. Heating verified Fresh matching measurements establish the effect. | Satisfied. | Heating becomes verified progress; its evidence obligation is resolved. | Check the recording action’s current prerequisites and provider eligibility. |
| 3. Recording fails Qualified readback shows the required record is absent. | Unsatisfied effect requirement, supported by qualified negative evidence. | The recording obligation stays open; still-valid heating progress is retained. | Evaluate the declared recovery permission and remaining budget. |
| 4. Replacement succeeds An eligible replacement recorder is authorized, executes, and returns qualified readback. | Satisfied. | The recording obligation is resolved; both required outcomes are established. | Authorize completion if the task goal and all required obligations are satisfied. |
| Population stage | Count |
|---|---|
| AI2-THOR scenes | 120 |
| Development scene excluded | 1 |
| Candidate scenes | 119 |
| Stable scenes after repeated reset | 114 |
| Generated task candidates | 25,590 |
| Static eligible candidates | 18,649 |
| Evaluation | Selected | Native-valid | Final paired |
|---|---|---|---|
| Fresh execution-time fault replication | 48 | 32 | 31 |
| Unified lifecycle | 64 | 49 | 49 |
| Grounding projection | 32 | 26 | 26 |
| Metric | Direct | ADF-EA |
|---|---|---|
| Normal, | ||
| Goal-confirmed completion | 43/43 | 43/43 |
| Oracle agreement | 43/43 | 43/43 |
| Physical actions | 114 | 114 |
| Faulted, | ||
| False completion | 15/43 | 0/43 |
| Metric | Direct | ADF-EA |
|---|---|---|
| Normal | ||
| Goal-confirmed completion | 31/31 | 31/31 |
| Oracle agreement | 31/31 | 31/31 |
| Physical actions | 93 | 93 |
| Faulted | ||
| False completion | 8/31 | 0/31 |
| Metric | Direct | ADF-EA |
| Prescribed disposition | 8/49 | 49/49 |
| Goal-confirmed completion | 24/49 | 12/49 |
| False completion | 8/49 | 0/49 |
| Pre-dispatch rejection | 0/49 | 26/49 |
| Invalid physical dispatch | 26/49 | 0/49 |
| Effect rejection | 0/4 | 4/4 |
| Direct | No reacq. | No cont. auth. | Full ADF-EA | |
| False completion | 15 | 0 | 0 | 0 |
| Goal-confirmed completion | 14 | 0 | 14 | 14 |
| Effect rejection | 0/15 | 15/15 | 15/15 | 15/15 |
| Appropriate no-continuation termination | 0/14 | 14/14 | 0/14 | 14/14 |
| Unsafe action | 14 | 0 | 14 | 0 |
| Unnecessary termination | 0 | 14 | 0 | 0 |
| Fault | Direct invalid dispatch | ADF-EA reject |
|---|---|---|
| Order violation | 8/8 | 8/8 |
| Budget violation | 7/7 | 7/7 |
| Unavailable binding | 8/8 | 8/8 |
| Target mismatch, original projection | 8/8 | 0/8 |
| Metric | Direct | ADF-EA |
| Corrupted proposal rejected | 0/26 | 26/26 |
| Invalid physical dispatch | 26/26 | 0/26 |
| False completion | 22/26 | 0/26 |
| Oracle agreement | 4/26 | 26/26 |
| Corrupted physical actions | 64 | 0 |
| Valid-control acceptance | N/A | 26/26 |
| Semantic dimension | Intact | Target weakened | Sham mutation |
|---|---|---|---|
| Invocation | 3/3 | 0/3 | 3/3 |
| Effect | 3/3 | 0/3 | 3/3 |
| Evidence | 3/3 | 0/3 | 3/3 |
| Interruption | 3/3 | 0/3 | 3/3 |
| Overall | 12/12 | 0/12 | 12/12 |
| Case | Condition | Direct behavior | ADF-EA behavior |
|---|---|---|---|
| H1 | Normal heating, measurement, and recording | Completes full proposal | Requires measurements from a provider distinct from the actuator and record/readback before verified completion |
| H2 | heater-a fails with no physical effect | Stops after failed heater action | Confirms missing effect, rebinds to heater-b , then completes unresolved suffix |
| H3 | Heating effect exists but evidence is insufficient | Eventually reaches physical completion without typed freshness authority | Preserves heating and obtains three missing measurements without reheating |
| H4 | Heating and measurements complete; recorder unavailable | Dispatches unavailable recorder and stops | Preserves verified prefix and rebinds only the unresolved record step |
| Metric | Direct | ADF-EA |
| Goal-confirmed completion | 2/4 | 4/4 |
| Effect-failure detection | 0/1 | 1/1 |
| Recovery/rebind | 0/2 | 2/2 |
| Unsafe action | 1/4 | 0/4 |
| Unnecessary termination | 2/4 | 0/4 |
| Physical actions | 7 | 9 |
| Condition | Direct | ADF-EA |
|---|---|---|
| Normal goal-confirmed completion | 8/8 | 8/8 |
| Effect-suppressed false completion | 8/8 | 0/8 |
| Effect rejection | 0/8 | 8/8 |
| Explicit stale-observation recovery | 0/8 | 8/8 |
| Physical retry after stale evidence | 0/8 | 0/8 |
| Unavailable successor dispatch | 6/6 | 0/6 |
| Mechanism | Heating | AI2-THOR | Meta-World |
|---|---|---|---|
| Pre-execution validation / binding | Detect unavailable recorder and rebind suffix | Reject order, budget, binding, and grounded-target violations | Validate and bind retained semantic skill plan |
| Post-action effect verification | Independently qualify missing heater effect | Reject suppressed household state transition 15/15 | Reject missing manipulation effect 8/8 |
| Fresh evidence acquisition | Acquire missing measurements without reheating | Fresh observation in all stale cases with no physical retry | Fresh native-state read in 8/8 with no skill retry |
| Continuation / recovery | Preserve prefix and rebind recoverable suffix | Terminate when no valid successor remains | Prevent unavailable successor skill |
| Goal-confirmed completion | Independent process oracle | Leaf-wise physical goal and TaskResult | Native task state and TaskResult |
| Endpoint | Direct | ADF-EA |
| Correct composed outcome | 0/16 | 16/16 |
| Correct first-fault detection | 0/16 | 16/16 |
| Correct fault precedence | 0/16 | 16/16 |
| Verified-prefix preservation | N/A | 16/16 |
| Unsafe/unavailable dispatch | 11/16 | 0/16 |
| False completion | 5/16 | 0/16 |
| Evidence perturbation | Expected | Observed | Recorded outcome |
|---|---|---|---|
| Values and numeric bounds | |||
| Nominal matching measurements | Verify | Verify | Satisfied |
| Measurements spanning both numeric bounds | Verify | Verify | Satisfied |
| Small numeric offsets within the range | Verify | Verify | Satisfied |
| Values exactly at the numeric bounds | Verify | Verify | Satisfied |
| Freshness and cadence | |||
| Condition | Runs | Model calls | Provider dispatches | Agent observations | Runtime observations |
|---|---|---|---|---|---|
| Normal | 15 | 59 / 59 | 44 / 44 | 0 / 0 | 0 / 0 |
| Allowed effect-failure recovery | 5 | 26 / 27 | 20 / 20 | 1 / 2 | 0 / 0 |
| Stale evidence | 5 | 23 / 18 | 13 / 13 | 5 / 0 | 0 / 5 |
| No legal continuation | 5 | 21 / 18 | 16 / 6 | 0 / 1 | 0 / 0 |
| Framework | Model calls | Provider dispatches | Agent observations | Runtime observations | Legal requests |
|---|---|---|---|---|---|
| OpenAI Agents SDK | 70 / 59 | 49 / 39 | 6 / 0 | 0 / 5 | 45 / 39 |
| LangGraph | 70 / 59 | 49 / 39 | 6 / 0 | 0 / 5 | 45 / 39 |
| Google ADK | 70 / 60 | 49 / 39 | 6 / 1 | 0 / 5 | 45 / 40 |
| Microsoft Agent Framework | 70 / 59 | 49 / 39 | 6 / 0 | 0 / 5 | 45 / 39 |
| smolagents | 71 / 60 | 49 / 39 | 7 / 1 | 0 / 5 | 46 / 40 |
| Model | Valid proposals | Model calls | Execution pairs |
|---|---|---|---|
| Qwen3.8 Max | 15/15 | 59 | 15 |
| DeepSeek V4 Pro | 15/15 | 59 | 15 |
| Kimi K3 | 15/15 | 59 | 15 |
| DeepSeek V4.1 Flash | 15/15 | 59 | 15 |
| Outcome | Direct | ADF-EA |
| Goal-confirmed completion | 5/15 | 5/15 |
| False completion | 5/15 | 0/15 |
| Effect-failure rejection | 0/5 | 5/5 |
| Appropriate no-continuation termination | 0/5 | 5/5 |
| Runs with violating actions | 5/15 | 0/15 |
| Oracle agreement | 10/15 | 15/15 |
| Expected endpoint | Unavailable calls | Repeat dispatches | Action dispatches | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Scenario | Expected | Direct | ADF-EA | Direct | ADF-EA | Direct | ADF-EA | Direct | ADF-EA |
| Normal | Complete | 3/3 | 3/3 | 0 | 0 | 0 | 0 | 9 | 9 |
| Order-sensitive locking | Complete | 3/3 | 3/3 | 0 | 0 | 0 | 0 | 9 | 9 |
| Suppressed door effect | Stop | 0/3 | 3/3 | 0 | 0 | 3 | 0 | 13 | 6 |
| Stale door evidence | Complete | 3/3 | 3/3 | 0 | 0 | 0 | 0 | 9 | 9 |
| Unavailable downstream lock | Stop | 3/3 | 3/3 | 7 | 0 | 4 | 0 | 13 | 6 |
| Expected endpoint | ADF-EA | |||
|---|---|---|---|---|
| Scenario | Expected | Direct | ADF-EA | Transport interruptions |
| Normal | Complete | 3/3 | 3/3 | 0 |
| Order-sensitive locking | Complete | 3/3 | 3/3 | 0 |
| Suppressed door effect | Stop | 0/3 | 1/1 | 2 |
| Stale door evidence | Complete | 3/3 | 3/3 | 0 |
| Unavailable downstream lock | Stop | 3/3 | 3/3 | 0 |
| DeepSeek V4.1 Flash | GLM-5.3 MAX | |||
|---|---|---|---|---|
| Terminal source | Direct | ADF-EA | Direct | ADF-EA |
| Goal-confirmed completion | 9 | 9 | 9 | 9 |
| False completion | 6 | 0 | 6 | 0 |
| Model-issued stop | 3 | 5 | 3 | 7 |
| Budget-triggered stop | 0 | 4 | 0 | 0 |
| Transport interruption | 0 | 0 | 0 | 2 |
| Endpoint | Direct | ADF-EA |
|---|---|---|
| Normal appropriate outcome | 50/50 | 50/50 |
| Late-fault appropriate outcome | 0/50 | 50/50 |
| Late-fault false completion | 50/50 | 0/50 |
| Verified-prefix preservation | N/A | 50/50 |
LLM Agent Reliability
Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.
See all 841 papers