cs.AIAug 26, 2026

Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy

Authors: Mazhar Shaikh, Anurag Rajkumar Bombarde, Harshal Pathak

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomous agents increasingly perform bounded software tasks under an orchestrator that retries, resumes, and budgets them. The machinery such orchestrators reach for is the service mesh's: retry, timeout, and error-rate circuit breaking. We report a failure study of a production agentic software-delivery platform over 147 numbered incidents spanning 81 runs, each with a measured cost and, in most cases, a mutation proof reproducing the failure. All three assumptions those primitives rest on are violated in practice, and we quantify the consequences: a loop of fifty-four consecutive successful tool calls no error-rate breaker could see; a progress signal constant by construction, guaranteeing a false trip on the third repair round and driving one run from six of six components to three; twenty-one events accumulated across six invocations of one delegation, making a correct, idempotent component unwinnable; a misrouted failure that woke five components for a two-component fault, leaving three bystanders regressing working code; and twelve incidents in which the enforcement layer blocked correct work, the most expensive costing 107 agent turns and zero accepted writes. We find one cross-cutting cause and its dual. Identity adequacy: in five separate subsystems an identity that failed to discriminate produced a confident wrong answer, and two of them derived the corrective rule independently. Evidence adequacy: a reliability decision may be taken only on evidence capable of moving, attributable to what it measures, and deterministic under identical conditions. From the findings we derive seven reliability primitives whose enforcement unit is the delegation rather than the message, and specify the controlled evaluation the study motivates but does not constitute.

Figures & tables

Fig. 1: Agent Mesh architecture. The mesh is defined against an abstract delegation interface with three interception seams (S1 model-call, S2 tool-invocation, S3 commit); any orchestrator exposing these seams can host the sidecar. The data plane runs seven primitives per delegation and reports through a verdict channel whose outcomes are deliberately distinct—in particular suppressed-as-duplicate (the ledger working) must not be confused with refused-as-stalled (the breaker tripping). The declaration authority sits in the control plane because it is admission control: it decides whether a delegation set may be created at all.
Fig. 2: Left: breaker scopes in the primary system, from run down to tool session, each with its own signal and budget; the highlighted vignette is a loop made entirely of successful calls. Right: the signal-adequacy pipeline. Round evidence is filtered against the delegation’s constant vocabulary before both progress guards read one shared function, so the stall detector and the budget refund cannot disagree about what a round measured.
Fig. 3: Signal adequacy, measured. Inside the window that killed the leading component the evidence moved monotonically toward green—one missing dictionary key from passing—while the breaker declared it unmoved, because the identifier it fingerprinted was a property of the strategy rather than of the round.
Fig. 4: A measured duplicate-effect harm. Effects committed by previous invocations of the same delegation remained visible to the current one, so a correct, idempotent component became unwinnable. This is the boundary transactional containment does not reach.
Fig. 5: Failure routing by checkpoint ladder. Spans are effect transitions rather than RPC calls; the owner of the first unproven transition is the routing target.
what the layer blockedmeasured costwhy it was wrong
Repair clamp restricted reads and writes to a mis-diagnosed target set107 turns, zero accepted writes , component lostThe one file holding the defect was outside the diagnosis; the delegation could not even read it
Husk check rejected a framework’s documented base-class idiom16 rejected writes across 4 servicesEach rejection costs a turn against the no-progress budget
Seed gate matched a handler by callee name3 corrections + 2 stronger-model rounds, component lostThe platform’s own generated dispatch seam read as “nothing is seeded”
Appeal verified a citation against only the file the model named3 real quoted lines refusedEvery frozen source is shown to the model; all must therefore be citable
Appeal accepted against a platform-generated artifactComponent killed holding a correct diagnosisA wrongly-permissive verdict forecloses every remaining route
Stall detector fingerprinted a constant identifierRun 6/6 →\rightarrow 3/6Every model-driven repair guaranteed stalled on round three
TABLE I: Enforcement-layer failures: the layer blocking correct work. Each row is a distinct production incident with a measured cost. This failure mode, not permitting a forbidden action, is the characteristic failure of an enforcement layer for agent delegation.
subsystemidentity that failed to discriminateconsequence
circuit breakerprogress fingerprint over the failing check name —a property of the strategy, identical whether four defects were fixed or noneevery model-driven repair guaranteed stalled on round three; the run went 6/6 →\rightarrow 3/6
effect ledgercommit id keyed on transaction id , so identical content from two transactions hashes differentlycannot deduplicate; the fingerprint is provenance, not identity
topology grapha logical node name collapsing two different service databases into one projectionfalse edge, false green graph
failure attributionscenario coverage used as causal identityfive components woken for a two-component fault; three bystanders regressed working code
work plannercoupling measured across components rather than per componenttwo components asserting one scenario would have had their artifact sets welded into a single slice
TABLE II: One failure, five subsystems: an identity that does not discriminate produces a confident wrong answer. The last two rows derived the same rule independently, for different objects.
Fig. 6: Delegation lifecycle under the effect contract. Effects observed at the tool and sandbox boundaries are verified against the declaration and fingerprinted into the ledger at commit. A retried or resumed delegation deduplicates at the effect boundary; the suppressed-as-duplicate verdict is what distinguishes correct deduplication from a breaker trip. The dashed path is the naive mesh policy the baseline arm measures.
Fig. 7: The declaration authority. Declarations are compiled, not hand-written per delegation, and approval is gated by a dry-run compile that refuses a declaration the platform cannot build. This is enforcement-layer exoneration at declaration altitude: prove the layer can be satisfied before committing work to it.
Fig. 8: The budget scope lattice, and the degradation ladder that replaces wall-clock kill. Budgets attach per run, component, failure fingerprint, strategy, and session; the per-fingerprint and per-component limits are separate quantities because they fail in opposite directions.
mechanismstatusevidence, or what remains
Effect declaration + transparent verificationrunning, measuredOverlay commit refuses undeclared writes; agent/subprocess writes discriminated by declaration. Divergence rate computable today and backfillable from archived workspaces.
Declaration authority + feasibility gaterunningStaged declaration, deterministic compile, dry-run admission control; refuses unsliceable units and obligations with no executable check.
Progress breaker + signal adequacyrunning, measuredConstant-vocabulary filter; both guards read one function. Pre-fix discriminating power was zero for four recovery paths.
Budget lattice + degradationrunning, measuredGraded refund on strict improvement; unlimited wall clock; one stronger-model round before the stop (four of eleven strategies wired). Per-delegation token ledger not built .
Failure routing (graph + ladder)running; success criteria unvalidatedBlast radius 5 →\rightarrow 2 measured. Two-point graph build and checkpoint ladder running; the motivating routing case is not yet observed live .
Enforcement exonerationrunning, measuredThirteen gates checked at boot; clamp yielding fired five times (previously structurally zero); unprovable guards deleted.
TABLE III: Mechanism status. The upper block is running in production; the lower block is what the evaluation still requires. We state this as a table because the paper’s central claim about itself is that the distinction is never blurred.
rung removeddefect that surfacedmeasured cost, and closure
Cluster slicingAn atomic component the planner could not slice was admitted without a size gateConvergence-complexity 120 against a budget of 6; closed by making the mode an explicit flag the feasibility gate consults
Oracle qualification and seed gatingRequired-fixture set derived as a raw union over every dependency11 unsatisfiable identifiers; recovery budget exhausted on a correct component. Closed by deriving the set rather than copying it: 11 →\rightarrow 3 real rows
Freeze and provenanceFinal verifier demanded a freeze record, and separately an optional cache entry, for every componentA fully-green, verifier-approved run failed. Closed by scoping the provenance requirement to modes that produce the artifact
(none — new boundary)Preview oracle recognised only bearer tokens while services authenticate via gateway-injected headers6 repair loops, ∼\sim 1 hour, byte-identical signature; unwinnable by repair. Closed by making the oracle authenticate as the gateway does
TABLE IV: Ablation of the verification ladder. Each defect surfaced only when its rung was removed, and each was closed at a layer that operates in both modes—so the ladder’s contribution is diagnostic rather than merely procedural. The final row is the dual result: a new evidence boundary introduced its own instance of F5.
Fig. 9: Evaluation design. One workload and one fault schedule drive two arms: the baseline runs naive service-mesh policies with transactional containment disabled—without which the harm number measures a property of our implementation rather than of agent delegation—and the mesh arm enables the seven primitives with distinct refusal verdicts.
metricstatus
Obtainable from the existing record
Delegation depth distributionreported (1/8/7 at depths 1/2/3)
Discriminating power, pre-fixreported (0 for four paths)
Declaration–observation divergencebackfillable from archived workspaces
Enforcement incidentsreported ( Table I )
Blast radius, before/afterreported (5 →\rightarrow 2)
TABLE V: Metric inventory. Five quantities are already derivable from the operating record; the rest are gated on instruments or on the two-arm study, and we say which.
 effect-levelprogress-basedfailureenforcementunit of
approachdedupbreakingattributioncorrectnessenforcement
Service mesh [ 20 , 21 , 22 ]— (key required)— (error rate)——request
Durable execution [ 16 , 24 , 25 ]enumerated boundaries— (retry policy)——activity
Agent authorization [ 1 , 2 ]———partial aaction
Transport proxies [ 23 ]————message
Distributed tracing [ 19 ]——spans (RPC)—call
TABLE VI: Capability comparison. “—” denotes not provided by the approach, not a deficiency: each row solves a different problem well. The gap this paper addresses is the empty effect-level deduplication column for operation sets generated at inference time, and the empty enforcement correctness column entirely. a Authorization systems verify that a policy is enforced, not that the enforcement layer admits correct traffic.
More in this topic

Agent Reliability

Agent reliability studies how to make AI agents behave consistently, recover safely from failures, and remain dependable under real-world uncertainty.

See all 45 papers

Explore similar work

CardsList
Sep 26, 2026

Agents as Software: A Programming Languages Agenda for Agent Reliability

AI agents increasingly resemble software systems: they call tools, remember facts, follow policies, delegate work, and take actions with real consequences. % Yet the ``program'' of an agent is scattered across prompts, tools, memories, workflows, and execution traces, making its behavior difficult to inspect through ordinary testing and debugging alone. % This essay argues that a programming-systems perspective offers a natural lens for making agents reliable. % We recast agents as programmable artifacts whose behavior can be specified over traces and state, checked before deployment, monitored during execution, and improved from observed failures. % The goal is not to make probabilistic agents behave like deterministic programs, but to give them enough structure that their behavior can be reasoned about, controlled, and repaired.

Shraddha Barke, Adithya Murali

Microsoft Research Redmond, USA · University of Wisconsin-Madison Madison, USA

AI Agent ReliabilityAI Agent MonitoringRuntime Enforcement for AI Agents
AskPDF
Aug 13, 2026

Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence

Compositional reliability bounds for multi-agent systems multiply component reliabilities, a step licensed by a conditional-independence assumption that is routinely stated and rarely tested. We test it. Two instances of one model, in a two-agent handoff, co-fail on 90.0% of the missions on which either fails (log OR 6.66, 95% CI [6.38, 7.00]; phi 0.916), in a preregistered evaluation of 18,000 missions scored by deterministic code with no LLM judge. Substituting a different model reduces the association in six of six contrasts; substituting a different vendor, model already different, does not -- a registered hypothesis reported as a null. The error is signed and runs against the operator: positive dependence inflates joint failure above the independence product, so redundancy is over-credited exactly when components share a model. The assumption-free alternative is often vacuous, and fitting a dependence model is worse: we prove a bootstrap bound on a fitted model's functional loses coverage of the truth as n grows, the identification gap being O(1) while the bootstrap haircut is O(n^{-1/2}). More data makes such a certificate worse, with no visible symptom. We give a finite-sample certificate assuming no dependence structure: a linear program over the joint, over a Bonferroni-Clopper-Pearson box around measured co-execution moments. It is sound, sharp for the information supplied, and monotone in the moment family. Enriching ten moment functionals to fourteen narrows the identified interval by 85.7% and lifts the certified floor from 0.2455 to 0.4116. A companion anytime-valid certificate holds type-I error at 0.0471 under optional stopping. Common dependence statistics are marginal-bounded and can reverse an apparent ordering of conditions when the compared agents fail at different rates. Contracts, scoring code, analysis scripts, and the preregistration are released.

Varun Pratap Bhardwaj, Garima Singh, Arun Pratap Bhardwaj

Qualixar / Independent Researcher, India · Independent Researcher, India

Agent Failure AnalysisAI Agent ReliabilityPartial IdentificationConditional Independence Testing
AskPDF
Jul 19, 2026

Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

Multi-step enterprise agent tasks fail in a characteristic way: single-pass inference has no checkpoint between deciding an answer and committing to it. We study one production system (Leni) whose architecture installs such checkpoints: verification loops (execute, observe, compare, correct) staffed by lightweight task-specialized post-trained models. We evaluate the unmodified production configuration on three public benchmarks stressing distinct failure modes: SpreadsheetBench Verified (silent computation error), BullshitBench v2 (premise confabulation), and the GAIA validation split (cascade error over long tool chains). The full system improves over its frontier base model by +11.0 percentage points on SpreadsheetBench (91.25% vs 80.25%, n=400, p<0.001), +7 to +10 percentage points on BullshitBench (98% vs 91%, n=100), and roughly +15 points on GAIA validation (75.2% pass@1, n=165; 83.0% best-of-k). Our central contribution is a decomposition of that uplift: most of it comes from scaffolding, routing, and specialist models rather than from the verification step itself, whose isolated contribution is small (+1.5 points) but concentrated at the top of the score distribution, where it converts otherwise-failing tasks. We instrument the loop end-to-end, yielding an empirical verifier confusion matrix (catch rate about 0.20, fix rate 0.75, no false-alarm regressions) that grounds a compounding-reliability model. Specialist-swap ablations suggest that the loop's value depends on who observes it: replacing the small trained verifier with the generating frontier model eliminates most rescues. A valid-premise control shows zero over-rejections in 100 expert-level questions.

Arunabh Dastidar

for the Leni Team · Leni Inc.

AI Agent ReliabilityLLM Answer VerificationLLM Agent EvaluationLLM Agent VerificationAI Agent BenchmarksLLM Agent Harnesses+1
AskPDF