cs.AIOct 12, 2026
SaveGet In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce
Organizations: Evolutionairy AI Toronto, Canada
Abstract
AI agents now browse and buy on people's behalf, and benchmarks measure how well the agents perform. Far less is known about the other side of the transaction: whether a merchant's website lets an agent in, publishes terms precise enough to decide a purchase, and leaves evidence of what was agreed. This paper presents a merchant-side benchmark built on those three questions and reports a pilot of it. The central measurement, decision sufficiency, rates each criterion of a concrete buyer mandate (total cost, cancellation or return terms, service date, variant identity) as determinate, indeterminate, or unknown from the merchant's own published evidence. Readiness scores carry their evidence coverage and missingness bounds; access is reported separately for a declared bot, a declared headless browser, and a user-present browser; and each observed term is mapped to the RAILS admissibility class of the evidence that could support it in a dispute. Four large merchants received the full assessment: three refused the declared profiles at their offer pages, and no mandate could be confirmed as met, because each merchant left a total or a refund term undetermined. An access census of 555 sites of merchants that two payment processors publicly name as clients found the home page unreadable to both declared profiles at 105 sites (18.9%). A declared assessment of 330 merchants, scored by deterministic rules without a rater, found that evidence coverage rather than the score decides who can be ranked: 115 merchants (35%) reached the ranking threshold, rankable profiles scored a median of 89 to 90, and material terms was the one pillar well below the others, because the all-in total was decidable at 18 merchants. The pilot shows that the measurements can be taken; it does not validate them or rank merchants, and the weights and the projected clearing band await validation against outcomes.
Figures & tables
Figure 1: The three questions of the benchmark and what each produces. Access is measured per agent profile; decision sufficiency is measured against a concrete buyer mandate; the evidence class of each term is reported as it stands before payment and as it would stand with merchant-signed offers, an independent witness, and processor records. The readiness score carries its evidence coverage and missingness bounds. The clearing band is an illustrative projection and is not calibrated against outcomes.
| Profile | Implementation | What it represents |
|---|---|---|
| Declared bot | HTTP client with an identifiable user agent that names the probe and links to the operator | A crawler or protocol client that identifies itself |
| Declared headless browser | Headless Chromium with the same token appended to its user agent, and no measure that conceals automation | A browser agent that renders pages without hiding that it is automated |
| User-present browser | A browser on the person’s own device, operated by an AI browser assistant with the person present and approving each step; it sends the browser’s ordinary user agent and performs no login, hold, cart, or payment | A consumer browser agent acting for a present user |
Table 1: Assessment profiles. Results are reported per profile and do not transfer between profiles.
| Pillar | Weight | Metrics |
|---|---|---|
| A. Access and discovery | 15% | A1 readable route; A2 offer discovery; A3 offer content exposure; A4 navigation resolves |
| B. Offer identification | 25% | B1 offer identity; B2 configuration specificity; B3 price interpretation; B4 availability |
| C. Material terms | 25% | C1 cost components disclosed; C2 total cost determinable; C3 delivery or service determinacy; C4 return or cancellation determinacy |
| D. Observable operability | 20% | D1 action target identifiable; D2 selection state visible; D3 prerequisites visible; D4 commerce handoff resolves |
| E. Evidence and recourse | 15% | E1 counterparty disclosed; E2 policy applies to this offer; E3 offer referenceable; E4 recourse instructions |
Table 2: The twenty readiness metrics. Weights are initial design choices.
| Criterion | Determinate when | Failing case from the pilot |
|---|---|---|
| All-in total within budget | Amount, currency, charging basis, and mandatory fees are stated for the selected configuration and dates, or a finite bound is | A “from” price with no unit, fee, or tax basis for the selected dates; a nightly rate that includes a mandatory charge and states no tax basis |
| Cancellation or return | Deadline, fee, exceptions, and refund method are stated for the selected rate or item before commitment | Terms deferred to the reservation confirmation; one page promising cancellation without penalty while another says certain charges are not refunded |
| Delivery or service date | A delivery window for the destination, or explicit service dates, is stated | No shipping or start-of-service estimate before an order flow that the permitted route may not enter |
| Variant identity | The price is bound to the exact variant, room type, or rate plan on the page, and structured data does not contradict it | A nightly price bound to a room type and to no rate plan; a recurring plan price while the required equipment depends on the service address |
| Total behind a disallowed step | The total, or a finite bound, is available before any step the permitted route may not enter | Hardware, shipping, and tax not stated before an order flow that robots.txt disallows |
Table 3: Determinacy rules, with failing cases observed in the pilot (paraphrased).
| Evidence source | Class | Claim it can support |
|---|---|---|
| The agent’s own record of what it saw | SELF | That the agent reports having seen it |
| A merchant-signed offer, checkout session, or confirmation | SIGN | That the merchant committed to the stated terms; non-repudiation, not truth |
| An independent capture of the displayed terms by a witness that meets the deployment’s witness requirements | WIT | That the merchant displayed these terms at a stated time under a stated profile; never that the terms are true or were honored |
| The processor’s authorization and settlement record | REC | Amount, merchant of record, time, and payment status |
| A carrier or fulfillment system scan | REC | A delivery or fulfillment event and its time |
Table 4: Evidence sources, the RAILS admissibility class each can support, and the claim that class licenses.
| Merchant and profile | Score | Coverage | Bounds | Determinate | Rule |
|---|---|---|---|---|---|
| M1 airline, declared | 0.0 | 15% | 0 to 85 | none | Not met |
| M1 airline, user-present | 90.0 | 100% | 90 to 90 | 6 of 7 | Met |
| M2 resort, declared | 0.0 | 15% | 0 to 85 | none | Not met |
| M2 resort, user-present | 64.5 | 95% | 61.25 to 66.25 | 6 of 7 | Not met |
| M3 subscription, declared headless | 50.0 | 90% | 45 to 55 | 3 of 7 | Not met |
| M3 subscription, declared HTTP client | 0.0 | 11.25% | 0 to 88.75 | none | Not met |
Table 5: Full assessments, 29 September 2026. One mandate and one observation per merchant; verdicts adjudicated by a single rater. “Declared” covers both declared profiles where they met the same barrier at every page (M1, M2, and M4); M3 is reported per declared profile. “Determinate” counts the mandate criteria, out of seven, in the DETERMINATE state; “none” means that no criterion was observable. The last column applies the publication rule of Section 3.4 .
Figure 2: Observed readiness score (dot) and missingness bounds (bar) for the eight assessed profiles. The bar spans the score the profile would receive if every unobserved test failed and if every unobserved test passed. Two profiles with the same observed score of 50.0 (M3 declared headless and M4 declared) carry bounds of very different width.
| Outcome | HTTP client | Headless browser |
|---|---|---|
| Readable | 418 (75.3) | 423 (76.2) |
| Challenged or refused | 87 (15.7) | 108 (19.5) |
| Empty shell | 24 (4.3) | 0 (0.0) |
| No response | 22 (4.0) | 20 (3.6) |
| Disallowed by robots.txt | 4 (0.7) | 4 (0.7) |
| Readable by neither profile | 105 (18.9) | |
Table 6: Home-page outcomes for 555 sites, 30 September 2026, by declared profile: number of sites (percent).
Figure 3: Share of sites whose home page neither declared profile could read, with 95% Wilson intervals, for the whole sample and for the six sectors with at least 20 sites.
| HTTP client | Headless browser | |
| Profiles | 330 | 330 |
| Coverage | 81.9% | 82.7% |
| Observed score | 87.5 | 87.0 |
| Rankable profiles | 103 | 102 |
| observed score | 89.0 | 90.2 |
| bounds | 86.0 to 89.6 | 86.3 to 90.6 |
Table 7: Declared assessment of 330 merchants. Pillar A observed 3 October 2026, pillars B to E on 7 October; rubric version 0.1. Values are medians over profiles unless stated. A profile is rankable at 90% coverage overall and 80% in every pillar.
Figure 4: Status of the checks of each pillar, both declared profiles pooled, 330 merchants. Pillar A counts a barrier as a failure of access; pillars B to E count it as a page not readable. Numbers are percent of the pillar’s checks.
Figure 5: Evidence coverage per pillar and overall under the declared HTTP client, 330 merchants: median (dot) and interquartile range (bar). The right column counts profiles at the threshold: 80% for a pillar, 90% overall.
Figure 6: Pillar scores of the 103 rankable profiles under the declared HTTP client: median (dot), interquartile range (bar), and mean. Material terms is the one pillar well below the others.
Figure 7: Share of merchants rankable under at least one declared profile, with 95% Wilson intervals, for the five categories with at least 20 merchants. The 59 merchants in smaller categories are not shown; 4 of them were rankable.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Measurement | Observable pass condition |
|---|---|---|
| A1 | Readable route | An eligible offer or policy URL returns usable content through the permitted assessment route; HTTP 200 alone is insufficient. |
| A2 | Offer discovery | A seeded eligible offer can be reached from the home page, category links, sitemap, or declared catalog route within the fixed discovery budget (three sitemap files and three candidate pages). |
| A3 | Offer content exposure | The selected route exposes the offer name, selected configuration, and displayed price as text or typed data, and not only as an image or an unresolved shell. |
| A4 | Navigation resolves | The next offer or policy destination is identifiable from a link or accessible control name and resolves to the intended destination. |
| B1 | Offer identity | The record identifies the merchant or location and an unambiguous item or service, with a stable identifier where appropriate. |
| B2 | Configuration specificity | The record binds the price to the actual variant, room and rate plan, dates, party size, quantity, modifiers, or subscription tier used in the scenario. |
Table 8: Pass conditions of the twenty readiness metrics (rubric version 0.1).
| Band | Illustrative condition |
|---|---|
| CRITICAL | No mandate criterion is observable under the profile. Clearing would need a human arbiter and delayed finality. |
| ELEVATED | Criteria are observable, but a total, cancellation, or return term is indeterminate. Clearing would need a multi-verifier mesh and an escrow delay. |
| MODERATE | Every criterion is determinate, but the financially material criteria rest only on SELF or SIGN evidence. A short appeal window applies. |
| LOW | Every criterion is determinate, and the financially material criteria are supported at or above the floor, for example by REC and WIT. Settlement can be instant. |
Table 9: Illustrative projection of the RAILS policy band from the criterion states. The rule is not calibrated and reports no probability.
| Criterion | State | Today | With added evidence | Basis |
|---|---|---|---|---|
| Flights and dates | DETERMINATE | SELF | WIT | Flight numbers, dates, and times shown |
| Passenger and fare class | DETERMINATE | SELF | WIT | One adult, standard cabin |
| All-in total within budget | DETERMINATE | SELF | REC | Fares stated to include taxes and carrier charges; with the bundles that include a carry-on, the total is inside the budget |
| Seats available | DETERMINATE | SELF | SIGN | Remaining seats at the fare shown |
| Full refund within 24 hours | INDETERMINATE | SELF | WIT | “Without penalty” on one page; a component of the fare “not refunded” on another |
| Operator of record | DETERMINATE | SELF | WIT | Operating carrier named in the site’s terms; a processor record would name the merchant of record, not the operator |
Table 10: Criterion states and evidence classes for M1, user-present profile. “Today” is the best class obtainable before payment as the site stands; “With added evidence” assumes a merchant-signed offer, an independent witness capture, and the processor’s record.
More in this topic
Agentic Commerce
Agentic commerce studies autonomous AI agents that discover, recommend, negotiate, and transact goods or services for consumers and businesses.
See all 45 papers