AI agents now browse and buy on people's behalf, and benchmarks measure how well the agents perform. Far less is known about the other side of the transaction: whether a merchant's website lets an agent in, publishes terms precise enough to decide a purchase, and leaves evidence of what was agreed. This paper presents a merchant-side benchmark built on those three questions and reports a pilot of it. The central measurement, decision sufficiency, rates each criterion of a concrete buyer mandate (total cost, cancellation or return terms, service date, variant identity) as determinate, indeterminate, or unknown from the merchant's own published evidence. Readiness scores carry their evidence coverage and missingness bounds; access is reported separately for a declared bot, a declared headless browser, and a user-present browser; and each observed term is mapped to the RAILS admissibility class of the evidence that could support it in a dispute. Four large merchants received the full assessment: three refused the declared profiles at their offer pages, and no mandate could be confirmed as met, because each merchant left a total or a refund term undetermined. An access census of 555 sites of merchants that two payment processors publicly name as clients found the home page unreadable to both declared profiles at 105 sites (18.9%). A declared assessment of 330 merchants, scored by deterministic rules without a rater, found that evidence coverage rather than the score decides who can be ranked: 115 merchants (35%) reached the ranking threshold, rankable profiles scored a median of 89 to 90, and material terms was the one pillar well below the others, because the all-in total was decidable at 18 merchants. The pilot shows that the measurements can be taken; it does not validate them or rank merchants, and the weights and the projected clearing band await validation against outcomes.
Figures & tables
Figure 1: The three questions of the benchmark and what each produces. Access is measured per agent profile; decision sufficiency is measured against a concrete buyer mandate; the evidence class of each term is reported as it stands before payment and as it would stand with merchant-signed offers, an independent witness, and processor records. The readiness score carries its evidence coverage and missingness bounds. The clearing band is an illustrative projection and is not calibrated against outcomes.
Profile
Implementation
What it represents
Declared bot
HTTP client with an identifiable user agent that names the probe and links to the operator
A crawler or protocol client that identifies itself
Declared headless browser
Headless Chromium with the same token appended to its user agent, and no measure that conceals automation
A browser agent that renders pages without hiding that it is automated
User-present browser
A browser on the person’s own device, operated by an AI browser assistant with the person present and approving each step; it sends the browser’s ordinary user agent and performs no login, hold, cart, or payment
A consumer browser agent acting for a present user
Table 1: Assessment profiles. Results are reported per profile and do not transfer between profiles.
E1 counterparty disclosed; E2 policy applies to this offer; E3 offer referenceable; E4 recourse instructions
Table 2: The twenty readiness metrics. Weights are initial design choices.
Criterion
Determinate when
Failing case from the pilot
All-in total within budget
Amount, currency, charging basis, and mandatory fees are stated for the selected configuration and dates, or a finite bound is
A “from” price with no unit, fee, or tax basis for the selected dates; a nightly rate that includes a mandatory charge and states no tax basis
Cancellation or return
Deadline, fee, exceptions, and refund method are stated for the selected rate or item before commitment
Terms deferred to the reservation confirmation; one page promising cancellation without penalty while another says certain charges are not refunded
Delivery or service date
A delivery window for the destination, or explicit service dates, is stated
No shipping or start-of-service estimate before an order flow that the permitted route may not enter
Variant identity
The price is bound to the exact variant, room type, or rate plan on the page, and structured data does not contradict it
A nightly price bound to a room type and to no rate plan; a recurring plan price while the required equipment depends on the service address
Total behind a disallowed step
The total, or a finite bound, is available before any step the permitted route may not enter
Hardware, shipping, and tax not stated before an order flow that robots.txt disallows
Table 3: Determinacy rules, with failing cases observed in the pilot (paraphrased).
Evidence source
Class
Claim it can support
The agent’s own record of what it saw
SELF
That the agent reports having seen it
A merchant-signed offer, checkout session, or confirmation
SIGN
That the merchant committed to the stated terms; non-repudiation, not truth
An independent capture of the displayed terms by a witness that meets the deployment’s witness requirements
WIT
That the merchant displayed these terms at a stated time under a stated profile; never that the terms are true or were honored
The processor’s authorization and settlement record
REC
Amount, merchant of record, time, and payment status
A carrier or fulfillment system scan
REC
A delivery or fulfillment event and its time
Table 4: Evidence sources, the RAILS admissibility class each can support, and the claim that class licenses.
Merchant and profile
Score
Coverage
Bounds
Determinate
Rule
M1 airline, declared
0.0
15%
0 to 85
none
Not met
M1 airline, user-present
90.0
100%
90 to 90
6 of 7
Met
M2 resort, declared
0.0
15%
0 to 85
none
Not met
M2 resort, user-present
64.5
95%
61.25 to 66.25
6 of 7
Not met
M3 subscription, declared headless
50.0
90%
45 to 55
3 of 7
Not met
M3 subscription, declared HTTP client
0.0
11.25%
0 to 88.75
none
Not met
Table 5: Full assessments, 29 September 2026. One mandate and one observation per merchant; verdicts adjudicated by a single rater. “Declared” covers both declared profiles where they met the same barrier at every page (M1, M2, and M4); M3 is reported per declared profile. “Determinate” counts the mandate criteria, out of seven, in the DETERMINATE state; “none” means that no criterion was observable. The last column applies the publication rule of Section 3.4 .
Figure 2: Observed readiness score (dot) and missingness bounds (bar) for the eight assessed profiles. The bar spans the score the profile would receive if every unobserved test failed and if every unobserved test passed. Two profiles with the same observed score of 50.0 (M3 declared headless and M4 declared) carry bounds of very different width.
Outcome
HTTP client
Headless browser
Readable
418 (75.3)
423 (76.2)
Challenged or refused
87 (15.7)
108 (19.5)
Empty shell
24 (4.3)
0 (0.0)
No response
22 (4.0)
20 (3.6)
Disallowed by robots.txt
4 (0.7)
4 (0.7)
Readable by neither profile
105 (18.9)
Table 6: Home-page outcomes for 555 sites, 30 September 2026, by declared profile: number of sites (percent).
Figure 3: Share of sites whose home page neither declared profile could read, with 95% Wilson intervals, for the whole sample and for the six sectors with at least 20 sites.
HTTP client
Headless browser
Profiles
330
330
Coverage
81.9%
82.7%
Observed score
87.5
87.0
Rankable profiles
103
102
observed score
89.0
90.2
bounds
86.0 to 89.6
86.3 to 90.6
Table 7: Declared assessment of 330 merchants. Pillar A observed 3 October 2026, pillars B to E on 7 October; rubric version 0.1. Values are medians over profiles unless stated. A profile is rankable at 90% coverage overall and 80% in every pillar.
Figure 4: Status of the checks of each pillar, both declared profiles pooled, 330 merchants. Pillar A counts a barrier as a failure of access; pillars B to E count it as a page not readable. Numbers are percent of the pillar’s checks.
Figure 5: Evidence coverage per pillar and overall under the declared HTTP client, 330 merchants: median (dot) and interquartile range (bar). The right column counts profiles at the threshold: 80% for a pillar, 90% overall.
Figure 6: Pillar scores of the 103 rankable profiles under the declared HTTP client: median (dot), interquartile range (bar), and mean. Material terms is the one pillar well below the others.
Figure 7: Share of merchants rankable under at least one declared profile, with 95% Wilson intervals, for the five categories with at least 20 merchants. The 59 merchants in smaller categories are not shown; 4 of them were rankable.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Measurement
Observable pass condition
A1
Readable route
An eligible offer or policy URL returns usable content through the permitted assessment route; HTTP 200 alone is insufficient.
A2
Offer discovery
A seeded eligible offer can be reached from the home page, category links, sitemap, or declared catalog route within the fixed discovery budget (three sitemap files and three candidate pages).
A3
Offer content exposure
The selected route exposes the offer name, selected configuration, and displayed price as text or typed data, and not only as an image or an unresolved shell.
A4
Navigation resolves
The next offer or policy destination is identifiable from a link or accessible control name and resolves to the intended destination.
B1
Offer identity
The record identifies the merchant or location and an unambiguous item or service, with a stable identifier where appropriate.
B2
Configuration specificity
The record binds the price to the actual variant, room and rate plan, dates, party size, quantity, modifiers, or subscription tier used in the scenario.
Appendix
Table 8: Pass conditions of the twenty readiness metrics (rubric version 0.1).
Band
Illustrative condition
CRITICAL
No mandate criterion is observable under the profile. Clearing would need a human arbiter and delayed finality.
ELEVATED
Criteria are observable, but a total, cancellation, or return term is indeterminate. Clearing would need a multi-verifier mesh and an escrow delay.
MODERATE
Every criterion is determinate, but the financially material criteria rest only on SELF or SIGN evidence. A short appeal window applies.
LOW
Every criterion is determinate, and the financially material criteria are supported at or above the floor, for example by REC and WIT. Settlement can be instant.
Appendix
Table 9: Illustrative projection of the RAILS policy band from the criterion states. The rule is not calibrated and reports no probability.
Criterion
State
Today
With added evidence
Basis
Flights and dates
DETERMINATE
SELF
WIT
Flight numbers, dates, and times shown
Passenger and fare class
DETERMINATE
SELF
WIT
One adult, standard cabin
All-in total within budget
DETERMINATE
SELF
REC
Fares stated to include taxes and carrier charges; with the bundles that include a carry-on, the total is inside the budget
Seats available
DETERMINATE
SELF
SIGN
Remaining seats at the fare shown
Full refund within 24 hours
INDETERMINATE
SELF
WIT
“Without penalty” on one page; a component of the fare “not refunded” on another
Operator of record
DETERMINATE
SELF
WIT
Operating carrier named in the site’s terms; a processor record would name the merchant of record, not the operator
Appendix
Table 10: Criterion states and evidence classes for M1, user-present profile. “Today” is the best class obtainable before payment as the site stands; “With added evidence” assumes a merchant-signed offer, an independent witness capture, and the processor’s record.
Agentic marketplaces are emerging where AI agents with varying capabilities autonomously complete specialized tasks for buyers. A major challenge of such marketplaces is that buyers cannot easily determine which agent will perform best on their tasks. Reported benchmark scores may be difficult to verify or compare across tasks, software, and budgets. We introduce LEGIT, a credentialing protocol connecting certification, reputation, and proposed marketplace allocation. Certification binds measured quality and cost per solved task to an agent configuration, task domain, evaluation budget, and evidence through a signed record. Reputation links records of past task outcomes to the same identity, subject to the reliability of the reported feedback. Buyers and agents can verify credential records and inspect optional visual profiles. Evaluations reveal cost differences between agent configurations with similar observed task success, and show that comparisons depend on the evaluation budget. These results support binding performance measurements to the tested configuration and resource limits. A complementary analysis quantifies the deposits and fees required for reputation manipulation under a stated Sybil attack model.
AI agents now hold spend authority and settle payments without per-action human confirmation. The resulting loss is often not a security failure: a counterparty with the correct domain, the correct settlement address and a genuinely delivered service can charge more than it should, and no check keyed on identity will see it. We present three artefacts for measuring and reducing that loss. First, a taxonomy of agentic commerce fraud that separates five observation levels (agent reasoning, wire, settlement rail, counterparty, principal) from the request-level and history-level evidence available at each, and records which levels can observe which attacks. Second, Agentic Commerce Bench (ACB), a benchmark of twenty fraud classes generated from production aggregates, 1,647 catalogued service operations and 1,068 settlements, of which six involve a counterparty that is exactly who it claims to be. Third, gordonguard, an open-source detector stack and offline harness with which an operator can audit an agent configuration, replay hostile counterparties without an account, and run the same detectors inline. Calibrating to a stated false-positive budget on clean training traffic gives a 6.5% clean flag rate, replicated across three independent generations, and leaves eight of twenty classes no better than chance. On the four classes a reasoning layer can observe, a widely used agent security scanner run over its jailbreak-detection panel scores zero on all four, while correctly scoring 1.0 on a jailbreak supplied as a control. A measured median payment of $0.007 places a hard constraint on deployment: one human review costs 143 times the value of the payment it examines.
As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.