cs.NISep 12, 2026

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

Authors: Rajarshi Chowdhury

Abstract

The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-purpose machine-access terms, plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. We define what the exchange can enforce, audit, and leave to contract. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.

Explore similar work

CardsList
  1. Towards an Agent-First Web: Redesigning the Web for AI Agents

    Jun 17, 2026Eranga Bandara, Ross Gore, Ravi Mukkamala +18Web AgentsWeb

  2. Skim: Speculative Execution for Fast and Efficient Web Agents

    May 15, 2026Mike Wong, Kevin Hsieh, Suman Nath +1Web AgentsWeb