Should My Website Block AI Agents or Let Them In?

Short answer: Do not block by label. Block by behaviour and account for what the traffic is trying to do. A crawler indexing your site for search is a different animal from an agent trying to check out on a customer's behalf. The tools to tell them apart (cryptographic bot identity, signed agents, agent payment protocols) are shipping in 2026, but most are young: pilots, closed betas and draft standards. As the first post in this series argues, "human vs bot" is already the wrong line. The test that matters is whether someone is accountable for a request, and whether you can bill or rate-limit them if it misbehaves. Amazon's fight with Perplexity shows how messy that gets when an agent does not identify itself.

Amazon tried to settle this in court, and lost on appeal

On March 10, 2026, Senior US District Judge Maxine Chesney, Northern District of California, granted Amazon a preliminary injunction stopping Perplexity's Comet browser from using its AI agent to shop inside password-protected parts of Amazon. She found that Comet accessed Amazon accounts "with the Amazon user's permission, but without authorization by Amazon." Amazon said it had warned Perplexity at least five times starting in November 2024, put a technical block in place in August 2025, and watched Perplexity ship an update that got around it within 24 hours. It also argued that Perplexity "deliberately disguised Comet's AI agent as a regular Google Chrome browser session" instead of identifying itself. Perplexity called the suit "a bully tactic" and said "consumers should be free to use any AI assistant they choose to shop online." (GeekWire, March 10, 2026.)

The injunction did not last. The Ninth Circuit paused it pending appeal, and on August 4, 2026 vacated it. The panel held that when a user tells the agent to act on Amazon.com, "it is the user who 'accesses' Amazon's computers," not Perplexity, so Amazon was unlikely to win under the federal anti-hacking law (the CFAA) or its California equivalent. The court was careful about scope. It said it was not creating "a new legal regime governing agentic AI," and that the ruling "does not impair Amazon's ability to regulate access to Amazon.com via private terms of service for its users." The case now goes back to the district court.

The appeals court also named the design choice at the centre of the dispute: Perplexity's decision not to send a user-agent string "that would communicate that the user has activated an AI agent." That is the difference between an agent you can make a decision about and one you cannot. For a site owner, the practical lesson is that the anti-hacking law is a weak tool against a customer's own agent. Your terms of service and your technical controls are the levers you actually have.

What makes an agent "accountable" instead of anonymous

There is now a real, if young, standards effort aimed at exactly this problem: letting a bot or agent cryptographically prove its identity instead of pretending to be a human with a browser.

The mechanism underneath it is RFC 9421, HTTP Message Signatures, an IETF Standards Track RFC published in February 2024. It defines how to attach a verifiable digital signature to parts of an HTTP request, so a server can confirm a message came from whoever holds a specific private key, not just whoever's IP address it arrived from.

Building on that, the IETF chartered a dedicated working group, Web Bot Auth, to standardise how crawlers, archivers and AI agents use that signature mechanism to identify themselves to websites. Its charter set milestones for sending standards-track specifications to the IESG, the IETF's review body, by April 30, 2026, and a best-current-practice document by August 31, 2026 (IETF Datatracker, webbotauth working group). Both dates have passed. The main working-group draft, draft-ietf-webbotauth-httpsig-protocol, is at version 00, dated September 1, 2026, written by engineers at Cloudflare and Google (IETF Datatracker). The effort is active and well backed, but it is still a draft. Nobody can hand you an "RFC-compliant Web Bot Auth" badge yet, because the RFC does not exist yet.

Cloudflare has not waited for the RFC to ship product. In August 2025 it introduced "signed agents," a category for tools directed by an individual user rather than a single company, whose infrastructure signs each HTTP request using Web Bot Auth so Cloudflare can validate the signature and expose a verified identity to its bot rules (Cloudflare Blog, The age of agents: cryptographically recognizing agent traffic, August 28, 2025). Signed agents are a separate category from Cloudflare's older Verified Bots program, though a well-behaved signed agent can also qualify as a verified bot. If you run your site behind Cloudflare, a signed request from a legitimate agent now shows up differently in your dashboard than an anonymous scraper, and you can write a rule that treats the two differently instead of guessing from a user-agent string anyone can fake.

The defaults are shifting too

Cloudflare made a bigger move on July 1, 2026: starting September 15, 2026, all new domains onboarding to Cloudflare default to blocking two categories of automated traffic, "Training" and "Agent," on pages that display ads, while "Search" crawlers stay allowed. Cloudflare splits AI traffic into three intents: Search ("any behavior that collects or indexes your content, so it can answer questions about it later"), Agent ("automated behavior that is acting, usually in real time, on a person's behalf, to get something done"), and Training ("a crawler taking your content to train or fine-tune a model"). A crawler that does more than one of these is judged on all of them, so if you block Training, multi-purpose crawlers such as Googlebot, Applebot and Bingbot get blocked too (Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, July 1, 2026). If you are already on Cloudflare, check your Security settings rather than assuming which default applies to you.

Cloudflare's payment layer for crawlers, pay per crawl, is further behind. It launched July 1, 2025 as a private experiment letting publishers charge a flat per-request price and have crawlers pay via HTTP 402 "Payment Required" responses (Cloudflare Blog, Introducing pay per crawl, July 1, 2025). As of September 2026 its own documentation still says it "is currently in closed beta," with sign-up by request, so it is not a feature you can switch on (Cloudflare Developer Docs, What is Pay Per Crawl). If a vendor pitches you "pay per crawl, live today," that is not accurate yet.

Not every AI company treats robots.txt the same way either, and this matters more than the standards fight for most small businesses. OpenAI documents several crawlers. The three that matter here are GPTBot, which collects training data and honours robots.txt; OAI-SearchBot, which indexes pages for ChatGPT search and also honours robots.txt; and ChatGPT-User, which fetches a specific page because a person, mid-conversation, asked ChatGPT to look at it. OpenAI's own developer documentation says robots.txt rules "may not apply" to ChatGPT-User, because the fetch is a direct, one-off action a user requested, not autonomous crawling (OpenAI Developer Docs, Overview of OpenAI Crawlers). That is by design. Block GPTBot to keep your content out of training data, and you still have not blocked the case where your own customer asks ChatGPT to pull up your pricing page.

"AI search is a double-edged sword"

That is Hannah Hicklen of Clutch, commenting on a survey her firm published on small businesses and AI crawlers. Her full line: "For small businesses, AI search is a double-edged sword. It can drive visibility and growth, but only if companies embrace the technology rather than block it." Sixty percent of small businesses said AI search tools like ChatGPT and Gemini were boosting revenue, with 44 percent reporting more leads, 42 percent more website traffic and 33 percent more inquiries traceable to AI search. Despite that, more than half block AI crawlers anyway, over content-ownership concerns (43 percent) and, separately, unauthorized scraping in the past year (16 percent). Seventy-one percent reported performance problems from bot traffic. Sixty-eight percent found a pay-per-crawl style model appealing, and 73 percent planned to revisit their crawler policy within the year (Clutch, Clutch Report: 62% of Small Businesses Say AI Search Boosts Revenue, Yet Most Still Block Crawlers, September 24, 2025).

Put those together and you get businesses cutting off a channel they also say is working, often with a block switched on because "AI bots" sounded like a threat.

The wider web looks different. In a May 2025 study, Ahrefs checked roughly 140 million websites and found GPTBot blocked, in some form, on 5.89 percent of sites, with explicit, GPTBot-specific rules on 0.5 percent; ClaudeBot had the fastest-growing block rate it tracked, up 32.67 percent year over year (Ahrefs, AI bot block rates). Across the whole web, blocking AI crawlers was still a minority move, even if most of the small businesses Clutch surveyed said they do it.

The agents that come with a credit card attached

This is the part that should change how you think about "block AI agents" as a policy. The card networks and payment processors are building rails for agents to pay merchants directly, and some of it is already live.

Visa introduced its Trusted Agent Protocol on October 14, 2025: an open framework, built with Cloudflare and with feedback from twelve payment and commerce companies including Adyen, Stripe, Shopify and Microsoft, that lets a merchant cryptographically verify an incoming agent request instead of guessing from behaviour. Visa says it is "available today in the Visa Developer Center and GitHub," and cites a surge of over 4,700 percent in AI-driven traffic to US retail sites over the prior year as the reason merchants need a way to tell a legitimate shopping agent from "malicious automation and rogue bots" (Visa, Visa Unveils Trusted Agent Protocol for AI Commerce, October 14, 2025).

Mastercard unveiled Agent Pay and Mastercard Agentic Tokens on April 29, 2025, extending the tokenization it already uses for card-on-file payments to purchases an agent makes. In September 2025 it said all US Mastercard cardholders would be enabled by the holiday season, with Citi and U.S. Bank cardholders first. Since then it has reported live agentic transactions in Latin America and the Caribbean, run "in controlled environments," after earlier ones in the US and several Asia Pacific markets. In Europe, Santander and Mastercard completed what they called Europe's first live end-to-end payment executed by an AI agent, a pilot the bank says is not yet a commercial rollout. In June 2026 Mastercard added Agent Pay for Machines, aimed at high-frequency payments between machines rather than consumer shopping.

Google took a different route. On September 16, 2025 it published the Agent Payments Protocol (AP2), an open protocol with its specification and reference implementations on GitHub. It can run as an extension of the Agent2Agent (A2A) protocol and the Model Context Protocol, and uses cryptographically signed "Mandates" as tamper-proof records of what a user authorized an agent to buy. AP2 launched with more than 60 partner organizations, including American Express, Coinbase, Etsy, Mastercard, PayPal, Salesforce and Worldpay (Google Cloud Blog, Announcing the Agent Payments Protocol (AP2), September 16, 2025). It is a specification with reference implementations rather than a finished product. Today, "using AP2" means a developer integrating a library, not a merchant flipping a switch.

Stripe and OpenAI shipped the most consumer-visible version of this on September 29, 2025: Instant Checkout inside ChatGPT, backed by the jointly developed, open Agentic Commerce Protocol. At launch, US ChatGPT users could already buy from Etsy sellers inside the chat window; Shopify merchants, including named brands like Glossier, Vuori, Spanx and SKIMS, were announced as "coming soon" rather than live on day one (Stripe, Stripe powers Instant Checkout in ChatGPT and releases Agentic Commerce Protocol codeveloped with OpenAI, September 29, 2025). Stripe says ACP is not Stripe-only: "businesses not processing with Stripe can still adopt it with their existing payment providers."

None of these four is finished, interoperable infrastructure yet. They overlap and compete, and adopting one does not mean supporting the others. But they are real: Mastercard reports live agent transactions across several regions, purchases have gone through Stripe and OpenAI's checkout, and a merchant blocking "AI agents" wholesale may be turning away a customer whose agent already has a way to pay.

Where blocking is still the right call

A scraper that will not identify itself and ignores robots.txt. Cloudflare's whole verified-bots model exists because plenty of bots still lie about who they are. Block, rate-limit or challenge those the way you always have.

Any agent touching a login-gated account without your say-so. This was Amazon's argument against Comet, and the Ninth Circuit's answer is a warning: under the federal anti-hacking law, the customer using their own agent is likely the one doing the accessing. The same court said terms of service remain yours to set. If you run an account area, decide whether agentic access is allowed, say so in your terms, and back it with technical controls, rather than counting on a lawsuit.

High-value actions with no identity behind them: account creation, OTP triggers, checkout on a site with no fraud tooling. An unverified agent hitting these is the same risk as an unverified bot, protocol or no protocol. The gate belongs at the action, not the front door; see how to stop bots spamming contact and signup forms.

Training crawlers on content you monetise or protect, if that matters to your business. For new domains, Cloudflare's September 2026 defaults already block training crawlers on pages that show ads.

Anything that costs you real infrastructure with no return. If a crawler hammers your server with no citation, referral traffic or payment behind it, it does not matter whether it is "AI" or any other abusive bot. Identity-based rate limits, covered in what bot protection actually costs a small business, handle this regardless of what the bot claims to be.

What this actually costs

The disclosure first: Ready Bytes has not built a bot-protection or agent-verification product. We build web apps, back-office automation and integrations for owner-led businesses, the pattern behind AI back office for small business. Nothing above is a case study. It is what we found reading the primary sources.

Most small sites do not need Web Bot Auth, Trusted Agent Protocol or a pay-per-crawl account this year. What they need is a decision, made once, about the few kinds of automated visitor that matter to their business: the AI search crawlers that decide their visibility, any checkout agent their platform is starting to support, and the forms and login flows worth protecting properly. Getting that decision right is mostly configuration, not code: Cloudflare's Content Signals settings, your robots.txt file split by bot purpose, and rate limits on the handful of endpoints that actually cost you money when abused.

If there is a scoped piece of work behind that decision, here is the ladder:

• A free AI opportunity audit at /ai-audit: fifteen to twenty questions, about five minutes, no cost. Shyam reviews it and emails a written audit within two business days.
• A $500 full audit if the free one surfaces something: read-only access, your top 3 opportunities ranked by ROI, a 90-day roadmap and a fixed pilot quote, credited against the pilot if you proceed.
• A fixed-quote pilot, typically $3,000 to $8,000 over 2 to 6 weeks, scoped to one piece. Here that is most often getting your crawler and agent settings right and wiring identity-based rate limits onto the two or three endpoints that actually matter, not a general "bot protection" build.
• An ongoing relationship after a pilot has proved itself.

The most useful outcome is often a recommendation to spend nothing, just to turn on settings you already have access to.

Start here

Do one exercise this week, before changing any setting.

Pull your server or CDN logs for the last 30 days and list every user agent responsible for more than 1 percent of requests. For each, answer three questions: does it identify itself honestly (a real user agent string, a published IP range, or a Web Bot Auth signature), what is it trying to do (index, train, answer a live question, or transact), and has it ever cost you something concrete, server load, a fraudulent order, a scraped price list. Sort the list into three piles: allow, gate at the valuable step, block.

Only then touch Cloudflare's Content Signals, a robots.txt rule, or a bot-management setting. Most of what you find will not be a swarm of malicious agents. It will be a handful of known crawlers and a growing trickle of legitimate ones acting for a real customer, and for both, the fix starts with a decision rather than a product.

---

Shyam Verma founded Ready Bytes in 2009 and has been building software since 2005. He writes about back-office automation, legacy modernization and applied AI at readybytes.in/blog.