Blog Playbooks · Who Keeps Score · Part 4
Amazon locks out the AI agents, but not Googlebot
Amazon's robots.txt tells 99 named bots to stay out, including AI crawlers and shopping agents from Google, OpenAI, Meta, Anthropic and Perplexity. Google's search crawler is not on the list. Part 4 of Who Keeps Score.
Every website keeps a short text file at its front door called robots.txt. It is a list of house rules for software: which crawlers may come in, and which rooms they should stay out of. Most owners have never opened theirs. Amazon clearly has.
We opened Amazon's this morning, October 24, 2026, at amazon.com/robots.txt. It is a plain list of names, and a long one.
What the file says today
The file starts with a set of rules for every visitor, written as User-agent: *. Those rules keep all crawlers out of private areas such as /gp/cart and /gp/sign-in. Below that come the named entries. We counted 99 separate bot names, and every one gets the same two lines:
User-agent: GoogleAgent-Shopping
Disallow: /
Disallow: / means "not one page." Here are some of the names Amazon lists that way, copied from the file:
- Google: Google-Extended, GoogleAgent-Shopping, GoogleAgent-Mariner, Gemini-Deep-Research, Google-NotebookLM, GoogleOther, Google-CloudVertexBot, Google-Firebase
- OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
- Meta: meta-externalagent, meta-externalfetcher, meta-webindexer
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web
- Perplexity: PerplexityBot, Perplexity-User
- Others: CCBot, Bytespider, DeepSeekBot, GrokBot, xAI-Grok, Copilot, Manus-User and dozens more
Googlebot, the crawler that builds Google's ordinary search results, does not appear anywhere in the file. It falls under the general rules, which let it read product pages like any search engine. Bingbot is not named either.
Source: Our count of amazon.com/robots.txt, fetched October 24, 2026. 99 named bots carry Disallow: / in total.
So the same company that stays open to Google's search crawler shuts out Google's shopping agent and its AI research tools. Many observers read that as Amazon staying in search while keeping AI tools off its shelves. Amazon has not explained the list entry by entry. As reported by PPC Land, the AI entries were added in batches in August and November 2025.
This is a snapshot. Robots.txt files change without notice, so anyone quoting this should open it again on the day they read this.
The pop-up and the lawsuit
The file is one part of a wider stance. In September, Amazon began blocking Meta's new Muse agent from shopping on Amazon.com, with a pop-up citing its Conditions of Use. We covered that standoff, and the front-desk problem it exposes, in Who gets through the door. We won't repeat it here.
The court case tells you more about where the line sits. Amazon sued Perplexity in November 2025 over Comet, Perplexity's AI browser, which can shop Amazon for a logged-in user. Judge Maxine Chesney of the Northern District of California granted Amazon a preliminary injunction in March 2026. On August 4, 2026, the Ninth Circuit vacated it.
It is the user who "accesses" Amazon's computers, with the help of the Assistant to carry out specific acts on Amazon.com.
The panel decided a narrow question: whether Perplexity was likely to have broken the federal anti-hacking law (the CFAA) and its California counterpart. It said no, on the record in front of it, and it was careful about the limits. "We do not establish a new legal regime governing agentic AI," the opinion says. A footnote adds that the outcome "does not impair Amazon's ability to regulate access to Amazon.com via private terms of service for its users."
On September 10, 2026, the court denied Amazon's petition for rehearing by the full court. The case went back to the district court for further proceedings. Nothing has been decided on the merits yet, so the case is still open, and Amazon's terms of service remain its main tool. This is not legal advice; it is our plain reading of public filings.
A sign on the door, not a lock
Robots.txt is a request. Well-behaved crawlers read it and follow it. Nothing forces them to. And for agents that act on a person's behalf, the companies themselves say the rules may work differently. OpenAI's own documentation says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply."
That is why Amazon backs its file with a lawsuit, a terms-of-service section and technical blocks. A gym or a med spa has none of those, and does not need them. What you do need is to know what your sign currently says, because someone chose it, and it may not have been you.
A default you inherited
Your web platform or developer set up robots.txt years ago. Nobody on your team has read it. It may block AI assistants you would welcome, or let in crawlers you never thought about. When a customer's assistant can't read your schedule, nobody knows why.
A choice you made
You opened the file, decided which kinds of bots may read your schedule, prices and locations, and wrote that down. You know what it says, who changes it, and you check it again after every site update.
Decide on purpose
Copying Amazon would be a mistake for most local brands. A 12-location studio brand wants the opposite of what Amazon wants from most of these bots: it wants an assistant to read the class schedule, the intro offer and the nearest location, and repeat them accurately. But it should be a decision, made by someone who knows what each name does.
- Open your file
Type your domain followed by /robots.txt into a browser. If you see a wall of unfamiliar names, or nothing at all, write that down.
- Sort the names into three kinds
Search crawlers that build results pages. AI crawlers that collect pages in bulk, often for training. Assistants that fetch a page because a person just asked. Each company documents its own names; OpenAI, for example, lists GPTBot, OAI-SearchBot and ChatGPT-User separately.
- Decide per kind, per page
Most fitness and wellness brands will want search crawlers and person-driven assistants reading the schedule, pricing, locations and FAQs. Training crawlers are a judgment call. Member logins and checkout stay closed to everyone, the way Amazon closes its cart.
- Write it down and assign an owner
One person, in-house or at your agency, owns the file. Any change goes through them.
- Check it after every site change
A theme update or a new security plugin can rewrite the file. Look again, and keep a dated copy.
If you missed the start, Part 1 covers Amazon's other quiet change: order emails that stopped saying what you bought. Next, in Part 5, we look at the line Shopify and Amazon both drew at the checkout, and what it means for your own join and booking flow. For the practical side of letting good agents in, see Who gets through the door.
Sources
- Amazon, robots.txt, fetched October 24, 2026. amazon.com
- U.S. Court of Appeals for the Ninth Circuit, Amazon.com Services, LLC v. Perplexity AI, Inc., No. 26-1444, opinion, August 4, 2026. cdn.ca9.uscourts.gov
- U.S. Court of Appeals for the Ninth Circuit, order denying rehearing en banc, No. 26-1444, September 10, 2026, as posted by Courthouse News. courthousenews.com
- OpenAI, "Overview of OpenAI Crawlers." developers.openai.com
- PPC Land, "Amazon blocks Meta's Muse AI agent from shopping on its site," September 2026, relaying Bloomberg and reporting the dates of Amazon's robots.txt changes. ppc.land
- Troutman Pepper Locke, "Ninth Circuit Holds Human User, Not Developer of AI Agent, Responsible for Websites Accessed," August 2026. troutmanprivacy.com
