Google on Robots.txt: AI Crawlers, Emojis & Rules

What Happened

In the rapidly shifting landscape of Generative Engine Optimization (GEO), clarity is hard to come by. However, Search Engine Roundtable reports a major update from Google's own experts. Gary Illyes and Chery Prommawin have released the second installment of their "Non-Canonical Cabinet" video series. Titled "What can you actually put in a robots.txt?", the video dives deep into technical weeds to clarify what is and isn't supported by Google's crawler, while simultaneously raising a red flag for businesses relying on these files to control AI visibility.

Key Details

The video’s core message hinges on a distinction between what Google supports and what other crawlers—including AI bots like Perplexity, Claude, and ChatGPT—might interpret. Here are the critical takeaways from the discussion:

• International URLs and Emojis: The experts confirmed that Google’s crawler supports international URLs and even emojis within the robots.txt file. This flexibility allows for more creative directives without risking syntax errors breaking the file.
• Google vs. The Rest: While Google adheres to a specific set of rules and standards for robots.txt, other search engines and AI crawlers operate on their own interpretations. A directive that tells Google to ignore a page might be completely ignored by an AI model trying to scrape that page for training data.
• The "Supported" Definition: Just because a directive is technically supported by Google doesn’t mean it’s universally enforced across the web. The video emphasizes that robots.txt is a set of requests, not guarantees.
• Legacy vs. Modern: The discussion touched upon the differences between older, legacy directives and newer, more complex rules. Understanding this distinction is key for maintaining a healthy site architecture.

What It Means For Your Business

For local and small businesses, this news is a wake-up call. Your strategy to control your digital footprint cannot rely solely on traditional SEO tools like robots.txt anymore.

1. AI Crawlers Ignore Robots.txt Frequently: Many AI models scrape the open web to learn. They often do not respect the User-agent: * or Disallow: rules found in a standard robots.txt file. If you use robots.txt to hide a page from Google, but an AI crawler ignores it, that page might still be used to generate an answer about your business in a chatbot.

2. Risk of Blocking Your Own Visibility: If you block a page to prevent duplicate content or thin content from being indexed by Google, you might inadvertently prevent your best content from being cited by AI tools if those tools bypass the block.

3. The Need for Content Strategy: Since we cannot rely on technical barriers like robots.txt to stop AI access, businesses must focus on what is on the page. If you want to be cited by AI, you need high-quality, structured, and authoritative content. Technical barriers are becoming less effective against AI models that prioritize raw data access.

4. Diversify Your Controls: While robots.txt is still vital for managing crawl budgets and security, it should not be your only line of defense for controlling AI citations. You must focus on schema markup and content quality to ensure you are the source the AI chooses to cite.

In short, the era of "set it and forget it" technical SEO is over. As AI crawlers become more sophisticated, your business must adapt its strategy to ensure it is both visible to search engines and controlled in the answers generated by AI.