edit_robots_txt: Agent-Applied robots.txt Changes

edit_robots_txt is the tool the Get AI Traffic agent uses to change your robots.txt after you approve a fix. It edits the rules that control which crawlers can read your site, then publishes the file through your site’s connection.

This page documents what the tool touches, what it refuses to touch, and how to undo it. For the decision of whether you should unblock a given crawler, read GPTBot, ClaudeBot, and the AI Crawlers Your robots.txt Is Blocking. That question is a business call. This page is about the mechanics.

What it does

When you approve a blocked_crawlers action item, the agent edits your robots.txt so the AI search crawlers can read the pages you want cited. In practice that means removing or narrowing the Disallow rules that hit the search and retrieval bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user fetchers.

The agent does not blindly open everything. It targets the rules costing you AI visibility, not the ones doing real work.

What it will not touch

• Your private paths. Existing rules for admin paths, carts, checkout, and member areas are left alone. The tool narrows crawler blocks, it does not remove path protections.
• Training bots you deliberately block. If you block GPTBot or Google-Extended on purpose, approving a search-bot fix does not silently reopen them. Training and search are separate decisions and the tool treats them separately.
• Anything above robots.txt. A WAF rule, a firewall, or a Cloudflare bot setting can block a crawler without appearing in robots.txt. This tool cannot see or change those. If you unblock here and stay invisible, the block is at a layer this tool does not reach.

Timing

robots.txt is read on the crawler’s next visit, so the change takes effect when the bots come back, not instantly. Re-crawl and re-indexing take time after that. Expect this to pay off over weeks, not minutes. Nothing is wrong if your AI traffic doesn’t move the next day.

Rolling back

Every applied change is written to the ledger with the prior state. roll_back restores the previous robots.txt exactly. Because crawlers only re-read the file on their next visit, a rollback has the same lag as the original edit.

Related tools

• purge_cache runs after an apply, so a cached copy of the old robots.txt does not linger at the edge.
• set_indexing controls noindex at the page level, which is a different mechanism. robots.txt controls whether a crawler may fetch a page. noindex controls whether it may list a page it already fetched. A page blocked in robots.txt can still be indexed if other sites link to it, because Google can see the link without fetching the page.

Findings this tool resolves

• blocked_crawlers: an AI search crawler is disallowed in robots.txt.
• missing_sitemap_directive: robots.txt does not point to your sitemap.
• missing_robots: no robots.txt exists at the domain root.