set_indexing: Agent-Applied noindex and index Control

set_indexing is the tool the Get AI Traffic agent uses to add or remove a page’s noindex rule after you approve a fix. It writes the <meta name="robots"> tag on the pages you name, then publishes the change through your site’s connection.

This page documents what the tool touches, what it refuses to touch, and how to undo it. For the decision of whether a given page should be indexed, read How to Remove a URL From Google When Staging Pages Get Indexed and Noindex Your Thin Pages Before They Drag Down Everything Else. Those are business calls. This page is about the mechanics.

What it does

When you approve a staging_pages_indexed or thin_pages_indexed action item, the agent sets the robots meta tag on the flagged pages. Choosing noindex writes noindex, nofollow; putting a page back into the index writes index, follow. A search engine that reads a noindex drops the page from results on its next crawl, no matter who links to it.

The tool takes a list of pages, so a batch of thin pages is handled in one pass. Each page is recorded and applied in turn, and if one fails the run stops there rather than half-finishing.

Setting noindex does not touch robots.txt, and that is deliberate. A page has to stay crawlable for an engine to read the noindex at all. Block the page in robots.txt and the crawler never sees the rule, so the page can keep appearing. The tool leaves the page fetchable on purpose.

What it will not touch

• The visible page. The tool writes only the robots meta tag through the curated plugin endpoint. It cannot change your headings, copy, or layout, so approving it never puts the content itself at risk.
• Your sitemap.set_indexing sets the robots rule and nothing else. A noindexed page left in the sitemap sends a contradictory signal, so pulling those URLs out is a separate remove_sitemap_url call. The two changes are applied independently.
• robots.txt. Fetch-level blocking lives in a different file and a different tool. This one never edits robots.txt.

Timing

The rule takes effect when the crawler next visits and reads the tag, not the moment it is published. Deindexing runs on the engine’s schedule after that: days for a frequently-crawled page, weeks or months for a low-priority URL. Nothing is wrong if a page lingers in results for a while. In Search Console, the URL Inspection tool can request a recrawl to speed removal along.

Rolling back

Every applied change is written to the ledger with the prior state. roll_back restores the previous robots meta exactly, so a noindex reverts to whatever the page carried before. Because engines only re-read the tag on their next crawl, a rollback has the same lag as the original change.

Related tools

• purge_cache runs after an apply, so a cached copy of the old page does not keep serving the previous robots tag at the edge.
• remove_sitemap_url pulls a noindexed URL out of the supplemental sitemap. The thin-pages fix usually pairs the two: noindex the page, then drop it from the sitemap so the two signals agree.
• edit_robots_txt controls whether a crawler may fetch a page; set_indexing controls whether it may list a page it already fetched. They are different mechanisms, and combining both on one URL backfires: a robots.txt block hides the noindex this tool relies on.

Findings this tool resolves

• staging_pages_indexed: a staging, test, or duplicate page is indexable.
• thin_pages_indexed: thin, low-value pages are indexed and listed in the sitemap.