AI Crawler Blocker

Choose what you want to block

Not every AI crawler visits your website for the same reason. Some collect content for model training, others support AI search, and some only open a page when a person asks an assistant or agent to retrieve it.

AI Crawler Blocker groups these crawlers by purpose, so you do not have to build and maintain the list yourself. The plugin adds your chosen rules to robots.txt whenever Publii generates the website. You never need to edit the file by hand, and any rules already in it are preserved.

Important: Blocking through robots.txt depends on crawlers respecting its rules. It cannot stop services that choose to ignore them.

Four profiles for different levels of protection

Start with the profile that matches the way you publish:

• Strict blocks every AI-related crawler included with the plugin.
• Training only blocks crawlers used for model training, datasets, scraping, and data collection. AI search services and assistants triggered by a user remain available.
• Individual crawlers lets you decide separately which of the 160 crawlers should be blocked.
• Custom lets you block complete categories, such as training crawlers, AI search services, assistants and agents, or other AI-related crawlers.

Strict provides the broadest protection. Training only is a more balanced option when you want to discourage the use of your content for model training without giving up possible visibility and referrals from AI search.

Make exceptions or add your own crawlers

When you choose Individual crawlers, the complete list is blocked by default. Filter it by category, find the crawler you want to allow, and turn off its switch.

You can also allow selected crawlers across every profile or add identifiers that are not included in the built-in list. Each custom entry can include a private note, which is useful when you return to the settings later.

Optional noai signals

The plugin can add noai and noimageai metadata to your pages without changing their existing index, follow, or noindex settings.

These values provide an extra preference signal, but support varies between services. They do not replace the rules in robots.txt.

Regular search engines stay available

The four protection profiles target AI-related crawlers only. Googlebot, Bingbot, and other regular search crawlers continue to follow your existing settings.

If you need to block every compliant crawler, you can enable Emergency Lockdown. This also blocks traditional search engines and may remove the website from search results, so it should only be used when that is your intention.

Part of the normal Publii workflow

AI Crawler Blocker updates robots.txt when Publii generates your website. Local Preview support can be enabled if you want to check the file before publishing. Once the website is synchronized, you can find it at your-domain.com/robots.txt.