Turn off Cloudflare's Robots.txt configuration

Cloudflare's Robots.txt configuration feature can generate and serve its own robots.txt for your domain, overriding the one served by your site. If it's left On, AI crawlers may keep getting blocked even after you've fixed permissions elsewhere, because Cloudflare's version wins over ours. This guide shows how to turn it off so your site's own robots.txt is served instead.

Why this happens

Cloudflare can manage robots.txt at the edge, independent of what your origin serves. When this setting is On, Cloudflare's generated file — not your site's — is what crawlers actually see. Your site's robots.txt is built to explicitly allow AI crawlers like GPTBot and ClaudeBot; Cloudflare's default doesn't account for that, so it can silently reintroduce blocks you already resolved.

Turn off Robots.txt configuration

1. From your Cloudflare dashboard, select your domain.
2. On the Overview page, find the Manage your robots.txt card on the right side.
3. Open the dropdown below it and select Disable robots.txt configuration.

Once it's disabled, Cloudflare stops serving its own robots.txt and your domain falls back to the one your site generates for you.

Re-check your permission status

1. Go to Settings → Cloudflare in your dashboard.
2. On the AI Crawler Access card, click Check permissions.

If AI crawlers are still shown as blocked right after turning this off, wait a few seconds and click Check permissions again — Cloudflare's settings can take a moment to propagate.