Seonglae ChoAI Crawling
- also defenderfirecrawlfirecrawl • Updated 2026 Sep 18 17:14
Blocker
Enable indexing but not training
Have it both ways: stay discoverable in search while disallowing AI training
Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft.
https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
Crawl entire websites
Crawl entire websites with a single API call using Browser Rendering
Browser Rendering's new /crawl endpoint lets you submit a starting URL and automatically discover, render, and return content from an entire website as HTML, Markdown, or structured JSON.
https://developers.cloudflare.com/changelog/post/2026-03-10-br-crawl-endpoint/
Declare your AIndependence: block AI bots, scrapers and crawlers with a single click
To help preserve a safe Internet for content creators, we’ve just launched a brand new “easy button” to block all AI bots. It’s available for all customers, including those on our free tier.
https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click?fbclid=IwZXh0bgNhZW0CMTEAAR1ccsG_uJCrxeLyF4WlAi4gm55SqVFMcYTS0aESfFDS8QTsSKbmocdBvFU_aem_wWcwmZsZOtUf-p-lCK5F6g
Introducing GPT Crawler - Turn Any Site Into a Custom GPT With Just a URL
Discover how to create a custom GPT in minutes using this new open-source tool. Learn to crawl websites for tailored AI assistance in your apps
https://www.builder.io/blog/custom-gpt
Defense
Block AI scrapers with Anubis
I got tired with all the AI scrapers that were bullying my git server, so I made a tool to stop them for good.
https://xeiaso.net/blog/2025/anubis/
