Websites Blocking Known AI Data Scraper Bots in Robots.txt

Share of top 1,000 websites restricting each bot in their robots.txt files

Source: Dark Visitors, January 2025; note: includes the 16 bots known to crawl and scrape web data for AI training; Dark Visitors started tracking in June 2024