Some of the most blocked bots

Top 30 scraping bots (user agents) mentioned in top news websites’ robots.txt files. Taken from a sample of 1,169 news websites on Aug. 16-17.