Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot
The Agentic Web Index, as presented on Known Agents, provides a detailed overview of AI bot traffic statistics, highlighting the rapid evolution of the internet from a human-centric environment to one increasingly dominated by machines. The index categorizes AI-related activity into several types, including AI scraping, AI fetching, AI search indexing, and AI browsing, and also covers robots.txt compliance, spoofing, and security. It offers insights into how AI agents, crawlers, scrapers, and other bots are reshaping the way information is discovered, accessed, and used across the web. The index includes sections on AI chat referrals, methodology, and a FAQ, suggesting a structured approach to understanding the impact of AI bots. The overview emphasizes the shift in web usage, indicating that machine-driven traffic is becoming a significant factor in web analytics and security considerations.
Developers need to understand AI bot traffic patterns to manage scraping and compliance effectively.