Aggressive AI scrapers are making it kinda suck to run wikis
The Weird Gloop team, which hosts several large wikis including RuneScape Wiki, has published a blog post detailing the impact of aggressive AI scrapers on their infrastructure. These scrapers, often used to train large language models, ignore robots.txt directives and scrape at rates that degrade site performance for human users. The team notes that the scrapers bypass caching layers, hitting the database directly and causing increased server load and costs. They have had to implement aggressive rate limiting, blocklists, and even serve captchas to mitigate the issue. The post highlights a broader trend where AI companies prioritize data collection over respecting web standards, leading to a 'tragedy of the commons' scenario for community-run wikis. The team calls for better enforcement of scraping policies and more responsible behavior from AI developers.
AI scrapers threaten the sustainability of community-run wikis by increasing operational costs and degrading user experience.