LobstersFriday · May 15, 2026FREE

Amazonbot is finally respecting robots.txt

Amazonbot, Amazon's web crawler used for indexing products and training AI models, has finally begun respecting robots.txt rules, according to a blog post by Xe Iaso. Previously, site owners reported that Amazonbot ignored disallow directives, scraping content against their wishes. This change, observed in May 2026, means that websites can now effectively block Amazonbot by adding it to their robots.txt file. The update addresses a major point of contention in the web scraping community, where Amazon was seen as disregarding standard protocols. While the exact date of the change is unclear, Iaso notes that testing confirmed compliance. This move aligns Amazonbot with other major crawlers like Googlebot and Bingbot, which have long respected robots.txt. The consequence for developers is that they can now control Amazon's access to their content without resorting to more aggressive blocking measures.