The Digital Press

AI web crawlers are aggressively scraping websites to feed large language models, causing severe traffic spikes that degrade site performance and increase costs. This poses significant challenges for website operators, especially smaller ones, as traditional defenses often fail against these advanced bots.

Why it matters: AI bots generate up to 30% of global web traffic, overloading websites and harming user experience with slow load times and downtime.

The stakes: Sites face higher operational costs and visitor loss; smaller sites risk being knocked offline entirely by AI crawler traffic surges.

The other side: Traditional bot defenses like robots.txt and CAPTCHAs are often ineffective against AI crawlers, which can bypass or ignore them.

Commenters say: Many express frustration over intrusive CAPTCHAs, unsustainable hosting costs from bot traffic, and the risk of a fragmented, paywalled internet replacing the open web.