Bar Graph listing days. Most days are low but there is a dramatic spike
Anyone who runs a website and actually tracks their bandwidth has probably had the same experience as I have: Things are humming along, numbers look ordinary, and then out of nowhere there’s an explosion of traffic burning through your allotted bandwidth. None of this traffic is coming through organically either. Like a page will be accessed, but none of the images, scripts, or associated files will get loaded. It will ignore any restrictions set in a robots.txt file, and sometimes hammer the same page hundreds of time in a minute. The user agent will pretend to be Chrome on a Mac, but if you check the IP it will trace back to a data center.

It’s AI bots, scraping your data.

And it’s not the mainstream ones. ChatGPT, Claude and Google tell you who they are in their user agent string. These ones are some shady models being used for who the heck knows what. And it’s just… frustrating. It ends up resembling a Distributed Denial of Service attack, and if you’ve noticed instability on my website recently, that’s why.

It’s hard enough figuring out what to do about the “honest” AI companies. So many people are relying on stuff like ChatGPT for search, part of me doesn’t want to hide my stuff in the hopes there will be discoverability, but I also don’t want the models training on my data. I find myself favoring telling those systems to not index my site.

But it’s frustrating.

Honestly I don’t know what the solution is. I end up spending time playing IP block whack-a-mole when they hit my site, and it burns through the little free time I have. I spent the time I meant to spend writing this weekend dealing with these crawlers and scrapers instead. That graph attached to this post is my bandwidth over the last few days, and you can see when the bots hit. It’s… it’s not great.

And I hate it.

View Comments