The "AI" companies are stealing data to train their models, and the authorities in most countries are seemingly A-OK with this. Considering I spent half my childhood being forced to watch anti-piracy ads on VHS/DVDs before I could watch my legitimate purchase, this does smart somewhat. My license is clear on the website, as on many others, but it's irrelevant - companies are flouting copyright regulations.

Now I am conflicted on this, as the way I learn as a human is by copying other people and then adapting it to myself. I'm quite good at copying, in fact, and it's how I've learnt to play music, write stories, or take photos. You look at how others have done it before you and learn to do it their way, but what you end up with over time is very different. If AI models are to be trained that way then it is oddly human, albeit on a vastly different scale. Whilst I've learnt how to play the diatonic accordion from emulating maybe 10-20 people, they've learnt to write by absorbing billions of texts.

I am pretty much resigned to everything I've ever written or published online already being part of these models, considering the barrier to them being scraped is so low. Most websites now receive vastly more traffic from scrapers than they do legitimate users, and while some of these scrapers are ones we want (such as Googlebot), others are just harvesting everything they can get their hands on and providing nothing in return. The cost is so negligible that large companies like Meta have entire clusters of high-power servers harvesting millions of pages per day; I should know, they've sent over 1 million of those requests to this small portfolio site in the past two weeks.

It's such a shame that all they got in return were some sexy toasters.

Good Bots

If you're not particularly technical, you might not know about robots.txt files. These are plain text files you put on your server, for example here's the one for this site, and instruct crawlers as to what to crawl and what not to crawl. Good Bots, such as Googlebot, check what they find against this file and won't try to fetch links you tell them not to. Others will be at pains to tell you that robots.txt files are merely voluntary; these people make Bad Bots.

On this site, my robots.txt file is not voluntary. If you don't follow it, you're going to have a very bad time. Much like a railway station that tells you to stay behind the yellow line - it's a voluntary recommendation, sure, but you're only putting yourself at risk if you don't follow it.

Iocaine

A brief mention for iocaine, which is the inspiration for much of how I implemented the following system. If you're not using Elixir or something similar, this might be a good route to go down for introducing similar defenses to your own projects and sites.

Also I appreciate the reference to The Princess Bride.

Hell is other robots

There's a trap in my robots.txt file and on this site - a hidden link that users don't see but crawlers do. The moment they hit that URL, they will no longer see my real content; instead they get trapped in an infinite blackhole of rubbish. They're served a page of nonsense photographic text with a bunch of links and images, the links being to more randomly generated pages with more junk text, links, and images, and the images are a collection of stupid AI-generated images of sexy toasters.

3 slots? Phwoar.

I've got a Google Gemini subscription for the month because I mistakenly thought it would give me a bargain for code generation, so hey - gotta use it for something. It does chastise me for wanting to generate such erotic images sometimes, but it gets there in the end.

Anyway, it gets worse for the bots. The metadata for every image served is randomly generated too, and then I have some nasty tricks courtesy of this website being written in the programming language Elixir.

Elixir is fun

Disclaimer ahead of time: I used LLMs to build a lot of this functionality. This is of course highly ironic when I'm complaining about data scraping for LLM training. I'm now pursuing photography full-time and I'm no longer a software engineer, so gone are the days when I can dedicate a few weeks to building out something fun like this entirely by hand. If I do open source this code at some point I'll rewrite it myself first.

I was a Ruby engineer for years and I still love the language, but Elixir is fun on a totally different level. I love Ruby because it's beautiful (opinions may vary), I love Elixir because it lets me completely rethink the architecture of my software. The Erlang BEAM that underpins it is an incredible bit of engineering and so much fun to build things on. Also LLMs are weirdly good at Elixir, possibly because of how excellent its documentation is, possibly because of fewer people using it and thus having higher quality reference code.

If you're not familiar with the BEAM, the extremely short story (go read this book for the long story) is that it's built for concurrency - it's very, very good at scaling horizontally across multiple CPU cores and servers, and excellent at balancing large volumes of work at the same time.

I built this website in it because Phoenix, the web framework, is fun and has some great features that I thought I might need someday despite this being a mostly static site. I was thinking live dashboards or building my own gallery system for clients. Turns out it's excellent for wasting the time of Bad Bots.

Let's waste some time

To start with, if you visit the trap pages, you'll no longer be able to see any legitimate content on this site. It will all be replaced with junk, and the block lasts for quite a while. This is by IP for individual visitors, and I have configurations for the biggest offenders (i.e. companies) that forcibly puts entire subnets into the trap by default. Conversely, bots I actually want - and that obey the robots.txt file, such as Googlebot, Duckduckbot, Bing, OpenAI etc - are exempt from the trap, and they can't fall into it. They are served 403 forbidden status codes if they accidentally stumble across the links somewhere. I also had to exclude Archive.org's crawler as that ignores robots.txt for different reasons, which I don't necessarily agree with but eh, I quite like the goal of the archive.

Most of the crawlers flouting my robots.txt file only get stuck in the blackhole very briefly, as they've likely been configured to travel only so many levels deep on each website. For these bots, they get pages filled with increasingly large amounts of junk JSON-LD data to waste token usage if fed into an LLM. Alongside the sexy toasters of course. There are also some fun fake database dumps for the more illicit crawlers, just as an added spice.

But where it gets really fun, and where the BEAM comes in, is the tarpit. All the Bad Bots are randomly served the images they try to scrape at extremely slow download speeds. I was there in the 90s and I think the bots deserve to experience 56k speeds too, if they're to truly become human. The BEAM is amazing at this, as each request consumes zero CPU whilst it's sat sleeping, waiting to return the next few bytes of data. I send the image headers and then gradually trickle the rest of the image to them over time. Some of the bots are apparently not currently configured to handle this, as I've seen a few requests take up to 15 minutes. Hohoho.

On the subject of Meta

Now, most of these scraping attempts have finished quite quickly. The difference in traffic between Meta's Meta-ExternalAgent scrapers and the next worst offender is absolutely ridiculous. If you're one of the muppets who wrote Meta's scraper and you're reading this, you really need to add a max crawl depth. And obey robots.txt, you cretins.

I'm currently serving about 2 requests per second to Meta's scraper network. For comparison, I usually get about 10 legitimate human visitors to this site per day, which is obviously a travesty but hey, that's the modern internet.

The level of traffic started causing problems a few days ago. I wasn't actually expecting them to get stuck like this, and the token inflation code was starting to peg my server's CPU at 100% generating the junk data. I needed a new solution to reduce CPU usage, but also to reduce the amount of concurrent traffic coming from Meta - it was just increasing every day as they found more junk links, and I wasn't sure there'd be an upper limit.

The solution was to drop the token inflation for the worst offenders, as they seemingly have infinite money anyway, and instead implement the tarpit system for every web request. This now happens randomly for every offender, but Meta is being subjected to it for every single request. Page loads now take about 18s for them, and they're filled with junk when they do load. My CPU usage is now running at about 20-40% depending on the burst rate, which is acceptable. I'm well within my egress traffic quota, thankfully, as the sexy toaster images aren't very large. I have noticed a general slowdown in the speed of image serving on the site, but it's a price I'm willing to pay.

Could I just use Cloudflare or block their IPs?

Cloudflare provides protection against many scrapers, but they themselves are also a large publically-traded company. I recommend them to less technical people who want to reduce bot traffic to their sites, but I went off using them a few years ago and I oppose the centralisation of the internet in the hands of a small number of people.

I could just block the scraper IPs myself, and may eventually have to if the traffic level just keeps increasing. But merely blocking them isn't enough, as they've already no doubt harvested much of my content. Blocking them just means they'll redirect this traffic to someone else who dared to put a part of themselves on the internet without paying a central authority to protect them.

I realise that by publishing how I have implemented these bot defenses, it gives information to these companies to figure out how to circumvent them. And that's fine, I have a surprising amount of spare time since I stopped working as a software engineer, and no OKRs.

Is this worth my time?

The consumption of all human endeavour to fuel the stock prices of these "AI" companies is inevitable. I'm under no illusions as to my efforts here making any dent in their efforts to do so - I am ultimately insignificant.

But I charge clients for my photography services. It seems only fair that the companies should pay too. What I create is an extension of me, of my experiences and time, of my life on this planet. As these enormous companies burn the world to fuel their share prices, and by chance possibly invent true Artificial Intelligence, I think it's fair that they put in the hard graft that I did in order to learn.

After all, to err is human.