rancidrabbit

@barbapulpe@gayfr.social @aphyr@woof.group If bots ignore the robots.txt, they deserve what they get, as far as I care. No sympathy for abusive corporations. Load has been fine with the current setup. User input has been not wanting to feed the models. What blocklist do you use? If I did that, I'd make sure to allow valid users. Bots use VPNs but so do valid users. I'm not checking access logs (logging to /tmp in mfs and deleting often) as often I could but in the past I've seen /

@aphyr@woof.group Any HTTP/1.1 requests are a red flag. One server was blocking any IP that did that. I haven't gotten to blocking IPs but fail2ban and a honeypot is a good idea. Will link to configs if you'd like then.

@barbapulpe@gayfr.social @aphyr@woof.group Go team Anubis! They have a paid IP blocklist too. I should check how much that costs. I replied to École des Bro-Arts with a list of things we're doing.

@0x2ba22e11@unstable.systems @aphyr@woof.group I think that's the idea. https://github.com/kurren/ai-bots-crawlers suggested that, but I didn't add that to my nginx.conf, not wanting to risk breaking things for people or other servers. That's also something that the bots could easily bypass. I'm really not sure why they are using HTTP/1.1. Looking at HTTP/1.1 activity in the logs reveals more networks to investigate and block, tho.

@Larvitz In news.html, "Perfect for all night owls" is unclear whether it's described people who stay up all night long, or all of the people who stay up late in to the night. If you even wanted to bother with that, you could remove the word "all". Aside from that one very small nit that's not even on the landing page, everything looks perfect. The landing page is perfect.

@barbapulpe@gayfr.social @aphyr@woof.group scrapers from LLM companies pretending to be other mastodon servers and giving "(http.rb/5.3.1; +https://mastodon.social/)" as their user agent. /

@barbapulpe@gayfr.social @aphyr@woof.group https://anarchism.space/nginx.conf.txt https://anarchism.space/sites-enable-anarchism.space.conf.txthttps://github.com/kurren/ai-bots-crawlers was my starting point for that. I added Anubis later.

@aphyr@woof.group Using Anubis here without any reports of problems so far except I promised to donate to Anubis and replace the splash screen image. All of the bots ignore robots.txt but have a long list of robots.txt rules in place anyway. Blocked all of the ASNs owned by a bunch of LLM companies. That helps more than anything else but some other places report OpenAI and others using home network proxy services to attack them. nginx is doing user-agent blocking plus ASN blocking. 1/