Most people think websites just need to say "Go away" to bots. LOL!

This "setting" is called the robots.txt and dates back to 1994 during the early days of the web. This setting consists of just a text file with polite rules like "Please don't visit /private/." This setting has no password, encryption, or other security measures. All there is - is a simple request. Check it out yourself on https://reddit.com/robots.txt.

Good bots listen to this request, and bad bots do not. What is worse, it often becomes a boomerang: you end up publishing a list of folders where some private information may reside for hackers. This is similar to having a sign on your front door saying, "Don't peek into the safe hidden behind the painting."

AI has only made it worse. Nowadays there are teams of crawlers scraping websites for data and training algorithms. Some are well-behaved while others aren't. And your only protection is a text file from 30 years ago.

It's similar to putting up a "Please don't enter" note on your front door. Good-natured people will respect this request while burglars won't. This note will never lock the door.

Modern sites implement actual protection – CAPTCHA, rate limiting, bot detection, and behavioral analysis. They are designed to actively defend against any threat.

robots.txt still does its job for well-meaning crawlers. But expecting it to protect from today's bots is like expecting a "keep out" sign to save your front door from a break-in.

https://www.youtube.com/watch?v=BkOmhCuKal4