Hi, I’m building a personal website and I don’t want it to be used to train AI. In my robots.txt file I blocked:
- ChatGPT-User
- GPTBot
- Google-Extended
- FacebookBot
What bots should I also add? Are there any other ways to block AI bots?
IMPORTANT: I don’t want to block search engine crawlers, only bots that are used to train AI.
OP still wants search indexing, in which case it’s a big no-no - it can be perceived as spam by search engines, and links your pages to tons of unrelated keywords.
I can block search engine crawlers from specific paths so that should be solved.