Hi, I’m building a personal website and I don’t want it to be used to train AI. In my robots.txt file I blocked:

  • ChatGPT-User
  • GPTBot
  • Google-Extended
  • FacebookBot

What bots should I also add? Are there any other ways to block AI bots?

IMPORTANT: I don’t want to block search engine crawlers, only bots that are used to train AI.

  • Luke@lemmy.ml
    link
    fedilink
    arrow-up
    59
    arrow-down
    1
    ·
    1 year ago

    FYI, bots and crawlers can simply ignore your robots.txt entirely. This is probably common knowledge around these parts, but I’ve run into clients at work who thought it was a law or something.

    I do like the idea of intentionally polluting the data robots will see, as suggested by this comment. There’s no reliable way to block them without also blocking humans, so making the crawled data as useless as possible is a good option.

    Just be careful not to also confuse screen readers with that tactic, so that accessibility is maintained for humans. It should be easy enough if you keep your aria attributes filled out appropriately, I imagine.