Recently, I wrote an article about my journey in learning about robots.txt and its implications on the data rights in regards to what I write in my blog. I was confident that I wanted to ban all the crawlers from my website. Turned out there was an unintended consequence that I did not account for. My LinkedIn posts became broken Ever since I changed my robots.txt file, I started seeing that my LinkedIn posts no longer had the preview of the article available. I was not sure what the issue was initially, since before then it used to work just fine. In addition to that, I have noticed that LinkedIn’s algorithm has started serving my posts to fewer and fewer connections. I was a bit confused by the issue, thinking that it might have been a temporary problem. But over the next two weeks the missing post previews did not appear.
    • TeddE@lemmy.worldEnglish
      4·
      1 day ago

      Kinda, but also not really. Any major tech player that has billions to lose will make a show of respecting robots.txt when presenting that information to third parties, lest they be exposed by basic journalism.

      However, they also have separate networks in R&D that sweep the net all the time and do not care about such restrictions. It’s theatre.

      And they’re still happy to punish people that have the gall to publicly decline their crawlers. Basically they can eat their cake and have it too.

    • ell1e@leminal.spaceEnglish
      311·
      2 days ago

      Often it is respected, but the resulting problem is platforms conflate things with the questionable AI scraping crawlers to blackmail websites into participating in feeding AI.

      For example, Googlebot if enabled won’t just list you for search, but will also scrape your contents for Google’s AI. Edit: see https://arstechnica.com/tech-policy/2025/07/cloudflare-wants-google-to-change-its-ai-search-crawling-google-likely-wont/ as source. I imagine LinkedinBot, given it’s microsoft, will feed some other AI of theirs as well on top of the previews.

      Until regulation steps in to require AI bots to separately ask for crawling permission, or to actually get a proper license for reuse of the contents, this situation isn’t going to improve.

      • General_Effort@lemmy.worldEnglish
        27·
        2 days ago

        Googlebot if enabled won’t just list you for search, but will also scrape your contents for Google’s AI.

        False.

          • General_Effort@lemmy.worldEnglish
            2·
            1 day ago

            Ok. That quotes a tweet by Cloudflare’s CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.

            ETA: I looked at what the Cloudflare CEO said again. To be fair to him, he is not actually claiming that Googlebot collects AI training data. He’s talking about the AI overview, which is a search feature. The data for search features is collected by Googlebot. I’m not sure why someone would want their link listed in search but not appear much more prominently in the AI overview.

            Here’s Google technical documentation on its crawlers: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

            • ell1e@leminal.spaceEnglish
              3·
              2 days ago

              So what’s the quote from the documentation that backs up your claim? The line “perform other product specific crawls” seems extremely vague by design.

              • General_Effort@lemmy.worldEnglish
                2·
                2 days ago

                I’m not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?

                • ell1e@leminal.spaceEnglish
                  2·
                  2 days ago

                  Nothing on this page seems to contradict the article. But if I simply missed the part that does, I’d be happy to learn.

                  • General_Effort@lemmy.worldEnglish
                    2·
                    2 days ago

                    You look up what Googlebot does. No AI.

                    You want to know what crawlers do AI? Just search for “AI”, or “training”, or some such, or skim through. It’s not long. Google-Extended collects training data. Note that Google-Extended is explicitly not used to rank pages.

                    Did that help?

        • cecilkorik@lemmy.caEnglish
          21·
          2 days ago

          Absolutely true. They’ll buy the data they want from some shitty crawler running from some data broker in some far-flung and lawless part of the world, hallucinate the actual source, and pretend they had no idea their “data partner” wasn’t respecting robots.txt if they have to, which they won’t ever have to do because it’s literally impossible to detect and prove and realistically unenforceable.

          This is a company that removed it’s company motto of “Don’t be evil” because it found it too “limiting”. Don’t be naive.

          • General_Effort@lemmy.worldEnglish
            2·
            2 days ago

            That’s very different from what I called false.

            What you describe may happen, but probably not as much as you think. Much of that stuff is just not that valuable. Some personal, colloquial writing is necessary, but Google already pays Reddit. Other stuff is better obtained from torrents or shadow libraries like Anna’s Archive.