• Archr@lemmy.world
    link
    fedilink
    English
    arrow-up
    16
    arrow-down
    1
    ·
    edit-2
    2 days ago

    I feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article.

    Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.

    • General_Effort@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      2 days ago

      that is not how general news media has been talking about robots.txt.

      Ahh, yes. I think there is a lesson there.