Hubi@feddit.de to Reddit@lemmy.worldEnglish · 8 months agoReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comexternal-linkmessage-square173fedilinkarrow-up1858arrow-down115
arrow-up1843arrow-down1external-linkReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comHubi@feddit.de to Reddit@lemmy.worldEnglish · 8 months agomessage-square173fedilink
minus-squareVeraxus@lemmy.worldlinkfedilinkarrow-up23·8 months agoIf it is publicly accessible, then yes, they scrape the data. This is literally how search engines have worked since damn near the inception of the internet.
minus-squareLaterRedditor@lemmy.worldlinkfedilinkarrow-up3arrow-down2·8 months agoWasn’t there robots.txt at least?
minus-squareSacrificedBeans@lemmy.worldlinkfedilinkarrow-up12·8 months agoRobots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
If it is publicly accessible, then yes, they scrape the data.
This is literally how search engines have worked since damn near the inception of the internet.
Wasn’t there robots.txt at least?
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
There was, and still is