Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 30 days agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square64fedilinkarrow-up1111
arrow-up1111external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 30 days agomessage-square64fedilink
minus-squareAsudox@lemmy.worldlinkfedilinkarrow-up6·30 days agoNot sure if that is effective at all. Why would a crawler check the robots.txt if it’s programmed to ignore it anyways?
minus-squareɐɥO@lemmy.ohaa.xyzlinkfedilinkarrow-up16·30 days agocause many crawlers seem to explicitly crawl “forbidden” sites
minus-squareCrashumbc@lemmy.worldlinkfedilinkEnglisharrow-up3·29 days agoGoogle and script kiddies copying code…
minus-squareMangoPenguin@lemmy.blahaj.zonelinkfedilinkEnglisharrow-up1·22 days agoYou could also place the same page as a hidden link on your home page.
Not sure if that is effective at all. Why would a crawler check the robots.txt if it’s programmed to ignore it anyways?
cause many crawlers seem to explicitly crawl “forbidden” sites
Google and script kiddies copying code…
You could also place the same page as a hidden link on your home page.