From Newsgroup: comp.security.misc
One can also use
robots.txt
with any webserver. As Mister Moock remarks, this shall not do anything
about IP addresses.
Cf.
HTTP://WWW.robotsTxt.org
I am not overwhelmed by crawlers downloading from a website by me, but I
did add to robots.txt . . .
User-agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +
http://ahrefs.com/robot/) User-agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +
http://www.semrush.com/bot.html)
User-agent: Mozilla/5.0 (compatible; Bytespider;
spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +
https://developer.amazon.com/support/amazonbot)
User-agent: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider;
spider-feedback@bytedance.com)
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +
https://openai.com/gptbot)
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +
https://openai.com/gptbot)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.119 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: facebookexternalhit/1.1 (+
http://www.facebook.com/externalhit_uatext.php)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.119 Mobile Safari/537.36 (compatible; GoogleOther)
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +
https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0;
+claudebot@anthropic.com)
User-agent: meta-externalagent/1.1 (+
https://developers.facebook.com/docs/sharing/webmasters/crawler)
User-agent: netEstate NE Crawler (+
http://www.website-datenbank.de/) User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.108 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: COIBotLinkSaver/2.0
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.69 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.6668.89 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.139 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.69 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.116 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +
http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.137 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (compatible; SemrushBot-BA; +
http://www.semrush.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.85 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (compatible; archive.org_bot +
http://archive.org/details/archive.org_bot)
User-agent: Mozilla/5.0 (compatible; YandexBot/3.0; +
http://yandex.com/bots) User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.58 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.113 Mobile Safari/537.36 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html)
Disallow: /drochdhliodoiri/mu/Rioghachd_Aonaichte_na_Breatainne_Moire_agus_na_h-Eireann_a_Tuath/mun_Chuimrigh_agus_mu_Shasainn/PC_EH_Hills/South-Yorkshire_police_perpetrates_subornations_de_perjuries_for_malicious_prosecutions.webm
Disallow: /stailc_teanga/La_Eile_gan_ghraiscinteacht.webm
On Mon, 24 Feb 2025, The Doctor wrote:
"Also what does 2FA and FA really stand for?
2FA Dual-failed authentication
MFA Multiple falied authentication."
:)
--- Synchronet 3.20c-Linux NewsLink 1.2