What pages do bad bots look for?, (Sat, Aug 1st)

3 years ago 142

BOOK THIS SPACE FOR AD

ARTICLE AD

1. August 2020

Read the original article: What pages do bad bots look for x3f;, (Sat, Aug 1st)

I&#x27ve been wondering for some time now about what pages and paths are visited the most by xe2;x80;x9c;badxe2;x80;x9d; bots xe2;x80;x93; scrapers, data harvesters and other automated scanners which disregards the exclusions set in robots.txtx5b;1x5d;. To determine this, I&#x27ve set up a little experiment xe2;x80;x93; I placed robots.txt on one of my domains, which disallowed access to commonly used paths and PHP pages which might of interest to bots (login.php, /wp-admin/, etc.), configured the server to provide HTTP 200 response for these paths and pages and started logging details about requests sent to them.xd;