White starts: are “good” bots really that good?
It would seem that the question of good and bad bots is quite simple for companies: block some, let others pass. However, this process has its own nuances. This is due to the fact that bad bots disguise themselves as good ones, and the fact that incorrectly configured bots from the white list can spoil traffic and put the site down, and the fact that companies do not even always need to block a bot, but you just need to understand where he walked. Let's see how bad bots differ from good ones.
Some statistics
GlobalDots 2019 Bad Bot Report, 39% - , . 20,4% , 17.5% - . , 2019 « » , « » . , , “”, , .
, , . , , - 22.9%, - - 17.9%.
: , , DDoS-, , , , - , - , - , , , - , .
. . , (, - -), , -.
(crawlers)
- , , , , , .. 30% -. : Googlebot, YandexBot, ematchers – , , YandexImages.
-
, , , , . : Amazon Echo, Google Home, Siri, . .
-, : , , . , . - Content ID YouTube, .
, Google Shopping. - , .
, (, , - ).
, whitelist Variti , , , preview - , - . , , ., AdSense-, , .
, , , , . .
, . . , - - . , , . .
. , , - , . .
, , , , - .. , , IP-, .
, . , Android , , . , , . , - , , , . , .
, , IP , . IP , CIDR-, .
“” IP , . , , , / IP . . , , token . . IP . , - , user agent -. .
. , , website.com/api. API, - , API IP . , API .
whitelists, IP , , – Variti, , Telegram- Variti .
, , . , , . - SMS-. , SMS- - . , . , . , SMS, .
, “”, . , 10 , 1000. robots.txt, , . , , . , - , .
- . - , IP-, , .. DDoS-. , .
, , . . . – , , . , .
GlobalDots 2019 Bad Bot Report 523 . « » (55,4%) Google Chrome. Firefox, - Android. Safari, Internet Explorer, Safari Mobile, Opera, Googlebot Bingbot .
, , , 20 , - , , Internet Explorer 5, 1999-. 0.8% Internet Explorer 7. , . , - , .
, 74% « » - advanced persistent bots (APB), , . , , IP-, -, Java- peer-to-peer , .
, , . , , .
Variti user agent, , IP , , , . , , ( Chrome, headless) , .
, , , , . .
, “ ”
“” , . , , . , , — . , , - - , .
, , , email. - , . , , .
Firefox, Chrome, Internet Explorer Safari , - , . .
robots.txt — , . , , . : , . , - - , robots.txt. - .
, , . , - - , (, - ), stealth-, , . - ?