The topic of bots on the site is very painful for some and gives a constant headache, while others prefer to ignore it entirely. And here you need to understand: if the damage from bots is not great, then investing money to fight them becomes not very justified from a business point of view. Most often, in such cases, they are limited to some fairly simple self-written solution that fights off the first generation bots (about the generations below). However, if attacks on the site harm the business, the company begins to think about how to deal with them.
From experience, I can say that air and large e-commerce companies most often suffer from the constant flow of bots. A typical case for an airline, when it can be hit by bots and get paid, is the well-known seat-holding. How it works: you want to buy a ticket and choose a flight, time, place and get paid. Very often, in this place, the airline reserves this place for you (not available for purchase at the moment to other site visitors) and gives you time to pay by card - from a couple of minutes to several hours. What happens: a wave of bots arrives at the site, which reserves the entire plane (and after the payment time expires, reserves a seat again). Thus, no one can buy a ticket, and the plane, in fact, can fly away empty. And this is money, and quite a lot.For e-commerce companies, this is most often the search for credentials and write-off of bonus points.
What kind of bots are there? Here, in fact, everything is very simple. There are two answers - good and bad. Good ones are search engine bots, bots of your partners, some kind of monitoring systems, and so on. In general, these are the bots that follow the rules:
impersonate who they are
do not try to harm the site
follow links robots.txt
The bad ones are those who have unfriendly intentions or are used for other purposes. Usually they try to either disguise themselves as people, or to fake other well-known good bots. Most often bad bots are used to:
Automatically log in using stolen credentials from a data breach or dark web.
Create new online accounts to receive registration bonuses
, , .
DDoS-,
, . – ?
. , :
cURL- -. IP-. cookie JavaScript, -.
-, «» (headless) (: PhantomJS SimpleBrowser), Chrome Firefox, . , cookie JavaScript. - headless JavaScript - .
– . , . .
, , , , , , . UA IP-. , « » - , , - . - , . , , .
, , .
? :
, , ,
,
,
, , , , .
? :
( CDN, reverse proxy)
, /.
, Akamai, Distill ( Imperva ABP) RadWare. – PerimeterX.
, Akamai.
, Akamai, , . Akamai – -, (wiki). 1998., CDN . 20 . Akamai CDN, , -. CDN, , , WAF, DDoS mitigation ( L3/L4, L7), BOT, DNS, Real User Monitoring (RUM), API Gateway . , , . Akamai ? , 40 ( – 5). .
, , Akamai Bot Manager. :
Bot Manager Standard (BMS)
Bot Manager Premier (BMP)
, .
.
, General bot management, BMS, Transactional endpoint protection – BMP. , .
Customer-Categorized Bots? , , . , , , . ? . (, “Partners”), .
, .
Akamai-Categorized Bots. , , , . Akamai 1400 17 . ? , : , , , , . , «».
, – . : (transparent) (active). , , , . - . :
, , . , “Impersonators of Known Bots” – , , , Google, . :
Chrome –
-
Active – JavaScript cookie ( – SDK):
Akamai cookie . . , , JS .
BMP. , : /, , . endpoint. , POST /login, BODY user pass. , endpoint-a Akamai . BMP – . , BMS BMP , BMS, , - , BMP , . , , , . , - , .
: ? , : – , . , , . .
: ? :
( )
( 1–3 )
( 8–10 )
( 403 , - )
Tarpit ( , )
Challenge ( Google Captcha, Akamai Crypto Challenge)
Conditional action ( , 20% , – )
, Deny ( 403 ) -. 403 – , . , . . Akamai Tarpit. , Linux, , action IPTables. Tarpit, , . , , . : Akamai? -. 270 , . , . , . , . , , .
– Captcha. , . ! Google Captcha – Akamai. Crypto Challenge. : Akamai , ( 30 , ). , : crypto challenge ( ). , Akamai , , – challenge ( ).
Akamai ? , - (, , ). : . , , . , 60–70% . – .
, . , , . : , . : Akamai , . , User-Agent. Akamai – , . , 30% , , .
How did you get the idea to write this article? For two days we attended Highload ++ this year with our stand GlobalDots, and quite a few people approached us with approximately the same questions: Is there Akamai in Russia? Anything besides CDN? Actually, this is how this small article appeared.
This is probably all. It turned out to be such a rather introductory article. I'm used to showing more than telling - it turns out more informative. If you have any questions - welcome to saint tropez in the comments.