51.143.124.155

Unusual bing activity has been observed since yesterday. And it would seem that there is nothing wrong with crawling sites by search engines (and even bing ), but only as long as they comply with their own identification rules .



In the analysis further, only 51.143.124.155 will be mentioned , although

52.250.62.82,

52.250.19.204,

52.250.119.153 also flashed in the logs ,

but they behave more culturally.



What the problem looks like on the example of a selection of bing bots from yesterday's log of one site:

$ cat ./20201026* | grep -F bingbot | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
 404582 51.143.124.155
  10449 157.55.39.174
  10037 207.46.13.162
  10036 207.46.13.86
  10008 157.55.39.152
   6638 207.46.13.207
   4993 207.46.13.238
   4113 157.55.39.200
   3897 157.55.39.205
   2995 157.55.39.166

What a typical "old" bing bot looks like:

$ whois 157.55.39.174 | grep -i "org[-]*name"
OrgName:        Microsoft Corporation

$ host 157.55.39.174
174.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-174.search.msn.com.

They are based on the networks

157.56.0.0/14,

157.54.0.0/15,

157.60.0.0/16,

207.46.0.0/16

and are successfully verified at www.bing.com/toolbox/verify-bingbot-verdict :







What the “new »Bots:

$ whois 51.143.124.155 | grep -i "org[-]*name"
org-name:       Microsoft Limited

$ host 51.143.124.155 
Host 155.124.143.51.in-addr.arpa not found: 3(NXDOMAIN)
$ host 52.250.62.82
Host 82.62.250.52.in-addr.arpa not found: 3(NXDOMAIN)
$ host 52.250.19.204
Host 204.19.250.52.in-addr.arpa not found: 3(NXDOMAIN)
$ host 52.250.119.153
Host 153.119.250.52.in-addr.arpa not found: 3(NXDOMAIN)

And accordingly:







At the same time, handwriting, activity on all sites at once, as well as messages from colleagues that after blocking these "new" ip notifications from Bing Webmaster Tools about the site unavailability are received , allow us to conclude that these bots really are the work of microsoft .



Support requests and abuses are, of course, ignored.



In general, do not immediately hit bots from networks 51.140.0.0/14, 52.224.0.0/11. They are probably useful (or at least harmless) bifidobacteria.



Upd (10/28/2020 08:48 MSK):

Yesterday (at 27 / Oct / 2020: 11: 56: 24 +0300) requests stopped.

There were no answers to the abuse.

@Facan tweeted a comment on a similar request:

… Some sites reported that a small percentage of Bing crawler was not DNS resolving appropriately and we were crawling outside of usual IPs for few days. We fixed that yesterday.