- Websites disappear from Google search and disappear into oblivion
- YouTube videos are demonetized and creators are losing revenue
- Android apps disappear from Google Play and can't reach their users
- APIs are becoming more expensive or just out of date
- Last but not least, a personal counterpart to all of the above: People lose access to their Gmail accounts and their entire digital life.
I swear I read the FAQ!
Everything goes according to one scenario. At first, businesses deliberately use Google services in a way that depends entirely on them for their survival. Then Google's automated behemoth does its job: It slightly changes the position of its ass on a leather chair the size of a planet and, without noticing it, destroys myriad of (relatively) small companies the size of an ant in the process. And finally, ant-sized companies are desperate to tell Google they are crushed, but can only communicate with an automated suggestion form.
Sometimes an ant-sized CEO knows someone on Google because they were college buddies, or a CTO writes a post that somehow ends up on the front page of Hacker News. Then Google notices the problem and sometimes considers it worthy of a solution, usually out of fear of the regulatory implications that the ant revolution could have.
For this reason, conventional wisdom from ants dictates that you should not over-rely on Google services. And if you can manage to avoid this addiction, everything should be fine.
Such a flat blue surface with a cool red roof! So convenient!
What's new under the moon
In today's series "The Internet Isn't What It Was Before," we'll talk about a new way Google can inadvertently overwhelm your startup. It does not require you to use Google services in any (intentional) way .
Did you know that your domains can be blacklisted by Google for no particular reason, and that this blacklist is used not only by Google Chrome, but also by several other software and hardware vendors? Did you know that these other vendors sync the list with wildly changing timings and interpretations, so that solving any problems becomes extremely stressful and unpredictable? At the same time, the terms for processing complaints about the blacklist in Google are measured in weeks?
Now this is how your site or SaaS application looks like
This "feature" of the blacklist is called Google Safe Browsing , and in the image above you can read the message that your site visitors see if one of the domains is included in the Safe Browsing database. Warning texts range from “Spoof site” to “Site contains malware” (complete list here ), but they are all set against an equally scary red background that tries to prevent the user from going to the site as much as possible.
The first time we heard about the issue was due to a spike in complaints from customers who came across this red warning while trying to use our SaaS application. The second time around we are better prepared, so we already have free time to write this post.
Our company InvGate - is SaaS-platform for IT-departments, working on AWS. It serves over 1000 corporate clients and millions of end users. Firms use the product to manage tickets and requests from their users. You can imagine the reaction of IT managers when all of a sudden their ticket system starts showing end users such ominous security warnings.
When we first encountered this problem, we were desperate to understand what was going on and understand how Google Safe Browsing (GSB) works, while tech support was trying to cope with the flow of requests from customers. We quickly realized that the URL to the Amazon Cloudfront CDN , from which we were serving static resources (CSS, Javascript, etc.), got into the GSB database , and this caused the entire application to crash for customers using this particular CDN. A quick look at the labeled system showed that everything looks fine.
While the Devops team was working in a rush to set up a new CDN and prepare to move clients to a new domain, I found in Google documentation that via Google Search Console(GSC) you can get additional explanations about the reasons why the site is flagged in the database. I won't bore you with the details, but in order to access this information, you must prove that you own the site , to do this, set up a custom DNS record or upload some files to the domain root. We did it - and in 20 minutes we received a report about our site.
The report looked something like this:
This is ... not particularly informative
The report also had a Request Review button that I quickly clicked without actually taking any action on the site as there was no information about the alleged issue. I submitted an application with the note that I did not have malicious URLs listed, although the documentation states that Google always provides sample URLs to help webmasters identify problems.
Fine! A request to check an invalid report may cause future checks to become even slower.
About an hour later, the site was removed from the GSB database, we didn't even finish withdrawing clients from this CDN. About two hours later, an automatic email arrived confirming that the verification was successful. No clarification as to what caused the problem.
What happened next
During the week following the incident, we continued to receive periodic reports of access issues from customers.
Google Safe Browsing provides two different APIs for use in commercial and non-commercial products. In our case, the problem was reproducible in at least some Firefox users, as well as some antivirus and network security devices. They tagged our site and blocked access to it many days later .
We continued to migrate all clients from the old CDN to the new one, and so we ended up fixing the problem for good. We never really found out the reason and blamed it on some stoned AI at Google headquarters.
How to prevent Google Safe Browsing from tagging your site
If you run a SaaS business and promise customers guaranteed availability, then listing in Google Safe Browsing for no specific reason is a very real business risk.
Unfortunately, given the purely Google opaque mechanism for tagging and viewing sites, this is unlikely to be guaranteed to be avoided. But you can certainly design your application and processes to minimize the likelihood of this happening, reduce the impact of actual blacklisting, and minimize the time it takes to resolve the issue.
Here are the steps we are taking ourselves that I can recommend:
- . , GSB . , . , company.om , app.company.net , eucdn.company.nt , useastcdn.company.nt . .
- . - , SaaS . , , . , , . , companyusercontent.cm .
- Google Search Console. , , . , , .
- Be prepared to change your domain if needed . This is the hardest thing to do, but it is the only effective anti-blacklisting tool: Design systems so that service domain names can be easily changed (via scripts or orchestration tools). For example, suppose eucdn.company2.net has a CNAME record for eucdn.company.net, and if the former is blocked, update the application configuration to load resources from an alternate domain.
What to do if your SaaS application or website is blacklisted by Google Safe Browsing
Here's what I would recommend:
- If you can easily and quickly switch your application to another domain, this is the only way to reliably, quickly and supposedly finally resolve the incident . If you can, do so. That's all.
- , , Google Search Console. , , .
- , (, ), . Safe Browsing , , .
- , , , . .
A few months after the first incident, we received an email from Search Console informing us that one of our domains was again blacklisted. A few hours later, as a G Suite domain administrator, I received another interesting email, which you can read below.
The “sc” in sc-noreply@google.com stands for Search Console.
Let me explain in my own words what it is, because it's just mind-blowing. This is an email warning from Search Console about being blacklisted. This second email says that the G Suite automatic phishing email filter considers this email from Google Search Console to be bogus . Of course it is not.because our domain was indeed blacklisted. So Google ca n't even decide if its own phishing alerts are phishing . (Lol?)
Some unpleasant thoughts about the future of the internet
To anyone in the tech industry, it's pretty clear that the big tech giants are pretty much the guardians of the Internet. But before, I interpreted it in a free, metaphorical sense. The Safe Browsing incident described here made it very clear that Google literally controls who can access your site, no matter where or how you operate it. Since Chrome has about 70% of the market, and Firefox and Safari to some extent use the GSB database, Google can make any site virtually inaccessible on the Internet with one flick of a finger.