Timeweb Experience
Any company needs an outside perspective on the state of information security of services and products. This problem can be solved in different ways, one of which is participation in Bug Bounty programs.
Bug Hunting Bug Bounty is a program that provides cash rewards or other benefits for finding bugs, exploits and vulnerabilities in software. Bug Bounty programs are implemented by many companies, including Facebook, Google, Reddit, Apple, Microsoft and others. A
company can launch such a program on its own, organizing all processes and interactions on its own. The second option is to turn to special Bug Bounty platforms: we conclude an agreement, and the army of bug hunters starts working.
Timeweb launched its Bug Bounty program about a year ago. At that time, the company did not have specialists with experience in this area, everything had to be done by trial and error. On the Internet, almost no one shares recommendations on how to build this process, so bicycles are often invented, and knowledge is usually transferred in conversations with colleagues at the coffee machine.
In this article we will tell you how to organize the launch of the Bug Bounty program, if you have never done this, what you should pay attention to and how else you can check the status of the information security system.
Check it out!
Why did we launch the Bug Bounty program?
Our task was to raise the information security system to a new, higher quality level and not spend too much money. It is important to note that Timeweb is a mature company with a complex infrastructure and a large set of critical services, so it was not at all obvious to us then in what order to test services and fix vulnerabilities, and what to start first of all.
There are several ways to check the information security system. Before launching the Bug Bounty program, we tried to consider and analyze various options. Among them, external audit is probably the most common solution to the problem. Thanks to the audit, you can find complex, non-standard problems, but even after paying a large amount, you cannot be sure that you will get what you really need.
Another way is to train the team and develop the competencies of the employees. Here you need to understand that the resources of the team are still limited, and of course, it is impossible to find all possible bugs on your own.
In the matter of checking the information security system, tickets and customer requests to technical support also help if it is possible to efficiently process user comments and messages.
In parallel, we talked with colleagues and collected expert opinions on how the process of finding bugs in other companies is structured. Thanks to the recommendations, we decided to pay attention to the Bug Bounty program.
Launching the Bug Bounty Program
What is important to consider at the start?
There are many different types of Bug Bounty programs and platforms.
At the first stage, we chose between public and private programs, deciding to focus on the latter. Access is not limited in the public program: everyone can search for bugs. In a private program - by invitation. In both cases, bugs are disclosed only by agreement of the parties. We decided that it is too early for us to open a public program: first of all, we must be sure that our services and products do not contain critical vulnerabilities.
As for the Bug Bounty platform itself, we analyzed the existing options and chose the most suitable for us, optimal in terms of the number of bug hunters and the cost of services.
Here is a list of the most famous Bug Bounty platforms on the market:
We should also mention the Open Bug Bounty platform - a non-commercial Bug Bounty platform that unites information security enthusiasts and popularizes ethical hacking. Researchers can report a bug found in the work of any software, for which the company provides a reward (cash payments, merchandise, discounts or their own products). Timeweb, for example, provides free hosting for bug hunters. Please note that according to the Open Bug Bounty policy, you can only report bugs that do not imply active intervention, for example, you cannot report RCE and SQL.
First of all, we built the work within the company: we determined how the departments should interact with each other, who is responsible for fixing bugs, who monitors the scope and responses to reports.
Before starting the program, we would advise you to make sure that you did everything you could, found all possible bugs on your own. It is also important to know the inside and out of the product that you plan to put into the work of baghunters: what problems are there now, what were; whether systematic errors appear. This information will allow you to form the correct scope - succinctly, but comprehensively, to give input data and describe the request for bug hunters.
The platform has been selected, the preparation has been carried out - we begin
We form a scop
We are going to tell you what we understood during the year of the existence of the
Scopes program , filled out on the Bug Bounty platform, a kind of offer - an agreement between the company and the community of bug hunters. The following information should be included in the scope:
- indicate the purpose: the presence of any harmful effects or destructive consequences we would like to check
- information about bugs already known, irrelevant or uninteresting to the company (no reward is provided for finding them)
- rules and boundaries for searching for vulnerabilities that should be followed
- size of awards.
The more time you spend scoping, the better. Try to fill in all the items carefully and in detail, so that in the future you can solve all incidents based on the specified information.
An example of scoped sections with the content of points on the example of our current program:
How do you choose what to check first? We would advise you to include in the program the service or aspect that most likely needs to be verified the most. It is impossible to immediately understand what exactly needs to be looked for, you can only choose a direction, it is important to find the most problematic points. With each new report and each new program, we increased our experience and more clearly understood where to go next.
Until now, not all of our services and products are listed on the Bug Bounty platform. This was done deliberately, since some of the services, for example, were created on the basis of open source solutions: third-party teams are engaged in their development and support, so we believe that it makes no sense to exhibit them within our Bug Bounty platform, since our team only monitors the relevance of these services.
It is worth considering whether you can change the product included in the Bug Bounty program: is there a team that can develop it; do the nuances of architecture allow.
For our part, during the Bug Bounty programs, we constantly researched all services and the network on our own. This allowed us to save money: we fixed the found bugs and updated the scope.
An important component of the program is to determine the level of severity of the vulnerability found and establish the amount of reward. Try to capture a transparent relationship between the severity of the error and the size of the reward. The more transparent, the fewer questions for you! It is good practice to tie the size of the award to the CVSS scale(an open standard for assessing the criticality of vulnerability). In addition, Bug Bounty platforms usually display manuals and instructions on how to determine the amount of reward. Site managers can help you with this. To navigate the level of payment for the work of bug hunters, you can go to the HeadHunter portal and analyze the indicated salaries. If hackers are no longer active, it might be worth raising the reward.
At Timeweb, we independently assess the criticality of the situation depending on the impact on the business. Our scale of business impact severity includes 4 levels:
- low (receiving non-critical information about other users, for example, login name or the ability to change avatars; as well as violation of the integrity and availability of this information)
- medium ( , , ; )
- high ( “” ; : , , , ; )
- critical ( ; ).
In addition to the information provided above, we look at the type of vulnerability (RCE, XSS, SQL injection) and the importance of the server that was hacked or managed to be accessed.
Thus, in order to establish the level of severity of the found bug, we analyze the type of vulnerability, the importance of the server and the degree of impact on the company. Based on these criteria, we determine the level for each vulnerability found, which determines the amount of remuneration for the bug hunter. However, we add that it is impossible to take everything into account, and often when determining the level of criticality of a found bug, one cannot do without subjectivity.
A more detailed description of the vulnerability severity assessment process is given in the tables:
Server / service score table
Bug Bounty Timeline: How Was It?
Program # 1
Letting go: Timeweb.ru hosting panel Scop
: Scop was compiled on the basis of examples from other companies. Spoiler: Don't Do It! But we had to start somewhere.
Results: During the week we received 20 reports, mainly with the indication of critical vulnerabilities, and ... spent all the money that we put into the account (several thousand dollars). During these 7 days, we saw repeating patterns of problems: multiple problems with filtering input and displaying data, various violations of the application's business logic, as well as some other risks for OWASP Top Ten. We decided to suspend the program, and the next month we were only fixing the found bugs and analyzing them.
As soon as we analyzed these 20 reports, we understood what to do next: where to dig during development, how to properly work with security.
Program # 2
We give it up: Timeweb.ru hosting panel (again)
Scop: We fixed Scop based on the reports received earlier: removed the fixed vulnerabilities and focused on what is interesting to us.
Results: This time we received a lot of critical reports, however more specific and specific. All Bug Bounty tasks were identified as urgent. Thanks to the second program, we have adjusted the development processes and bug fixes.
Program No. 3
We are giving away to be torn apart: VDS Timeweb.ru panel , checkofficial site
Results: Received about 40 reports with bugs of varying severity.
Since our products are similar in functionality, sometimes they inherit not only functions, but also bugs from each other. It turned out that the new panel contained bugs that had already been found in previous Bug Bounty programs. Such bugs were registered in the scope as a duplicate, so we did not have to pay for them again.
Many problems with forms and XSS vulnerabilities were found in the site.
Program # 4
Scope: The main goal of the fourth program was to find SQL injection.
Results:Before launching the program, we independently studied how it works, and carried out research on our product ourselves: we found only 1-2 non-critical vulnerabilities. 2 weeks after the launch of this program at night, we received a long-awaited report: a bug hunter demonstrated an attack vector with a destructive effect on a billing database using blind-SQLi. We were able to quickly close this vulnerability within 5 minutes: it was in the previous global version of the kernel, which is still used as plug-ins for several actions in the latest global versions of control panels (virtual hosting, VDS, webmasters). We were delighted that we were able to detect such a serious problem and fix it in time. Among other things, we thoroughly checked all other similar code sections by analogy.
Program # 5
We give it to the mercy: Virtual hosting Timeweb
Virtual hosting is, roughly speaking, a physical server divided among hundreds of clients. Clients host their web applications and working directories on this server, have SSH and FTP access to the server. Each of our clients has their own clients. On such a server, our serving service scripts, services and other customizations are running, which interact with other serving services and databases.
Scope:The main goal for bug hunters here was simple: to find vectors for escalating privileges to root. We also expected a search for vectors of influence on other users and their resources, a search for vectors of influence on our service scripts and a search for vectors of attack on other servers, both virtual hosting and service servers, to which there would be a hypothetical accessibility.
Results: We have allocated a special server without clients for bug hunters, similar in functionality to production servers. At the moment, bug hunters have found only 10 bugs. Two reports showed attack vectors with escalation of privileges to root, but failed to reach other servers. These problems were immediately fixed.
Before launching the fifth program, we plunged into updating servers and software, dealing with weaknesses, deploying service services to a local network that is inaccessible to the external network. Thanks to this program, we refactor the internal systems on the server. This allowed us to notice errors even before they write to us.
And where did it take you?
About the results of the Bug Bounty program
For almost a year of the Bug Bounty program, we received 72 reports. Of these, 36 reports do not comply with the rules of our scope. However, bug hunters found 7 critical vulnerabilities, 9 high and 10 bugs of medium and low severity.
To get such a result, we spent more than $ 15,000 on remuneration for bug hunters (excluding platform fees). The smallest reward was $ 50 (for a vulnerability that allows you to receive information about the payment method for any invoice through IDOR). The highest reward paid so far is $ 1,500. Average remuneration: approximately $ 423.
As for the qualitative results: we
preserve the tone of the IB muscles
Since the Bug Bounty program implies a constant and continuous search for bugs, our information security team is on alert 24/7.
We can say that bug hunters simulate the actions of hackers. They create sanctioned “harmful” activity every day, forcing us to keep our eyes open and keep our guard.
keeping up with the trend
Baghunters use new services, unfamiliar utilities and modern hacking techniques. Thanks to this, our specialists can update their competencies and knowledge.
improving service and products
Information security in general and Bug Bounty processes in particular are always aimed at improving and developing services and products for the client.
engaging a community of experts
In the implementation of information security, not only our internal admins are involved, but also a full-fledged community of experts with various experience and knowledge.
We often asked bug hunters how to reproduce the found bug and even how to fix it. The specialists were ready to communicate directly in Telegram, record video, for which we are very grateful.
we observe discipline
Working with Bug Bounty platforms, we are responsible for closing bugs not only to ourselves and clients, but also to bug hunters, they are waiting for feedback. Interaction takes place in accordance with the established regulations, we must regularly update the information on current reports.
Within 3 days, we had to answer the bughunter to his report: is the error found a bug, what is its level of severity. Of course, we could give an answer within a week, but this behavior will not be hacker-friendly and may alienate bug hunters.
We understand users better
Baghunters are the same users and customers who provide us with a certain user experience in terms of information security.
Is there life after Bug Bounty?
We continue to work with a private Bug Bounty program with a proven platform. We would like to efficiently build all internal processes for closing and fixing bugs. As soon as we understand that the bug hunters we are currently interacting with have become less active, we will try to move to other, including larger, Bug Bounty sites in order to receive even more reports and find all possible problems.
Another area of development for us was the implementation of the principles of secure development. Typically, developers are doing functional programming, and security takes a back seat. It's important to make code security review a key part of any code review.
Timeweb team is trying to introduce new, modern tools for auditing information security of services. We learned about some of the resources through cooperation with baghunters.
Gradually, we involve technical support staff in solving information security problems: we train colleagues and strengthen the team.
We are ready to talk about the Bug Bounty program indefinitely. Perhaps you still have questions for our experts - write in the comments, what else is interesting and useful to read about. We will try to answer all the questions below or explain in more detail in the following articles.