Where there are developers, there should be testers nearby. And the more complex the system, the more important the role of the latter. But not all testers perform the same function in the same way, and today we want to talk about the emergence of a special unit in M.Video-Eldorado, which deals with cross-system testing. Read about why we decided to create a separate caste of testers, how it helped the business and how we came to this decision under the cut.
If you do not check whether the system will work after the update is rolled out into production, a small bug can "put" the entire client service or, for example, back-office processes. To prevent this from happening, any agile team should have its own tester who checks the execution of basic scenarios and reports bugs to the developers. If a service interacts with other components of the system, then teams often conduct integration testing - checking the influence of the closest components of the service ecosystem on each other.
But in our time, business processes are often much more complicated, and to achieve the desired result requires the work of a dozen or so services. In this case, it is necessary to test for compatibility of changes not only two or three neighboring systems, but the entire stream.
Therefore, integration tests no longer guarantee error-free operation. Here is an example: when the developers of a mobile application release a release, then, as a maximum, its compatibility with Bitrix (on which the Eldorado website is running) is checked. This allows you to evaluate the work of only the service itself, but does not give any guarantee that the orders will be fulfilled, the goods will be provided to customers, and the prices for issuance will correspond to the prices in the application, and so on.
In fact, complex processes are the modern norm. For example, in M.Video and Eldorado, online ordering requires the participation of 20+ systems. From the side of the client, everything looks simple: the buyer simply goes to the mobile application or to the website of the online store, selects the product, puts it in the basket and assigns delivery to the next convenient date for him. The courier delivers the goods at the time indicated by the buyer to the specified address.
But to make all this possible on the part of IT services, there is an interaction of authorization systems, site engine, price calculation module, payment gateway, CRM system, accounting, and so on. And in order to assess whether some change in one of the components will lead to a “breakdown” of the purchase process, it is necessary to check the interaction of all systems.
- ?
On the one hand, each team already has its own tester. And, it would seem, he can be involved in end-to-end tests. But we have seen from our experience that this approach is not the most effective. A cross-system test is, by definition, difficult; to carry it out, you need to take a tester from each system, instruct, and put them on the job.
After completing his stage, tester N must report to tester N + 1 that his stage has been completed. guys are usually busy with their own business - functional and integration tests, and the cross-system testing process is slow.
A real case: the tester went to lunch, but when he returned, he forgot that he had to complete his stage - he has a lot of other work. Until he is reminded, the process is worthwhile.
If we find a problem at some stage, then the question immediately arises: "On whose side is it?" For example, tester N + 1 returns an error to the previous specialist. And the tester N is indignant: “What do I and our system have to do with it?”. Until the guys figure out on whose side the problem arose, which they sincerely do not consider “theirs”, it may take several days. The result is missed deadlines and the need to constantly push the process.
Finally, to conduct such a test, you need to launch a project, connect a Project manager, periodically gather 20 people in one room (one for each system) and figure out on whose side the problem arises when taking the next step. This is very labor-intensive, inefficient, and distracts people from the work that they rightly consider to be their main job.
We decided that it would be better to create a cross-system testing department, which now tests the entire business processes before the release of all key updates to production.
Own experience - PROMO-actions
When we decided to do this, a big plus was the presence of internal experience in cross-system testing. Back in 2017, we started implementing a new end-to-end testing practice to support federal advertising campaigns for the M.Video brand. The company constantly conducts various promotions, for example, “Your price”, “Buy a TV, you will receive a gift,” and so on.
Such promotions imply special prices, the presence of gifts in the kit, and the issuance of gifts. Promotions are held simultaneously, combine or exclude each other in a specific purchase.
But, in the process of pick-up, for example, the purchase of promotional goods took place in one system, the work of the call center employee - in another, and the receipt of the goods - in the third. As a result, the TV could be left without a gift, or when another item was added to the receipt, the promotional price for the main position was canceled. Obviously, any updates to the systems could break something, and we created a small team that was specifically engaged in end-to-end testing of the promo before the launch of federal promotions.
About a year later, this team built an efficient work process and went to the error-free and trouble-free operation of federal promotions. And as an additional bonus, the business began to receive feedback and reports on the technical feasibility and effectiveness of launched advertising campaigns.
When we realized that the company needs a full-fledged cross-system testing process on other projects, this experience turned out to be very useful. The need for a new type of tests was especially clear when M.Video and Eldorado merged. We have increased the number of complex and volumetric changes, it became necessary to conduct projects for two brands at once, the number of products and back-office systems included in one change has exceeded 20.
Benefits of the new approach
Today, the cross-system testing department employs three test managers and six testers, and in the future we plan to permanently employ about 40 people in cross-system testing for M.Video-Eldorado.
The task of our department is to carry out end-to-end testing, catching those bugs that might not have been noticed during functional and integration testing, before the release of changes launched in the processes and products of the company into production. The division assesses the impact of updating each of the components on internal and external processes. For this, a unified methodology for preparing and running tests was developed, as well as rules for interacting with internal testers of each development team.
By mid-2021, M.Video-Eldorado already supports more than 100+ products and back-office systems / services, and, accordingly, 100+ teams are engaged in their development. Every week there are more and more software components (after all, we live in the era of microservices), and the changes are almost continuous. The company constantly runs 5-6 large-scale projects that affect dozens of systems at once.
In such conditions, the process of continuous cross-system testing becomes necessary, because it allows you to establish a timely check of the interaction of services and prevent incompatible updates from entering the product.
If we talk about specific indicators, then we managed to :
- , -, - end2end 2 ;
- 15%;
- 20%;
- end2end 10%;
- 50%;
- , -, Jira
The story of how we conducted the pilot project and on what business processes we worked out the testing methodology deserves a separate text. Therefore, in the next post, we will talk about our path to ubiquitous cross-system testing, as well as share the existing methodology and criteria for the successful implementation of end2end tests in the company, which we have developed from our own experience.
PS There are several interesting vacancies in our team . Welcome!