It's rare to find a project that you fall in love with during the interview and which you are proud of when it conquers new markets. It is all the more pleasant when the professionalism of colleagues is at their best, and in your team you feel like in the bosom of your family. I was lucky not only to find such a project, but also some time ago to start influencing the testing process in it. I will tell you what is included in our understanding of the optimal process; how we came to monthly releases and how they work for us; and how we have adapted to the conditions of quarantine.
Our project is a distance learning system (LMS, Learning Management System), which is used by more than 7 million people in different countries of the world. The system contains 1000+ web pages and about 10,000 test cases.
Now the project employs about 15 development teams - from the customer's side in Norway and from Arcadia's side in Russia. I joined the project 8 years ago as a QA; For the last 2 years I have been working as QA lead, participating in the optimization of the testing process.
What is included in the concept of an optimal process
Our main task is to meet the needs of end users, which includes the creation of new functionality and system support. Particular attention is paid to the speed of the system and the stability of work under heavy load conditions.
The development process should primarily allow the sustainable achievement of business goals, such as getting work done on time and with an agreed level of quality. But the process should also be comfortable for the development team, so that everyone involved in the process is satisfied. In our teams, we have established an optimal process for us, and here are the means by which this is achieved.
Development process in general:
a) A development approach that meets the needs of the team
We work using Scrum and sprints lasting 3 weeks. Before the sprint, a presentation of its goals takes place and a set of requirements for this sprint is formed. Next comes the planning, in which we evaluate all the tasks and determine the set of tasks that will be included in the sprint. At the end of the sprint, a Sprint review takes place, where we demonstrate all completed tasks and announce the achieved goals. This approach is optimal for us: in a sprint we manage to make a sufficient amount of new functionality and at the same time fix and test a certain number of bugs from end users - 10% of the sprint time is allocated for such bugs.
Line-up: team lead, developers, testers. The ratio of developers to testers in our teams is 3: 1. This ratio makes it possible to achieve the sprint goal evenly - there are no periods when one of the participants in the process is idle: while developers make any change, testers create or update test cases related to this change; When development is complete, testers check the changes, and the developers either move on to the next tasks in the sprint, or help with testing (this is necessary when testing large-scale changes).
The Product Owner sets goals and requirements at the beginning of each sprint and accepts them at the end. Also, each team has a Scrum Master to help solve emerging problems.
In order for the team to have the opportunity to carry out various activities that are not directly related to the tasks of the sprint, as well as taking into account various risks, the time calculated for completing the tasks of the sprint is 80% of the total working time of the team. That is, 2 hours a day during a regular sprint are reserved for communications, various meetings and seminars, as well as fixing bugs found in the sprint.
b) Clear requirements and good planning
In order to ensure that planning does not drag out and the sprint does not become a failure due to previously unknown details and laborious additional changes added already in the sprint process, we try to take into the sprint only changes with sufficiently clear and precise requirements. If the project area that concerns the changes is unknown to the team, or during planning many questions arise that the Product Owner cannot answer, the team can sprint a task to study this area or a research task, which results in clear requirements or even some kind of prototype.
c) Retrospectives
Correcting bugs is also important. This applies to both sprints and releases. It happens that we do not have time for something, or we have to work overtime to be on time - for example, if it is necessary to pre-test a release (and these are additional costs that the company would like to avoid in the future). Therefore, we conduct retrospectives where we discuss what was good and bad in the sprint or release, and develop measures to avoid such mistakes in the future.
d) Management support
Sometimes questions or situations arise that cannot be resolved at the team level - you need to change the process or involve the company's management in the decision. It is very important to see that they want to help you and are ready to support you in such a situation. In our company, everything is fine with this - this concerns both Arcadia and the management on the part of the customer. All issues are discussed and a solution is always found that satisfies all parties.
e) And, in my opinion, the fundamental is good communication. And what we have in the company, for me, is one of the main advantages of comfortable work — benevolence, the desire to make compromises.
This applies to everyone: each member of the team, the Product Owner, the Scrum Master, the management of the company, and many other participants in the process. Everyone is open to questions and discussions, developers consult with testers on requirements and together decide how best (both from a development point of view and from a testing point of view) to make this or that change. The Product Owner, in turn, is constantly in touch with the team, promptly resolves all issues and always tries to meet halfway in achieving the sprint goals. The Scrum Master is always ready to help: find additional resources (testers / developers, if they are required for the sprint or for the release) or suggest how best to organize the sprint in time.
Testing process:
a) QA-standards (guidelines) related to writing test cases.
In our project, test cases are the main detailed documentation on the functionality of the system, so much attention is paid to them. For each change made by the team, a new test case is written or an existing one is updated.
Previously, when the system was not yet so huge, test cases were written in quite detailed, with various specific steps, but now this approach has become unacceptable - it takes a lot of time to update such test cases. Therefore, we (QA leads) have developed standards, the main requirement of which is to write test scripts with expected results instead of detailed steps in test cases.
b) QA standards related to testing in sprints.
Sprint testing standards have been developed to ensure that each team makes good quality changes.
These standards are based on maximum test coverage, which includes:
- Testing the changes made by the team directly (functionality and, if necessary, UI);
- Testing the system after this change using a checklist: these are mandatory checks, which include testing accessibility for people with disabilities, checking translations, checking existing data, testing using special characters, security checks, testing changes in a mobile application, and other checks ...
c) QA standards related to release testing.
The release process and the standards used in it are discussed in more detail below.
d) Use of automated regression testing.
The creation of automated tests for regression testing made the life of teams much easier - now, if you need to check some area after command changes, you can simply run regression autotests for the desired area of the system (if this area is covered by autotests). Also, if some part of the system has been broken by team changes, regression autotests will detect this.
e) Mutual assistance of testers, assistance of developers.
We do not have very many testers (on average, 1 tester for 3 developers), besides, from time to time they are distracted from the sprint tasks for testing releases, and there may not be enough time for everything.
In this case, there is always someone from the testers of other teams with less workload at the moment who can help. The QA lead or Scrum Master helps to find such a tester. In addition, developers can help with testing changes in the sprint if test cases have already been written for them.
f) Communication between testers.
For communication, a chat in Microsoft Teams is used, in which everyone can ask questions and receive answers promptly, while the rest of the testers will find out the latest information. We also hold monthly QA meetings, where testers share with each other the current tasks of their team and can discuss any issues - the approach to testing and / or the location of test cases related to an area unfamiliar to the tester; questions about the release (composition of the future release team, changing the testing timeline); additional mandatory checks added after missing a critical bug in the release, etc.).
The approaches listed above allow you to ensure good quality testing in a quiet operating mode.
:
We used to have quarterly releases . With such a time frame, there were a lot of changes that were made by the teams for each release. This volume of change certainly required a pre-release regression, so there was a separate time frame for regression and all teams were connected to it. This usually took about 2 weeks, and both testers and team developers participated in the regression.
After some time, autotests began to be used for regression in the release testing process. Not all regression tests were covered with autotests, some were manually tested. Overall, this reduced the regression testing time, but due to the size of the system, full regression was still time consuming.
The entire development cycle looked something like this:
It became obvious that this approach to releases is not optimal, is excessively labor-intensive, and does not allow for quick and flexible satisfaction of end-user requests. The approach to the release process required changes, and it was decided to release more often - once a month.
When we moved to monthly releases , it became clear that there was no time to run regression during the release testing phase - even with partially automated regression tests. Now the total preparation period for the release takes only 2 weeks. Therefore, now regression is done in sprints and only when necessary (if the developers report that a regression of a certain part of the system is required).
But since at the stage of testing the release it is necessary to check not only the new functionality, but also the general state of the system, we began to use stabilization instead of regression.
Stabilization includes:
a) testing new functionality included in this release by each team;
b) Critical Areas Testing is a test of the basic functionality of the main areas of the system (which obviously takes much less time than a full regression cycle);
c) testing bugs found in team changes for this release.
The entire development cycle now looks like this:
Let's consider what else is needed to prepare for the release.
When the stabilization is finished and the release package is of good quality, before rolling it out to production, the configuration is tested in the pre-production environment. Thus, Operations practice updating the environment, and testers check the configuration with the current release package before the real release.
You might think that 2 weeks of preparation for the release is too much, and there is little time left for testing in the sprint. But usually it takes 4-6 days for a tester to prepare for a release. It depends on the:
- the complexity and scope of functionality that his team is going to release,
- participation of the tester in the release team of the current release.
All testers of the project (including the release team) participate in stabilization testing; testing the configuration and the release itself is done only by the release team.
The general release testing schedule looks like this:
Since critical areas and configuration contain constant tests, we have partially automated these test cases. This has significantly reduced the testing time in preparation for the release, so in the future we plan to expand the use of autotests.
From the point of view of organizing testing, a QA coordinator (someone from the QA leads) is selected for each release. The lead time for the release is usually planned for a QA coordinator more than for regular testers, since the QA coordinator has more responsibilities. These include:
- defining the release team and checking the availability of all its members at the time of release;
- preparation of test plans for the release;
- launching autotests at the stage of stabilization and configuration testing;
- coordination of testers' work during release testing;
- help testers in solving all kinds of issues related to the release;
- control over the execution of testing and, if necessary, redistribution of the load to assist in testing overloaded teams (these can be either testers of other teams who have already completed release testing, or team developers, or the QA coordinator himself).
And of course, no release is complete without problems. We solve them and develop a plan on how to avoid them in the future. Here are some that we have come across lately:
- : , - , , - .
: . - : pre-production . — .
: . - : , .
:
a) - ( , ),
b) ,
c) ,
d) , . , , .
Solution: in such cases, we try to have time to test everything necessary for the release, even if it requires participation in the release for all 2 weeks or overtime work, since the release is a higher priority task than the sprint tasks.
But no matter what problems arise, we can always solve them by the forces of the release team or by involving people competent in a particular area - the main thing is that this is always well-coordinated team work leading to a good result.
Working in quarantine: how to keep testers working
In our project, work from the office is a prerequisite. This is done so that any participant in the process can be found during working hours - if suddenly he does not answer in chats, then the people working with him can inform him that he is needed by some of his colleagues. This is relevant, because some of the teams are in Norway, and some in St. Petersburg.
During the pandemic, when both Norway and Russia sent the majority of the population to self-isolation, we had to switch to remote work.
We continued to work as usual: teams still finished sprints with good sprint productivity, and releases were released on time.
Communication still remained at a good level - the Teams application covered all the needs: there were active conversations in the chat, meetings were held without problems; if there were any questions that needed to be urgently discussed, they called up any project participant.
From the point of view of organizing testing, there were no problems: during business hours, all testers answered in chats and promptly participated in testing the release. In case we need to find someone, but he does not answer in the chat, we have prepared a list of mobile phone numbers, but we never had to use it.
The only problem we faced while sitting at home at a remote work - due to the peculiarities of the VPN, it was impossible to test the system in a team environment from phones / tablets. But this problem was circumvented - thanks to the project manager and IT service, who found a solution. We started using proxies when connecting over a home network, and now we can test on mobile devices from home.
Now we are partially returning to work in the office, but part of the company continues to work from home. And even being in different parts of the city, we still remain a single close-knit company that continues to work in such difficult times, with such non-working conditions when children and pets demand attention, the Internet slows down and periodically disappears, the lights turn off, and there are so many distractions factors that you don’t understand at all, how did you manage to work again ?!
But together we will survive all this, and finally returning to a full-fledged office life, we will smile at each other even more widely than usual.
Conclusion
Summing up the above, I would like to note a few points:
- , , . — .
- , , .
- , , , — . , .