This is exactly what happened in my project, and today I will tell you how you can further reduce the number of test cases without losing quality.
Test object
First, I'll tell you a little about the product. At Tinkoff, our team developed blocks - these are React components consisting of implementation and configuration. The implementation is the component itself, which we have developed and which the user sees in the browser. The configuration is JSON that sets the parameters and content of this object.
The main task of the blocks is to be beautiful and appear the same for different users. At the same time, the block can change very significantly from the configuration and content.
For example, a block can be like this - without a background, with a button and a picture on the right:
Or like this - with a background, without a button and with a picture on the left:
Or, in general, like this - with a link instead of a button and without a list in the text:
All the examples above are the same block that has one version of the configuration (a JSON structure that this particular React component can handle), but its different content.
The circuit itself:
{
components: {
background: color,
panel: {
panelProps: {
color: {
style: ['outline', 'color', 'shadow', 'custom'],
background: color
},
size: ['s', 'm', 'l'],
imagePosition: ['left', 'right']
},
title: {
text: text,
size: ['s', 'l'],
htmlTag: ['div', 'b', 'strong', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
},
description: {
text: html,
htmlTag: ['div', 'b', 'strong', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
},
image: {
alt: text,
title: text,
image: {
src: image,
srcset: [{
src: image,
condition: ['2x', '3x']
}],
webpSrcset: [{
src: image,
condition: ['1x', '2x', '3x']
}]
},
imageAlign: ['top', 'center', 'bottom']
},
button: {
active: boolean,
text: text,
color: {
style: ['primary', 'secondary', 'outline', 'outlineDark', 'outlineLight', 'textLink', 'custom'],
backgroundColor: color
},
onClick: {
action: ['goToLink', 'goToBlock', 'showBlock', 'crossSale', 'callFormEvent'],
nofollow: boolean,
url: url,
targetBlank: boolean,
title: text,
noindex: boolean,
guid: guid,
guidList: [{
guid: guid
}],
formId: guid,
crossSaleUrl: url,
eventName: text
},
htmlTag: ['div', 'b', 'strong', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
},
href: url
}
}
}
In this case, the block with the picture on the right will have a value
components.panel.imagePosition = right. And the block with the picture on the left has components.panel.imagePosition = left. For a block with a button - components.button.active = trueand so on. I hope the principle is clear. This is how all the parameters of the block are set.
Cases from a combination of parameters
In this article, I will not touch on the issues of block diagram versioning, content filling rules or where the data comes from. All these are separate topics that do not affect the compilation of a set of test cases. The main thing to know: we have many parameters that affect our component, and each of them can take its own set of values.
For the example above, I chose a block with a fairly simple configuration. But even in it, checking all combinations of the values of all parameters will take an unacceptably long time, especially if you have to take into account cross-browser compatibility. Usually Pairwise Testing, or pairwise testing, comes to the rescue here. Tons of articles have already been written about him and there is even training . If you suddenly did not come across - be sure to read.
Let's estimate how many test cases we will get when applying it. We have more than 25 parameters, and some of them take as many as 7 and 9 variants of values. Yes, you can neglect something: for example, if you are checking the layout, the guid is not important to you. But using Pairwise Testing, you will still get more than 80 test cases. And this, as I already wrote, is for not the most complex block and without taking into account cross-browser compatibility. We now have more than 150 blocks, and their number is growing, so we cannot afford so many cases if we want to keep the speed of testing and releasing new versions.
Cases from one parameter
The pairwise testing method is based on the statement that most defects are caused by the interaction of no more than two factors. That is, most bugs manifest themselves either on one value of a parameter, or on a combination of the values of two parameters. We decided to ignore the second part of this statement and assumed that when checking one parameter, most bugs would still be found.
Then it turns out that for testing, we need to check each value of each parameter at least once. But at the same time, each block carries the entire configuration. Then, in each new case, you can check the maximum of not yet verified values in order to minimize the number of cases.
Let's analyze the algorithm for constructing cases using a simplified example. Let's take the button component from our scheme and compose test cases for it:
button: {
active: boolean,
text: text,
color: {
style: ['primary', 'secondary', 'outline', 'custom'],
backgroundColor: color
}
To simplify the example, I have reduced the length of the list to
button.color.style.
Step 1. Compose content options for each field
Everything here is like in pairwise testing: you need to understand what values each of the fields can take. For example,
button.activein our case, there may be only two values: trueor false. Theoretically, more options may arise, for example, the undefinedabsence of the key itself.
Here, in my opinion, it is important to very clearly define the boundaries and functionality of your system and not check unnecessary things. That is, if checking for mandatory keys or validating a value is implemented in a third-party system, then this functionality needs to be checked in a third-party system. And we should only use “correct” data as cases.
By and large, the same principle is used in the testing pyramid. If desired, the most critical integration tests can be added to us - for example, to check the processing of a key that has not arrived. But there should be a minimum number of such tests. Another approach is the pursuit of exhaustive testing, which, as everyone knows, is impossible.
So, we have identified the content options for each field and made the following table:
This table includes each equivalence class of each parameter, but only once.
These are the value classes in our case, the classes:
- text_s - short string;
- text_m - longer string;
- no_color - no color;
- rnd_color is any color.
Step 2. Enriching the table with data
Since each block carries the full configuration, we need to add some relevant data to the blank cells:
Now each column is one case.
At the same time, since we select the missing data ourselves, we can generate cases based on priority. For example, if we know that short text is used in a button more often than medium-length text, then it is worth checking it more often.
In the example above, you can also pay attention to the "dropped" cases - cases in which some parameter is not checked at all, although it is present in the table. In this case,
button.color.style: secondaryit will not be checked for appearance, because it does not matter what style the disabled button has.
To prevent the "dropped" cases from leading to bugs, we used to analyze the resulting sets of values. The analysis was performed once during the generation of test cases, and all "dropped" cases were added manually to the final test case. Such a solution to the problem is rather clumsy, but cheap (unless, of course, you rarely change the configuration of the tested objects).
A more general solution is to split all values into two groups:
- unsafe values (those that can lead to the "loss" of cases);
- safe (which cannot lead to "falling out").
Each unsafe value is checked in its own test case, you can enrich the case with any safe data. For safe values, a table is compiled according to the instructions above.
Step 3. Clarifying the values
Now all that remains is to generate concrete values instead of equivalence classes.
Here, each project will have to choose its own variants of values, based on the characteristics of the tested object. Some values are very easy to generate. For example, you can simply take any color for most fields. For some blocks, when checking the color, you have to add a gradient, but it is moved to a separate equivalence class.
With text, it's a little more complicated: if you generate a string from random characters, then hyphenations, lists, tags, non-breaking spaces will not be tested. We generate short and medium line lengths from real text, trimming it down to the desired number of characters. And in the long text we check:
- html tag (any one);
- link;
- unnumbered list.
This set of cases stems directly from our block implementation. For example, all html tags are connected together, so there is no point in testing each one. In this case, the link and the list are checked separately, because they have separate visual processing (highlighting on hover and shootouts).
It turns out that for each project you need to compose your own actual set of content based on the implementation of the tested object.
Algorithm
Of course, at first glance it may seem that the algorithm is complex and not worth the effort. But if you omit all the details and exceptions that I tried to describe in each paragraph above, it turns out quite simply.
Step 1. Add all possible values to the parameter table:
Step 2. Duplicate values into empty cells:
Step 3. Turn abstract values into concrete ones and get cases:
Each column of the table is one case.
Advantages of the approach
This method of generating test cases has several important advantages.
Fewer cases
First and foremost, there are significantly fewer cases than in pairwise testing. If we take a simplified example with a button, we got 4 cases instead of 8 in pairwise testing.
The more parameters there are in the tested object, the greater the savings in cases will be. For example, for the full block presented at the beginning of the article, we get 11 cases, and with the help of pairwise - 260.
The number of cases does not inflate with the complication of functionality
The second plus is that when new parameters are taken into account during testing, the number of cases does not always increase.
For example, suppose a parameter
button.color.textColorwith values equivalence classes no_colorand is added to our button rnd_color. Then 4 cases will remain, just one more parameter will be added to each of them: The
number of cases will increase only if a parameter has more values than there were cases.
You can check the important more often
By enriching the values (step 2 in the algorithm), higher priority or riskier values can be checked more often.
For example, if we know that earlier users used shorter text more often, and now they use longer text, we can enrich cases with longer text and more often get into real user cases.
Can be automated
The above algorithm is quite amenable to automation. Of course, the cases generated by the algorithm will look less like real ones than human-generated ones. At least by matching colors and cropping text.
But on the other hand, already in the development process without the participation of a tester, cases appear, which greatly reduces the feedback loop.
disadvantages
Naturally, such case generation is far from a silver bullet and has its drawbacks.
Difficulty analyzing the result
I think you noticed that in the process of generating cases, test data is mixed with each other. Because of this, when the case falls, it becomes more difficult to identify the cause of the fall. After all, some of the parameters used in the case do not in any way affect the test result.
This really makes it difficult to parse the test results, on the one hand. But, on the other hand, if the object under test requires a large amount of required parameters, this also makes it difficult to find the cause of the bug.
Bugs may be missed
Returning to the very beginning of the article: when using this method, we allow the possibility of skipping bugs caused by a combination of two or more parameters. But we win in speed, so it's up to you to decide what is more important for each specific project.
In order not to miss bugs twice, we introduced the Zero Bug Policy and began to close each missed bug with an additional test case - no longer automatically generated, but written by hand. This gave excellent results: we now have over 150 blocks (tested objects), several releases per day and from 0 to 3 missed non-critical bugs per month.
conclusions
If your tested object has a wide range of input parameters and you want to try to reduce the number of cases and, as a result, the time for testing, I recommend trying the above method of generating cases using one parameter.
In my opinion, it is ideal for front-end components: you can reduce the time by more than three times, for example, for checking the appearance through screenshot testing. And the development will go faster due to the appearance of cases at the earliest stages.
Of course, if you are testing the autopilot of the new Tesla, you cannot neglect even the small probability of missing a bug. But in most cases, do not forget that speed in the modern world is a very important quality criterion. And the increase in speed gives more positive results than a couple of minor problems found.
And for the most responsible, in the next article I will tell you how you can additionally protect yourself from tricky bugs caused by a combination of parameters using custom cases and StoryBook.