In this article, we will compare the testing capabilities of native and cross-platform applications. I will share my impressions of working with Flutter and tell you what tools we use in Surf when testing, why Flutter is convenient and what problems we encountered.
Testing capabilities in Flutter are on par with native ones
When a company changes its approach to development or a new technology emerges, it is important that testing capabilities are not severely affected. Ideally, when working with a new language or framework, QA specialists continue to use the familiar stack of tools and technologies that have proven themselves in the best way.
When testing native apps, we at Surf use autotest and read and replace packages. Nowhere without autotests, especially with regression, and without the help of a proxy, the variability of the application and the coverage of many cases decrease.
It was important for us that the familiar features remain when testing Flutter applications.
Autotest
Surf works with the Calabash and Ruby frameworks for autotesting native apps. When Flutter appeared, the first thing we wondered was: is it possible not to use Calabash, but at the same time fully work with autotests the way we are used to - or even cooler.
It turned out that it is not only possible - but even without third-party services: in Flutter, integration tests and testing of widgets in the console are available out of the box. Our Flutter developer told about this in more detail in the article about autotesting on Flutter .
Autotests on Flutter are both cross-platform and native at the same time: you can write tests inside the application project, and they will work on both platforms. When the whole project is in front of your eyes, you can add missing id-schnicks or even find bugs and fix them - this is another opportunity to improve the quality of the application.
Flutter also supports Behavior Driven Development - BDD - approach. We use it for UI tests. We chose Gherkin as the language: you can use feature files in it, write scripts in English and Russian. It is understandable, it has the implementation of script steps without additional arguments inside or with them, the ability to customize the launch of autotests: for example, running some scripts by tags, and not all written tests as a whole.
To use Gherkin when testing Flutter applications, we connected the flutter_gherkin opensource framework .
When we realized that there are autotests on Flutter, we wondered what are the differences between Calabash and Dart + Gherkin technologies, which approach is better. Let's compare them together.
1. Feature files are absolutely identical for both approaches.
For example, the script for authorization by pin code will be correctly interpreted in both Dart and Ruby using Calabash:
: ( )
Both technologies support Russian, English and other languages.
2. Steps are different in implementation.
| Dart + flutter_gherkin
|
Calabash
|
|
|
3. Flutter uses an additional .dart file to configure and work with tests. With Calabash, no such single file exists.
In our opinion, it is impossible to say that this is a flutter in Flutter or Calabash - this is just the specifics of working with specific tools and technologies.
4.For the convenience of working with elements in the application, it is necessary that each element has its own id. When working with autotests with Calabash, you need to take care in advance that both platforms have id in the tested applications. On Dart, you can add id in the process of writing autotests, without reloading the files of iOS and Android applications separately - this is convenient and saves time.
Our conclusion: autotests on Dart are not inferior to autotests using the Calabash framework.
Proxies
To increase application coverage by cases, Surf uses programs to read and spoof traffic - for example, Charles. Analysis of client-server interaction allows:
- Determine if there is real interaction with the backend.
- Find out on which side the problem is: on the client or on the server.
- , .
- : , , . Charles , , , .
Dart has its own client for working with the network. Since all requests go through it, the necessary settings for working with a proxy must be entered inside the application. For the convenience of testers, all the necessary settings are placed on a separate screen: in Surf we use the Debug Screen, which we developed ourselves.
The Debug Screen is an additional settings screen that is only available from the debug build and helps with testing. In it, you can select the required server, enable the use of reading and saving http requests in the application, view the fcm-token of the device and more - there are many opportunities for testing.
Debug Screen is custom: developers add additional elements to it at the request of testers - for example, fields for configuring proxies from the application. Therefore, we have full opportunities to work with Charles: you can connect a proxy server to Debug Screen without restarting the application.
As you can see, the possibilities for testing Flutter applications are not limited. Everything that we are used to working with when native is convenient and easy to use. This is good news.
Problems: framework bugs, flaws in third-party libraries, expected native behavior
The problems we face when testing Flutter applications are also native. It cannot be said that these are Flutter's specific shortcomings: in any technology, problem solving is not always obvious and simple.
Let's tell you what to look for when testing Flutter applications. Forewarned is forearmed.
Flutter-framework bugs
During testing, we encountered a problem with displaying and formatting fonts on iOS: the letter spacing on the iOS platform was noticeably wider than on Android. This caused a lot of visual bugs.
It turned out that the problem was in the framework itself. When our mobile app developers approached the guys from the Flutter framework developer community with a request to fix such an unpleasant bug, the framework was updated soon and the text display on iOS was fixed.
Surely such situations will be repeated. But this cannot be called a big problem: the guys from the Flutter community quickly respond to issues, support and develop the framework.
Deficiencies in third-party libraries
On iOS versions 10 and 11, implementation flaws were encountered in third-party libraries. For example, we fixed a bug when the permission to access notifications pops up immediately when the application is launched, and not on the button, as planned by the technical specification and design.
Such problems can occur both in the cross-platform and in the native. They are solved by fixes within the project, or together with the library developers.
Dealing with Expected Native Behavior
With long-term use and testing of native applications on iOS and Android, it is easy to predict user expectations from different application behavior. So, for example, going back with a backwipe on iOS is a standard gesture. And on Android, you don't need it.
System dialogs on both platforms are different: in iOS, you need to request permission to access notifications, and on Android, this access is given by default.
It is these parts of the OS specifics that often have to be finished manually. And sometimes - to cut, if suddenly the expected behavior for the iOS platform migrated to Android, as was the case with backwipe.
In native applications, problems such as updating the screen, incorrectly minimizing the application, and the operation of an application that is unusual for the current OS are rare: tools and technologies for developing an application for a specific platform are obviously aimed at covering all versions and capabilities of a specific system.
When testing one of the Flutter applications, we faced an interesting situation: the ability to update the screen was not available on iOS devices with bangs - starting with the iPhoneX and higher. At the same time, iOS devices without bangs and Android functioned correctly.
We encountered another bug on Android version 6: when minimized, the application was completely unloaded from memory.
Such bugs were fixed by our developers inside the project.
iOS are well aware of their devices and systems, what chips they release with the new version of the OS and which will no longer work in previous versions, what to focus on when updating the same Swift. Android understands that they have to target a large number of devices and completely different screen sizes, and they also know their specifics.
I would like the cross-platform to contain all the subtleties of implementation from native development. Of course, there are some shortcomings when working with Flutter, but this is not a problem: you just need your own approach to the cross-platform.
Benefits: one codebase, one development team
A single codebase reduces testing time.
The reason for bugs can be fuzzy technical specifications, lack of rendered states in the design, backward compatibility when updating the back-end. It is easier to create such bugs, because you need to create half as many tasks, and this already saves time.
You can search for bugs and check features on one platform: with a high degree of probability, they are repeated on both platforms - after all, there is one implementation.
The logic of new features on both platforms is also the same, since the same code is written: testing complex processes within an application is reduced to testing them on one platform and confirming them on another. We carry out a full cycle of activities on one platform: we do exploratory testing, feature runs, smoke / sanity / full, analyze feedback. After that, it remains only to confirm the quality by exploratory testing on another platform. This approach saves logic testing time by about 1.3 times.
, , , : , . , .
, (, iOS), , (Android), event .
If assemblies for both platforms need to be delivered to the customer on the same day, they come out at the same time and there is only one QA engineer on the project, there may not be enough time for verification in native development: the testing cycle must be performed on both platforms separately. We save time by testing cross-platform applications: regression testing of both platforms takes place within a single cycle.
We tried to roughly evaluate the testing of two similar projects - one on native Android and iOS, the second on Flutter - and compared them apically.
Analysis and testing was performed on one iOS device and one Android platform device. As you can see in practice, Flutter really gives a gain in time, albeit not twice. This is understandable: you cannot completely remove testing on one of the two platforms. Whatever one may say, they have different specificity and focus on the user.
|
|
Native iOS
|
Native Android
|
Flutter Android + iOS
|
| Password recovery
|
2h
|
2h
|
3h 20m
|
| Authorization
|
1h 30m
|
1h 30m
|
2h 20m
|
| Push notifications
|
2h
|
2h
|
4h
|
When testing a ready-made feature that does not completely affect the specifics of the operating system and is not custom-written for each platform separately, the time for checking a Flutter application on both platforms is reduced by about 1.3-1.5 times. For example, authorization and password recovery features that do not have specific behavior on different platforms reduce the testing time of the Flutter version by 1.3 times.
As for features that require custom behavior from each platform, you shouldn't wait for a reduction in time. The behavior for iOS and Android is expected to be different, which means that both platforms need to be tested completely and separately from each other. For example, it is often necessary to test push notifications in full cycle on both platforms due to differences in permissions, work with notifications connection, pusher settings for sending notifications on iOS and Android, as well as other implementation subtleties - so that the user's usual work with notifications is supported. TK and design were respected.
It is easier to organize communication within the team
When there are a lot of people in the project, it is difficult to organize the process so that even the smallest subtleties do not pass by. Especially if there are a lot of improvements ahead, implementations of new features and changes in general. Most of the problems are solved when the development team is one.
First, it is easier to test an application by implementing a single command than working with two different implementations on two platforms. The quality assurance professional is, of course, interested in having complete information about the status of the application, both on the iOS platform and on the Android platform. It's easier when working with Flutter.
Secondly, there is an important point in native development: about the changes that one platform has learned and accepted, it is necessary to notify the other platform, which is sometimes forgotten or lost in the course of large and intensive changes. There is no such problem when developing Flutter applications: there is only one team - that is, one task for revision applies to both platforms.
We love testing Flutter applications
Being a part of a cool community
For us, a new framework is a plus: solving non-standard problems, we broaden our horizons. We find many interesting and unique bugs that develop our skills and capabilities when testing applications.
At the same time, developers from the Flutter-framework community quickly give feedback on identified problems, improve the library and allow us to contribute to the project: we are moving forward together, and it's nice.
Be a specialist
When working with cross-platform applications, it is important to keep in mind the differences in operating systems and not lose focus on the specificity of the platform. Seeking and seeing differences where they should be minimal in theory, learning something that you have never encountered before - such work increases professionalism.
When developing and testing native apps, it is impossible to build an iOS app from, for example, Android Studio or Visual Studio Code. When working with Flutter, the IDE is the same for both Android and iOS. That's cool.
Be independent
When working with Flutter, at Surf we are faced with very different projects: from e-commerce to banking. Practice has shown that a QA engineer can single-handedly test both platforms. It is necessary to connect another specialist only closer to the release, when the pace of work increases, and the time for completing tasks is running out.
Flutter - a step forward
Testing cross-platform apps isn't difficult. Sometimes it is even faster and more convenient than working with native ones. All the difficulties that one may encounter do not overlap the convenience and advantages.
Experience in developing and testing Flutter applications has shown Surf that this framework is a big step forward.