Hello! My name is Mikhail Bulgakov (no, not a relative), I work as a release engineer at Badoo. Five years ago, I started automating the release of iOS applications, which I covered in detail in this article . And then he took up Android applications.
Today I will sum up some results: I will tell you what we have come to during this time. Long story short: any employee involved in the process can release at least all our applications on both platforms in a few clicks - without headaches, time consuming, registration and SMS. So, our department of release engineers in 2019 saved about 830 hours.
For details - welcome under the cut!
What is behind the mobile release
An app release on Badoo has three phases:
- Development.
- : , — , App Store Google Play.
- , -.
When the application is completely ready and the first stage is completed, it is important not to fix it at the release stage and bring the product to the “counter”. This last stage seems to be the easiest, but in fact it takes a lot of time and its success depends on a few people.
Most of the time is spent on preparing the application window on the App Store or Google Play: you need to fill in beautiful screenshots, make an enticing description optimized for better indexing, select keywords for search. The popularity of the application directly depends on the quality of this work, that is, in fact, the result of the activities of developers, testers, designers, product managers, marketers - all involved in the creation of the product.
If the application must exist in several languages, at least a separate person is required to prepare the showcase, or even several employees: a product manager who will write texts for the description, organize translation into all languages and prepare a ToR for creating screenshots, a designer who will draw screenshots with text overlaid, device contours, etc., and, of course, translators who translate all screenshots and texts into different languages.
The final part of the work is the release process itself. It takes a considerable amount of time for a small release engineering team. At this crucial, but rather routine stage, we tried to minimize the number of errors and the influence of the human factor. To do this, first of all, it was necessary to automate the loading of metadata (text and graphic design of the application showcase): this allows you to significantly reduce time costs and quickly implement business releases (for example, styling the application for Valentine's Day).
Since the decision on whether the application is ready for release in Badoo is made by a team of QA engineers, we decided to give them the right to click on the “red button” for launching the release. At the same time, we wanted it to be accessible even from mobile devices (with a visual display of the progress of the script).
First Steps Towards Automation: Loading Metadata
How it worked at the very beginning: for each release, a table was created in Google Sheets, in which the product manager uploaded the verified master text in English, after which the translators adapted it for a specific country, dialect and audience, and then the release engineer transferred all the information from this table on the App Store or Google Play.
The first step we took towards automation was to integrate the translation of texts into our overall translation process. I will not dwell on this - this is a separate large system, about which you can read in our recent article... The main point is that translators do not waste time on tablets and work with the interface for convenient loading by hand (read: ctrl + c ctrl + v) translated versions in str. In addition, there are the makings of versioning and the foundation for Infrastructure-as-Code.
At the same time, we added unloading ready-made translations from the database and embedding them in the collected IPA file (iOS application file extension). The application is built with TeamCity. Thus, each version of the application always had a fresh translation without manual intervention in the assembly process.
For a while we lived like that, and on the whole, everything suited us. But the number of applications increased, and with it the time to prepare each release.
Our reality as of 2015
On average, the release of one application in the presence of the current version of the screenshots took about one and a half to two hours of work by the release engineer in the case of iOS and about half an hour in the case of Android. The difference is due to the fact that iOS applications have to go through the so-called Processing, which takes some time (it is impossible to send an application to Review before the successful completion of Processing). In addition, the App Store itself, by most operations at that time, was much slower than Google Play.
It became obvious that we needed an additional tool for delivering applications to stores. And just at that moment, a product called Fastlane began to gain popularity in the open-source market. Despite the fact that he was still damp then, he could already solve a huge layer of our problems ...
I will say a few words about him, so that it would be more clear what will be discussed further.
Fastlane at a glance
Today Fastlane is a product that is able to almost completely automate all actions from the moment of development completion to the release of an application in the App Store and Google Play. And it's not just about downloading texts, screenshots and the application itself - here is certificate management, and beta testing, and code signing, and much more.
We met Fastlane during his “youth” and instability. But now it is a confidently working and integral component of many mobile application development teams that face the huge time-consuming problem of delivering their products to users. The most interesting thing about it is the ability to write your own plugins for the main component and use plugins written by the community. For such a specific product, this is a good solution, which (which is important) helps not to produce "extra" technologies in DevTools.
Confidence is also inspired by the fact that the founder and chief developer of Fastlane was hired by Google: now the component supports not only the community, but also Sam.
Over time, we have implemented most of Fastlane's capabilities into the build, sign, fill, and more systems of our applications. And they are incredibly happy about it. Why reinvent the wheel, and even maintain its correct shape, when you can write a unified script once that will spin itself in a CI / CD system?
Automate iOS releases
Due to the fact that Google Play is more friendly to developers, it took very little time to release an Android application: a couple of minutes without updating texts, videos and screenshots. Hence the lack of need for automation. But with the App Store the problem was very tangible: it took too much time to send applications to Review. Therefore, it was decided to start automation with iOS.
We pondered the resemblance of our system of automation of interaction with the App Store (and even made prototypes), but we did not have the resources to finish and update. Also, there was no more or less adequate API from Apple. Well, the last nail in the coffin of our custom solution was driven by regular updates to the App Store and its mechanisms. In general, we decided to try Fastlane - then another version of 2015.
The first step was to write a mechanism for unloading translated texts for applications into the desired structure as a component of our common internal system AIDA (Automated Interactive Deploy Assistant). This system is a kind of hub, a link between all systems, technologies and components used in Badoo. It works on a self-written queuing system implemented on Golang and MySQL. Maintains and improves it mainly Department Release Engineering. We talked about it in more detail in an article back in 2013, since then a lot has changed. We promise to tell you about it again - AIDA is awesome!
In the next step, the uploaded texts were fed to Fastlane, which uploaded them to the App Store. After that, I had to go into the App Store interface, manually select the desired downloaded version and send the application for verification if Processing had already been completed by then.
This reduced the preparation time for the release from a couple of hours to about 30 minutes, of which only one and a half minutes had to be done by hand! The rest of the time is to wait. Wait for the end of Processing. The mechanism was a breakthrough at that time just because it almost completely saved us from manual work in preparing the AppStore for release. Under the script, we made a repository, which was given access to people who are directly related to releases (project managers, release engineers).
In this mode, we lived for some time. But at some point, this scheme led to the accumulation of a lot of "sacred knowledge", the owner of which and, as a result, the general picture of events became a single person, and this is not good. Especially for this person himself: you can't even go on vacation without a laptop.
In addition, there were a lot of scattered infrastructure components around this mechanism, practically unrelated to each other.
- You had to go to TeamCity for a fresh build, download the IPA file from there, upload it to the App Store through the Application Manager.
- Then go to the interface with translations in AIDA, see if all translations are ready, run the script, make sure that it worked correctly (after all, Fastlane was still damp at that time).
- App Store , Processing.
- Review.
And so with every application. Let me remind you, at that time we had eight of them.
The next step was to transfer the script to our AIDA, at the same time combining and automating all the steps until the application was sent: checking for readiness of translations, collecting data from TeamCity, notification, logging and all the other benefits of the 21st century. In parallel with this, we started uploading all the built versions to TestFlight at the build stage.
TestFlight is a third-party application that was once purchased by Apple to test a finished application by external testers in a practically production environment, that is, with push notifications and that's all.
AIDA well done, be like AIDA!
All this led to a reduction in time from half an hour to one and a half minutes for everything about everything: the IPA file managed to go through Processing before the moment when the team of QA engineers gave the go-ahead to launch the release. However, we still had to go to the App Store, select the version we want, and submit it to Review.
Plus, a simple interface was drawn: we all love click-click.
Like this, tab by tab, Ctrl + C Ctrl + V ...
Android Release Automation
Then the question arose about the automation of releases of Android applications. Although this process was much faster, it was necessary to do quite a lot by hand:
- Go to the Google Play console to make sure that the previous version is rolled out to 100% of users or frozen.
- Create a new version of the release with updated texts and screenshots (if any).
- Download the APK file (Android Package), download the Mapping file.
- Go to HockeyApp (used to log crashes at the time), upload the APK file and the Mapping file there.
- Go to the chat and unsubscribe about the release status.
And so with every application.
Yes, Google Play has its own API. But why make a wrapper, monitor changes in the protocol, maintain it and spawn entities unnecessarily if we are already using Fastlane for iOS releases? In addition, it comfortably exists on our server, is brewed in its own juice and generally updated. And by that time, he had also learned how to adequately release Android applications. The stars came together!
First of all, we cut out everything old from everywhere that was: individual scripts, automation sketches, old API wrappers - this was created once as an experiment and was not of particular value. Immediately after that, we added a team to AIDA, which already knew how to pick up what was needed from TeamCity, upload what needed where needed in HockeyApp, send notifications, log activity, and in general she was a member of the team.
Fastlane was responsible for uploading APK and Mapping files to Google Play. I must say that it is much easier to go along the beaten track: it was realized quickly enough with a minimum amount of effort.
At a certain stage in the implementation of automation, there was a transition from APK archives to AAB (Android App Bundle). Again, we were lucky that hot on the heels we quickly managed to fix everything, but “entertainment” was added in connection with this transition. For example, he spoiled HockeyApp, which did not know how to use AAB archives in connection with preparing for self-sawing. So in order to comfortably continue to use it, after assembling the AAB, it was necessary to disassemble the assembled archive, get the Mapping file from there, which will fly to the HockeyApp, and from the AAB it was necessary to separately collect the APK file and only then upload it to the same HockeyApp. Sounds fun. At the same time, Google Play itself perfectly decomposes the AAB, takes out the Mapping file from there and inserts it where needed. So we got rid of one step and added a few, but there was no getting around it.
An interface was written (again, by analogy with iOS), which was able to download a new version, check the release along and across, manage the current active release (for example, increase the rollout percentage). In this form, we gave it to the members of the Android QA team responsible for the releases, began to collect feedback, fix bugs, refine the logic (and what else happens after the 1.0 release?).
By the way, in the future, automation gave us the opportunity to upload beta versions of applications to Google Play automatically on a schedule, which, in turn, greatly accelerated the process of automatic and regression testing.
Unification of flow of mobile releases
By the time the Android releases were automated, Fastlane had finally learned how to submit versions of iOS apps for review. And we have slightly improved the version control system in AIDA.
It's time to outsource iOS releases to the QA team. To do this, we decided to draw a beautiful form that would fully cover the needs arising during the release of iOS applications: it would make it possible to select the desired build in TeamCity according to predefined parameters, select the option of loaded texts, update or not optional fields (for example, Promotional Text) ...
No sooner said than done. The mold turned out to be very nice and fully satisfies all requests. Moreover, with its implementation, it became possible to select all the necessary applications at once with all the required parameters, so that interaction with the interface was minimized. On command, AIDA sends a link to the build log, by which you can track the errors that occur, make sure everything went well, get some kind of debug information like the version of the downloaded IPA file, the release version, etc. This is how beautiful iOS releases are. and were transferred to the iOS QA team.
Well, pretty?
We liked the idea with the mold so much that we decided to do the same for Android releases. Whereas we have an app written entirely in React Native and that the QA team of that app is responsible for both iOS and Android releases.
The interface already used by the Android QA team was integrated with changes in a similar form, the process was adapted to new realities - everything was as close as possible to the processes of the iOS team. This gave an incentive to finally sketch out a more or less final version of the documentation for both platforms (in the process of constant changes, we categorically did not want to do this), and also to decouple the release process from all the artificial restrictions that had developed historically and led to unnecessary gestures in non-standard situations, requiring action team of release engineers.
Output
In such a boring way, for about five years (from the moment we began to develop the mobile direction to the present day), we have fully automated the build, testing and release processes, made them as efficient as possible and transferred responsibility for releases to the QA team members, who accept decision on the degree of application readiness.
In addition to the obvious advantages, we completely got rid of scattered scripts, all kinds of "secret knowledge" tied to a single person, integrated a new component into our ecosystem, which is supported by a small team of release engineers.
It's great that now there is the possibility of automating most of the routine actions, that engineers can write code whenever they want, and any code can be supported by third-party developers without wasting precious time digging into complex interfaces. It can be especially difficult to understand the moments like Where should I put a check mark? ”, When the clock is midnight, no one is in the office, and the hotfix needs to be filled here and now.
Five years. Why so long? Firstly, mobile releases are far from the only area of responsibility for our small team. Secondly, of course, it took time to develop the new open-source project Fastlane; our release system evolved with it.
We have come a long way in this area. Perhaps it is not the most effective, perhaps some rake could have been foreseen and bypassed. But it was as it was. When we started it, there were no similar articles of this kind - we made our own way. And if you are facing a similar task now and this article will help you with something, I will be incredibly happy. But even if you have not learned fundamentally new information, I hope that at least it was interesting to read at your leisure. And, perhaps, compare with your experience. And if you have anything to say on this topic, welcome to the comments!