Aptly. How to organize control of packages from external repositories and delegate control to product teams

Nowadays, many companies operate without the ability to directly control the composition of packages in external repositories, even if they use mirroring, proxying and caching. This leads to the fact that the execution environment is constantly changing, in particular, the composition of docker images changes more often than production requires.



There are situations when unwanted changes contained in external dependencies can get into the composition of the product being developed. This is especially true during product certification. As a result, certification delays, nighttime tests and integration testing failures, breakdown of on-premise production (a production environment located on the organization's own resources) when rolling a hotfix, and so on. In a new article, we have described an approach to avoid these problems.

What we wanted to achieve

Before proceeding with the description of the approach, a couple of words about the tasks that we wanted to solve:

• Get full control over the composition of external packages in the release (predictability).
• Fix the compositions of external repositories for quick roll-out of hotfixes with minimal additional testing (speed).
• Provide product QA stands with a repeatable, predictable, fixed environment (repeatability).
• Independence from the presence of an external communication channel (autonomy).
• Instant switching to official repositories in case of failure (fault tolerance).
• Guaranteed validation of keys for external repositories in assembly pipelines (trust).
• And most importantly, transfer management and control over the composition of external packages into the hands of product teams and release managers (self-management).

Feature Build Life Cycle Analysis

Our approach solves the problem of fixing the composition of external repositories for a specific date, for a release or a feature. The following diagram illustrates the management of the release, feature build, and hotfix lifecycle.



Let's take the Debian Stretch conditional repository as an example. This approach is applicable to Docker, SaltStack, etc. repositories. Three slices were recorded on the timeline for dates T1, T2 and T3.







We have tabulated the contents of the external Debian Stretch repository for building the Feature1, Feature2 and Feature3 distributions. From the table you can see that the composition of the external repository is controlled by each branch independently. We have made an agreement for ourselves to commit the master branch for Debian Stretch on a daily basis and tag each slice in YYYYMMDD format, for example 2020304 for the March 4, 2020 slice. The table summarizes the snapshots of the external repository used for the distribution in each branch at three different points in time and the composition in the wizard for Debian Stretch. The team for each feature or for each release updates the composition of external repositories at its discretion and according to its development cycle.



Using Feature1 as an example: the product team starts developing a new feature and fixes the composition of the external repository in the configuration files as of the date 20200228 (see the diagram above).



Switch to 20200228

deb http://repository.co/debian-stretch-20200228 stretch main contrib non-free



During development, due to the appearance of new packages, it becomes necessary to update the package database to the date 20200304. Switch the working repository to the desired date.



Switch to 20200304

deb http://repository.co/debian-stretch-20200304 stretch main contrib non-free



Next, another switch of the package base takes place on the date 20200501.



Switch to 20200501

deb http://repository.co/debian-stretch-20200501 stretch main contrib non-free



If we now draw time slices, we will see that at times T1 and T2 Feature1 development is on the package base, "frozen" on March 4, 2020. And in the T3 cut, development is already underway on a new package base for May 1, 2020.

Product multi-release dependency management

Now let's look at dependency management for several active product releases. There are three releases on support, 2.5, 2.6 and 2.7.



In the table, we see the correspondence of releases and dates for which the repository snapshot was made. It shows which snapshot of the composition of the external repository was used to build a specific version of the distribution.







Instead of naming slices by YYYYMMDD dates, we also use tag naming in the format <ProjectName.ReleaseVersion> (release_name.release_version of the product, for example name.2.2) or <ProjectName-FeatureNumber> (add a feature number, for example 3).



Snapshot for 2.5 as of 20200301

deb http://repository.co/debian-stretch-projectname-2.5 stretch main contrib non-free # 20200301



Thus, during the development of release 2.5, the team fixes the composition of the dependent repositories as of 20200301. Somewhere in April, the team starts a new 2.6 release and decides to use the composition of the external repository packages from 2.5. Create a new 2.6 snapshot from the 2.5 snapshot. In the future, the repositories for 2.5 and 2.6 releases could easily diverge. We have made our own tag debian-stretch-projectname-2.6 for 2.6.



Snapshot for 2.6 as of 20200301

deb http://repository.co/debian-stretch-projectname-2.6 stretch main contrib non-free # 20200301



In the case of release 2.7, the team can start development from the master branch - a daily snapshot of the original repository.



Snapshot for 2.7 as of 20200404

deb http://repository.co/debian-stretch-projectname-2.7 stretch main contrib non-free # 20200404

Multi-product dependency management

Let's look at multi-product dependency management using the example of two products with different release cycles and their own product teams: Stealth and Infiniti.







Let's comment on the table what happens and when.



1. Let the development of version 2.2 of the Stealth project start on March 1, 2020, for this a snapshot of the composition of the package database for the current date was created. The release of release 2.2.124 is made with the package base of the external repository from 20200301.



Stealth 2.2

deb http://repository.co/debian-stretch-stealth-2.2 stretch main contrib non-free # 20200301



2. On the fifth day, the package base is updated. The debian-stretch-stealth-2.2 working repository instantly switches to the desired date, the release of releases 2.2.131 and 2.2.162 is made with the external repository packages from 20200305. Without additional manipulations in the environment, all 100500 microservices of the product immediately received a new environment in the assembly pipeline 20200305 ...



Stealth 2.2

deb http://repository.co/debian-stretch-stealth-2.2 stretch main contrib non-free # 20200305



3. Parallel to the third day, the development of the Infiniti version 4.0 project starts and a slice of the repository for the date 20200303 is created for it. Versions 4.0.235 and 4.0.241 are released with the external repository packages for 20200303.



Infiniti 4.0

deb http://repository.co/debian-stretch-infiniti-4.0 stretch main contrib non-free # 20200303



4. After the release of version 4.0.241, the team decides to update the composition of the repository to 20200308 and release a new release with a new composition of external packages. Version 4.0.250 is released with package bundles for 20200308.



Infiniti 4.0

deb http://repository.co/debian-stretch-infiniti-4.0 stretch main contrib non-free # 20200308



Two options for switching between the states of the repositories allow you to choose an approach that is convenient for the development process. In the first case, we switch to the desired state by specifying a snapshot of the repositories on a specific date. In the second case, for multicomponent products, we use a named slice and move it to the desired date. This mechanism provides a one-time cutoff switching in all 100500 product components.



We manage the slices of each external repository in a separate Docker container, so at any time we can switch a specific repository to download from the external network in the event of some accidents.



Download a list of all repositories

# For example
curl repository.co/info/sources.list | grep $(lsb_release -cs) > /etc/apt/sources.list

Automatic creation of slices of external repositories

The repositories are updated every night according to the GitLab scheduler. When a new repository is added, the changes are automatically applied to the server.







At the time of committing a new slice of the external repository, its certificate is checked, if it differs from the one saved with us, then the update does not take place, and we receive an error message.

Outcome

1. Preparing a new version of a distribution kit for certification is no longer a headache. For the certification period, we fix the composition of the distribution, and if you need to fix something promptly, then with a high probability there will be no errors in the released hotfix due to changes in the environment.
2. All feature builds get the managed state of external repositories.
3. Hotfix roll-out and QA verification are accelerated with a predictable, quick and successful result.
4. Feature- , .
5. .

Note that Debian has an official snapshot.debian.org resource with daily snapshots and large storage depths. This is sufficient for certain tasks.



Thanks to Sergey Smirnov and the community for an excellent tool for managing the composition of Aptly's external repositories. From us - a small contribution to the best practices of using this useful tool in production conveyors.



In the next articles, we will talk about the Aptly + Simple-CDD bundle for preparing ISO images of distributions, delegating the management of external dependencies to product teams, and the problems of using Aptly in the process of managing external dependencies.



Authors : Nikita Drachev , Alexander Pazdnikov , Timur Gilmullin