The article was prepared by Konstantin Bryukhanov, head of the "CI / CD" course . In it, Konstantin revealed a number of problematic points related to the delivery of deployment of software product code in IT companies, and collected recommendations from among the best international practices.
In IT operation, the most demanded direction is precisely the adjustment and provision of continuous delivery and deployment. Technologies and methodologies are constantly evolving, tools are being improved. As such, the latest delivery and deployment requirements include the readiness and continuity of change testing, preparing and setting up a test environment for the QA team.
Technological breakthroughs and free software have led to a significant change in the approach to organizing CI / CD processes. The transition to new principles greatly influenced the corporate culture, the in-demand skills of employees and the very principles of working in organizations, which led to large-scale changes in the world of software development.
Cloud solutions are becoming more and more priority. Continuous software delivery requires effective collaboration between development, test, and operations teams, and the cloud is great for this collaboration.
However, the deployment phase, performed in a complex distributed topology, is error-prone and typically requires manual troubleshooting. The product deployment phase of a continuous delivery process often creates bottlenecks and negatively impacts the effectiveness of the DevOps process.
Continuous delivery allows you to achieve automation of testing incremental software changes and rapid deployment of updates in the most efficient and safe ways. This approach gives users the confidence that the latest version of the code is being used in the production environment, and changes made by programmers can reach customers in a matter of hours or even minutes.
Let's consider the most common scenario for implementing CI / CD in a project:
- The development team releases a new version of the product (new functionality or bug fixes from the previous release).
- The Continuous Integration (CI) service validates new code with a series of tests that include multiple levels of testing, such as syntax, unit, and regression tests.
- , (CD).
- (, ) , (staging), , .
- stage- CI/CD , .
- .
This scenario is the most general and covers most of the needs of the development and operations teams, but still has some problems , for example:
- Replacing files . Often it is necessary to update or replace configuration files or regenerate some static content. In this case, users may receive errors until the traffic is switched from the old software version to the new one. In the case where the deployment fails, there is a risk of file incompatibility.
- . , , . . , - , - .
- . . , , .. . , - , , .. .
It is worth noting that the problems listed above can arise even in a near-ideal environment, but one of the main difficulties in implementing the DevOps methodology is that there is no single picture of what the continuous delivery and product deployment process should look like. Many IT companies know too little about DevOps, sometimes they do not understand this methodology at all, while others already have historical solutions, on top of which new processes have to be built. Taking into account the high requirements for the qualifications of Devops specialists, and their shortage in the labor market, the employer is often forced to use the resources already at his disposal and give Devops tasks to novice engineers. As a result, there are even more weak points in the system.
When using CI / CD without a correct understanding of the methodology, without an analytical approach to building infrastructure and methods for delivering code, the following problems arise:
1. Human factor . The first and most significant risk is associated with human factors. Let's imagine a situation when it is necessary to configure several more servers similar to the existing ones. If the specialist who made the previous installations or settings is currently unavailable for any reason (sick, quit, etc.) and has not prepared detailed instructions, then the situation becomes much more complicated. In this case, each new specialist must study the entire process of setting up the server completely, while he has no margin for error. And besides, it is impossible to accurately estimate how long it will take to set up and guarantee its success.
This also includes the risks associated with the fact that the author of the method made a mistake, forgot to cover the processes with tests, or simply did not take into account something, and his successor did not notice it.
It should also be remembered that companies often develop several projects, and the IT Operations Department is usually one, and one Operations Engineer serves several projects. If there is no single scheme and concept, then the processes in different teams will be built in different ways, which will significantly complicate the subsequent development of Devops in the company and make a high threshold for the entry of an operations engineer into another project, where processes are already used that are different from those with which he worked earlier.
2. Non-idempotent scenarios... Idempotency is a critical attribute of continuous delivery and code deployment scenarios, especially in infrastructure deployments. The engineer must be confident that each time the scripts are executed, the result will be uniquely guaranteed, expected, and unchanged when replaying the same script. Often, when implementing Devops in a company, engineers are trying to develop a business solution and may not take into account idempotency or simply not know about this requirement. In this case, the company receives a time bomb. there is a possibility of delivering unexpected code to production facilities. For example, if someone updated a CMS module for one project, and thereby influenced others, where this is not expected.
3. Storage of sensitive data and organization of access. One of the most important points in the Devops approach to storing secret data, restricting rights, organizing network and user access. Until now, there are no uniform accepted practices and tools to solve this problem, and engineers have to conduct research each time, depending on the current organization of the infrastructure and the adopted methods of restricting access. For this reason, the implementation of the Devops methodology in an enterprise is complicated by the fact that it is impossible to unequivocally find a solution for your particular case, and the use of other people's practices does not always guarantee security.
4. Defined budgeting model , more suitable for Waterfall methodology.
5. High safety requirements.Consequently, it is impossible to place the infrastructure of national IT projects in the area of responsibility of commercial, foreign companies, for example, Amazon, Microsoft.
6. Large amount of "legacy code", "legacy infrastructure" that needs to be maintained. The need for integration with a large number of legacy systems.
Thus, the process of building Devops in an enterprise can be accompanied by a number of problems and not always solve the tasks for which it was created.
The first important step is to abandon the relationship to servers as a difficult to customize, unique element of infrastructure., transition from manual server configuration to automated, centralized infrastructure management. The process of setting up each server should be described in the form of a configuration that is easy to read, changeable, and ready for multiple safe reuse, providing an unambiguous guaranteed result. Examples of industrial orchestrator systems are Chef or Ansible. These systems allow you to manage a large number of servers with minimal costs.
The next important step is to apply automated testingto cover as much as possible the functionality of the code being developed (both software and infrastructure). In other words, having a deployed infrastructure, but without automated testing, the bottleneck of the development process will be timely verification of functionality. Automating the testing process should begin with the actual writing of the software code (unit testing), the application of primary tests on the server that is responsible for building the software, and a test of the server configuration. This will reduce the workload on the software quality assurance team and significantly reduce the time for software to pass through the pipeline.
The final logical step is the centralized collection and analysis of log files from all serversfor timely notification of all stakeholders and proactive monitoring of the state of the infrastructure as a whole.
The above guidelines will help you build a resilient, scalable infrastructure that can handle an intensive development process. DevOps implementation requires the involvement of everyone in the process, from test and development to managers and operations. At each stage, a constant retrospective analysis of the process is required, because as a result of random errors when changing the configuration, the systems completely stop working. Telemetry needs to be improved to better detect errors and recover, and to protect the deployment pipeline and meet management change goals. This will allow you to receive maximum support from management in implementing DevOps initiatives, create a livelier and friendlier work environment,so that any participant can learn throughout the time - this will not only help each performer to achieve goals, but also lead the organization to success.
In 3 months on our online course "CI / CD" you will develop an understanding of the architecture of cloud providers, learn to automate code analysis and search for vulnerabilities, and learn how to customize the processes of building, testing and installing an application from the three largest providers . The program is designed for specialists with experience in administration - a special entrance test will help you to find out whether you have enough preparation for training.