Beginner's Guide: Building a DevOps Pipeline

If you're new to DevOps, take a look at this tutorial on how to create your first five-step pipeline.







DevOps has become the standard solution for fixing slow, disconnected, or broken software development processes. The problem is that if you are new to DevOps and don't know where to start, then you may lack understanding of these techniques. This article will walk you through the definition of a DevOps pipeline and will provide instructions on how to create one in five steps. While this tutorial is not comprehensive, it should give you a foundation to get you started and expand your knowledge in the future. But let's start with history.

My DevOps Journey

I used to work on the Citi Group cloud team developing an Infrastructure-as-a-Service (IaaS) web application to manage Citi's cloud infrastructure, but I've always wondered how to make the development process more efficient and bring positive cultural change to the development team. I found the answer in a book recommended by Greg Lavender, Citi's CTO for Cloud Architecture and Infrastructure. The book was titled The Phoenix Project , and it explains DevOps principles while reading like a novel.



The table on the back of the book shows how often different companies deploy their systems in a release environment:



Amazon: 23,000 per day

Google: 5,500 per day

Netflix: 500 per day

Facebook: Once a day

Twitter: 3 times a week

Typical company: Once every 9 months



How are Amazon, Google and Netflix frequencies even possible? This is because these companies figured out how to create a near-perfect DevOps pipeline.



We were a long way from that until we implemented DevOps at Citi. My team had different environments back then, but deploying to the development server was completely manual. All developers had access to only one development server based on IBM WebSphere Application Server Community Edition. The problem was that the server would shut down whenever multiple users tried to deploy at the same time, so the developers had to communicate their intentions to each other, which was quite painful. In addition, there were issues with low-level test code coverage, cumbersome manual deployment processes, and inability to track the deployment of code associated with a specific task or user story.



I realized that I needed to do something and found a like-minded colleague. We decided to collaborate on the initial DevOps pipeline - it set up a virtual machine and a Tomcat app server while I was working on Jenkins, integrated Atlassian Jira and BitBucket, and worked on test code coverage. This side project was very successful: we almost completely automated many processes, achieved almost 100% uptime of our development server, provided tracking and improved test code coverage, and also added the ability to link branches in Git with issues in Jira or deployments. Most of the tools we used to build our DevOps pipeline were open source.



Now I understand how simple our DevOps pipeline was: we didn't use extensions like Jenkins files or Ansible. However, this simple pipeline worked well, possibly due to the Pareto principle (also known as the 80/20 rule).

A quick introduction to DevOps and the CI / CD pipeline

If you ask several people, “What is DevOps?”, Then you will probably get several different answers. DevOps, like Agile, has evolved to encompass many different disciplines, but most people will agree on a few things: DevOps is a software development practice or software development lifecycle (SDLC) that focuses on changing the culture in which developers and non-developers exist in an environment in which:



Automated operations that were previously performed manually;

Everyone does what he knows best;

The number of implementations for a certain period of time increases; The throughput increases;

Development flexibility is increased.



While having the right software tools isn't the only thing you need to create a DevOps environment, some tools are needed. The key tool is continuous integration and continuous deployment (CI / CD). In this pipeline, environments have different stages (for example, DEV, INT, TST, QA, UAT, STG, PROD), many operations are automated, and developers can write high-quality code, achieve development flexibility, and high deployment frequency.



This article walks you through a five-step approach to building a DevOps pipeline similar to the following diagram using open source tools.

Step 1: CI / CD methods

The first thing you need is a CI / CD tool. Jenkins, an open source Java-based tool released under the MIT license, is the tool that popularized the DevOps direction and became the de facto standard.



So what is Jenkins? Think of it as some kind of magic universal remote control that can talk and organize various services and tools. By itself, a CI / CD tool like Jenkins is useless, but it gets more powerful as it connects to different tools and services.



Jenkins is just one of many open source CI / CD tools that you can use to build your DevOps pipeline.



Jenkins: Creative Commons and MIT

Travis CI: MIT

CruiseControl: BSD

Buildbot: GPL

Apache Gump: Apache 2.0

Cabie: GNU



This is how DevOps processes look like with a CI / CD tool:







You have a CI / CD tool running on your localhost, but there isn't much you can do at this point. Let's move on to the next leg of the DevOps journey.

Step 2: Manage source control systems

The best (and arguably easiest) way to test that your CI / CD tool can do the magic is to integrate with a Source Code Control Tool (SCM). Why do you need source control? Let's say you are developing an application. Whenever you create an application, you are programming, it doesn't matter if you use Java, Python, C ++, Go, Ruby, JavaScript, or any of the gazillions of programming languages. The code you write is called source code. In the beginning, especially when you are working alone, you can probably put everything in a local directory. But as the project gets bigger and you invite other people to collaborate, you need a way to prevent conflicts while sharing modifications efficiently. You also need a way to restore previous versions,because creating backups and copying / pasting into them is already obsolete. You (and your teammates) need something better.



This is where source control becomes practically a necessity. This tool stores your code in repositories, keeps track of versions and coordinates the work of project participants.



While there are many source control tools out there, Git is the standard and rightly so. I highly recommend using Git, although there are other open source options if you like.



Git: GPLv2 and LGPL v2.1

Subversion: Apache 2.0

Concurrent Versions System (CVS): GNU

Vesta: LGPL

Mercurial: GNU GPL v2 +



This is what a DevOps pipeline looks like with added source controls.







The CI / CD tool can automate the processes of verification, source code retrieval and collaboration between members. Not bad? But how do you make it a working app so billions of people can use and appreciate it?

Step 3: Create a build automation tool

Fine! You can check the code and make changes to the source control system, as well as invite your friends to collaborate on development. But you haven't created the app yet. To make a web application, you need to compile and package it in a deployable batch format or run it as an executable file. (Note that an interpreted programming language like JavaScript or PHP does not need to be compiled.)



Use a build automation tool. Regardless of which build automation tool you choose to use, they all have the same goal: build source code in whatever format you want and automate the task of cleaning, compiling, testing, and deploying to a specific environment. Build tools will vary depending on your programming language, but here are some common open source options.







Great! You can put the build automation tool configuration files into a source control system and let your CI / CD tool pull everything together.







It's okay, isn't it? But where to deploy your application?

Step 4: Server for web applications

So far, you have a compressed file that can be executable or installable. For any application to be really useful, it must provide some kind of service or interface, but you need a container to host your application.



The web application server is just such a container. The server provides an environment in which the logic of the deployed package can be defined. The server also provides an interface and offers web services by opening sockets to the outside world. You need an HTTP server as well as some environment (like a virtual machine) to install it. For now, let's assume you learn more about this (although I'll talk about containers below).



There are several open source web application servers.







Your DevOps pipeline is almost ready to use. Good job!







While you can stop there and integrate yourself, code quality is an important thing for an application developer to worry about.

Step 5: code testing coverage

Implementing tests can be another cumbersome requirement, but developers must catch any bugs in the application early and improve the quality of the code to ensure that end users are satisfied. Fortunately, there are many open source tools out there to test your code and make recommendations for improving its quality. Better yet, most of the CI / CD tools can connect to these tools and automate the process.



Code testing consists of two parts: code testing frameworks that help you write and run tests, and proposal tools that help improve the quality of your code.



Code testing systems







Code improvement recommendation systems







Note that most of the tools and frameworks mentioned above are written for Java, Python, and JavaScript, since C ++ and C # are proprietary programming languages ​​(although GCC is open source).



Now that you've implemented test coverage tools, your DevOps pipeline should look similar to the diagram shown at the beginning of this tutorial.

Additional steps

Containers



As I said, you can host your server in a virtual machine or server, but containers are a popular solution.



What are containers? The short explanation is that a virtual machine needs a huge amount of operating system memory that exceeds the size of the application, while the container only needs a few libraries and configurations to run the application. Obviously, there are still important uses for a virtual machine, but a container is a lightweight solution for hosting an application, including an application server.



While other container options exist, the most popular are Docker and Kubernetes.



Docker: Apache 2.0

Kubernetes: Apache 2.0 Automation middleware







Our DevOps pipeline is mainly focused on co-creation and deployment of applications, but there are many other things that can be done using DevOps tools. One of them is the use of Infrastructure as Code (IaC) tools, which are also known as middleware tools. These tools help automate the installation, management, and other tasks for middleware. For example, an automation tool can retrieve applications like the web application server, database, and monitoring tool with the correct configurations and deploy them to the application server.



Here are some open source middleware tools:



Ansible: GNU Public

SaltStack: Apache 2.0

Chef: Apache 2.0

Puppet: Apache or GPL







Learn the details of how to get a sought-after profession from scratch or Level Up in skills and salary by completing SkillFactory paid online courses:

• DevOps course (12 months)

Useful

• 450 free courses from the Ivy League
• Trends in Data Scene 2020
• Data Science is dead. Long live Business Science
• Data Science on Steroids: Introducing Decision Intelligence
• Most successful and most controversial Data Science project: Cambridge Analytica