For future students of the "Infrastructure as a code in Ansible" course and all those interested, we prepared a translation of useful material.
We also invite you to sign up for an open lesson on the topic "Managing Kubernetes with Kubespray" .
Welcome back! This is another technical article in a series of terraform and kubernetes articles on infrastructure as code from Contino.
TL; DR
Command size does not matter. In any case, implementing a good terraform infrastructure configuration analysis and end-to-end sanity testing does not have to be a lengthy and complex process.
I was faced with an exciting challenge: exploring, developing, and presenting a suitable open source testing framework for the terraform codebase as part of the infrastructure release pipeline. The principle of “quality control in everything” is far from new. But its implementation depends on the maturity of the organization's infrastructure and the acceptable level of risk - until the stage of reaching a [in any way] finished product.
Taking a fresh look at the field of testing infrastructure code allowed me to become familiar with the latest testing frameworks and update my knowledge.
Admittedly, at the beginning of my journey, I suffered from the prejudice that preparing and implementing such “enterprise-class quality control” can be labor intensive.
After all, Hashicorp Terraform has enough functionality out of the box to validate and validate your codebase .
- Code quality control -
terraform fmt -check
andterraform validate
. - Preview -
terraform plan
. - Construction -
TFLOG=debug terraform apply
for meticulous verification.
Terraform static code analysis tools
A Google comb has revealed a vast array of potentially useful terraform testing tools.
But first, let's go through the list of our requirements.
- Having unit tests in your terraform resource configuration and the ability to extend any such generic list of best practices * checks for a specific cloud provider. In addition, we are interested in easy-to-use tools that you can get started with right away.
* Lack of ec2 instances exposed to 0.0.0.0/0 world - and so on.
- — «» , . , , , EKS.
- , , — , , . , , .
- , . , , Go Python. , , , , . , , . , .
Spotlight: Analyzing and Comparing Terraform Testing Tools and Platforms
Hopefully the following list will make your work on static infrastructure-as-code analysis and quality control easier. Note, this is a complete list of all relevant terraform testing tools I have found, which is a mixture of configuration validity testing, code quality control, and secOps-focused best practices with unit tests. This list is presented below.
- tflint - github.com/terraform-linters
- terrafirma - github.com/wayfair/terrafirma
- tfsec - github.com/liamg/tfsec
- terrascan — github.com/cesar-rodriguez/terrascan ( TF 0.13)
- checkov — github.com/bridgecrewio/checkov
- conftest — github.com/instrumenta/conftest
Let's summarize. I tried to find a standardized unit test for terraform resource components and a more customizable set of tests that take resource configuration to validate against results
terraform plan
.
After reviewing the advantages and disadvantages of each platform, I opted for a tool
checkov
and platform with a very apt name
terraform-compliance
- both written in python. They met all my requirements described above.
The infrastructure-as-code release pipeline looks like this in general terms.
Having thoroughly delved into these platforms, I inevitably revised my own experience and came to the following relevant conclusions on the topic under discussion:
- .
- , , , .
- , , , - « » « ».
— Checkov BridgeCrew
www.checkov.io
Checkov is a static code analysis tool for infrastructure as code.
It scans cloud infrastructure provisioned with Terraform, Cloudformation, Kubernetes, Serverless, or ARM templates and identifies misconfigurations for security and compliance.
There are several default unit tests run when scanning the terraform code repository that show deviations from best practices - for example, when, according to your security configuration, you have a virtual machine on port 22 exposed to the world (0.0.0.0/0).
All tests can be found at this GitHub link.
It's very easy to get started with the platform.
- Install the binary.
- Initialize the terraform directory with terraform init.
- Run chechov on this directory.
All unit tests that run by default can be listed on the command line. In addition, when checkov runs, the platform will by default return all passed and failed unit tests. Very convenient, easy to start using. Terraform advanced methods are tested, but not all. This is a fundamental difference.
Chechov will happily evaluate ONLY your code
terraform
. The platform can work immediately after
terraform init
. She doesn't care about yours
terraform plan
- with all the advantages and disadvantages. The platform does what is stated, namely "static code analysis". Be aware of the potential implications, as well as any logic considerations, for your resources.
checkov .
If you're ready to do deep Python development, you can write additional unit tests. The platform development language was one of my requirements, because sometimes I have to analyze the code base of the tests to estimate how difficult it would be [if necessary] to create such additional methods. This moment, coupled with the service issues for the group as a whole, became the main factor in choosing this platform over an alternative one that allows you to get the same result.
To summarize, the checkov platform is great in the area of static code analysis. Specifically if I need to whitelist the originally defined IP subnet. But this option is not suitable for e2e tests that need a separate testing platform.
On the other hand, as a solution, I can replicate the unit test and hard code my subnet / IP settings. And then what if I have multiple instances and projects - skip this test, even if I need it? May be. Or maybe not.
This is where the second testing platform comes into play -
terraform-compliance
.
Terraform-compliance
terraform-compliance.com
Terraform-compliance is a lightweight testing platform designed for security and compliance audits in terraform to ensure your infrastructure is as negatively tested as code.
Background
Again, behavior testing end-to-end development (BDD) has recently come into use as a testing framework, highlighting the need for a universal testing framework. But this is not the only benefit. Simplicity.
In fact, it seems to me that BDD is not getting enough attention. You may have heard of Test Driven Development (TDD), which takes deep roots, primarily in the software development environment. But this is where platforms like BDD make it easier to create additional logic, offering the average infrastructure maintainer a simpler, concise, and repeatable way to develop end-to-end custom tests without in-depth knowledge of any specialized and new programming language.
And while code can describe, in fact, everything in the world, ultimately everything comes down to manageability, the ability to comprehend the complexity of the code (which may require the preparation of extensive documentation), not to mention support and maintenance. Read more about BDD here.
Cucumber.io is not just a language, it is a system that simplifies testing work by taking a WYSIWYG approach to test design, understanding, and maintenance. These examples are determined prior to development and are used as acceptance criteria.
They are part of the definition.
Testing with Terraform-Compliance
Each platform is reviewed for its merits, with an in-depth study of where its features and nuances can be best used. Looking ahead, I can say that both platforms can be used.
Here is an example of such a test developed using the
terraform-compliance
BDD platform . It allows you to perform fairly complex end-to-end testing.
The platform
terraform-compliance
uses the output
terraform plan
... As a result, it allows complete release “plans” to be formed and thoroughly tested. For example, control that the correct encryption key pair [for your cloud provider] is used for account, environment, and so on. You have a lot of creative freedom, and most importantly, the platform is very easy to use.
Just check out the steps and examples below.
- Step 1. Initialize the terraform directory: # terraform init
- Step 2. You can quickly generate a terraform plan with the following command: #terraform plan -out = plan.out
- Step 3. Write some tests. It's a simple matter - there is already a folder with examples. Let's walk through my own test examples below, written based on my terraform plan output.
This is a snippet of
terraform
the terraform configuration plan that creates the EKS with the specified launch group. Let's make sure our code
terraform
doesn't use the infrastructure
instancetype
, but uses the "approved"
a1.xlarge
or
a1.2xlarge
.
Now I will intentionally change it to
t2.small
to simulate test failures.
Let's write a test to ensure that this requirement is successfully validated.
- Step 4. Let's get the
terraform-compliance
boards to be evaluated using test scenarios:#terraform-compliance -p plan.out -f ./<test-cases-folder>
Running tests
Example Pass and Fail Result
If our Terraform framework code is using correct
instancetype
, then all results will be green SUCCESS.
If our Terraform infrastructure code violates the requirement because it is incorrect
instancetype
, then the results will be red FAIL.
Let's write even more tests.
Some more simple tests taken from the examples directory:
If one fails, the user will see "actual_value", which is retrieved and displayed for help and debugging purposes.
Test results
After all tests have been run, a convenient summary of all passed and failed tests is displayed, which also includes the missed tests. I like it because it allows me to write a long list of rigorous tests, as well as to provide clear information at the end of which tests failed and when. In addition, if it fails, some tests can be skipped with a tag
@warning
, as shown in the example below.
habrastorage.org/getpro/habr/upload_files/c22/910/cb9/c22910cb95fb4ccc7555d44bd8b5436b
Outcome
Without a doubt, this was a great opportunity to take a fresh look at some of the superior validation and testing frameworks available for code as Terraform frameworks.
I enjoyed looking at both of these platforms and was particularly surprised by the ease of integration of checkov, as well as the amazing validation
e2e terraform plan
and custom testing options it offers
terraform-compliance
.
The latter reminds me of the behavior of behave, another great BDD e2e kubernetes testing framework that I have worked with in the past.
Test frameworks written entirely in Python make it easier to share Python knowledge across platforms and reduce the amount of brainpower needed to maintain and develop tests in the future.
If you need to check your configuration against best practices when you don't need a terraform plan, then checkov might be the one for you. Otherwise, the answer might be a platform
terraform-compliance
that has a richer set of validation features
terraform plan
. Best of all, being a BDD platform is
terraform-compliance
very easy to learn.
Unit testing comes first. As easy as pie. Bridgecrewio's Checkov platform enables best practices compliance checking out of the box.
There is really no good reason to skip any of these QC tests, no matter how large your group is. Especially considering the insignificant labor costs that need to be applied to implement them (see examples in the article).
PS Contino has a fair amount of fantastic projects. If you would like to work on ultra-modern infrastructure projects or are looking for serious tasks - contact us! We are hiring staff and looking for bright minds at all levels. At Contino, we pride ourselves on developing cutting edge cloud transformation projects for midsize companies and large enterprises alike.
«Infrastructure as a code in Ansible».
« Kubernetes Kubespray».
