The purpose of the article is to show one of the possible approaches for organizing flexible deployment of dev / test benches. Show what advantages the IaC approach provides us in combination with modern tools.
Background
There are several stands for developers - devs, tests, production. New versions of product components appear several times a day.
As a result, the existing stands are occupied, the developers are idle waiting for the release of one of the stands.
The creation of additional static stands will solve the problem, but will lead to an overabundance of them during a decrease in developer activity and, as a result, will increase the company's infrastructure costs.
A task
To enable developers to deploy and remove stands on their own, based on current needs.
Stack
Gitlab CI, Terraform, Bash, any private / public cloud.
Technical difficulties:
Terraform state file - out of the box we do not have the ability to use a variable in the value of the state file name. You need to invent something or use another product.
Subnets - each new environment must be created on an isolated subnet. You need to control free / busy subnets, a kind of DHCP analog, but for subnets.
Algorithm
Gitlab CI runs pipeline. Ties all other components together.
Terraform .
Configuration manager(CM) - .
Bash .
development-infrastructure/ deploy/ env1/ main.tf backend.tf ansible-vars.json subnets.txt env2/ ... cm/ ... modules/ azure/ main.tf variables.tf scripts/ env.sh subnets.txt .gitlab-ci.yml
deploy - - terraform CM, .
cm - , Ansible .
modules - terraform
scripts - bash
.gitlab-ci.yml:
stages:
- create environment
- terraform apply
- cm
- destroy environment
.template:
variables:
ENV: $NAME_ENV
when: manual
tags: [cloudRunner01]
only:
refs:
- triggers
Create environment:
stage: create environment
extends: .template
script:
- ./scripts/create_env.sh -e $ENV -a create
artifacts:
paths:
- deploy/${ENV}/backend.tf
- deploy/${ENV}/main.tf
- deploy/${ENV}/vars.json
Create instances:
stage: terraform apply
extends: .template
script:
- cd ./deploy/$ENV
- terraform init -input=false
- terraform validate
- terraform plan -input=false -out=tf_plan_$ENV
- terraform apply -input=false tf_plan_$ENV
Deploy applications:
stage: cm
extends: .template
script:
- # CM
- # , $ENV ,
- # ..
- # terraform
Destroy instances and environment:
stage: destroy environment
extends: .template
script:
- cd ./deploy/$ENV
- terraform init -input=false
- terraform destroy -auto-approve
- ./scripts/delete_env.sh -e $ENV -a delete
:
Create environment - , NAME_ENV, , git .
Create instances - ( ) , .
Deploy applications - Configuration Manager.
Destroy instances and environment - bash , . scripts/subnets.txt.
NAME_ENV, :
Git pipeline.
modules/base/main.tf:
#
provider "azurerm" {
version = "=1.39.0"
}
…
# , Azure. ,
resource "azurerm_resource_group" "product_group" {
name = "${var.env_name}"
location = "East Europe"
}
#
resource "azurerm_virtual_network" "vnet" {
name = "product-vnet"
resource_group_name = azurerm_resource_group.product_group.name
location = azurerm_resource_group.product_group.location
address_space = [var.vnet_address]
}
# bash
resource "azurerm_subnet" "subnet" {
name = "product-subnet"
resource_group_name = azurerm_resource_group.product_group.name
virtual_network_name = azurerm_virtual_network.vnet.name
address_prefix = var.subnet_address
}
#
resource "azurerm_virtual_machine" "product_vm" {
name = "main-instance"
location = azurerm_resource_group.product_group.location
resource_group_name = azurerm_resource_group.product_group.name
network_interface_ids = [azurerm_network_interface.main_nic.id]
…
}
# ...
, , , .
, , , .
scripts/env.sh:
#!/bin/bash
set -eu
CIDR="24"
DEPLOY_DIR="./deploy"
SCRIPT_DIR=$(dirname "$0")
usage() {
echo "Usage: $0 -e [ENV_NAME] -a [create/delete]"
echo " -e: Environment name"
echo " -a: Create or delete"
echo " -h: Help message"
echo "Examples:"
echo " $0 -e dev-stand-1 -a create"
echo " $0 -e issue-1533 -a delete"
}
while getopts 'he:a:' opt; do
case "${opt}" in
e) ENV_NAME=$OPTARG ;;
a) ACTION=$OPTARG ;;
h) usage; exit 0 ;;
*) echo "Unknown parameter"; usage; exit 1;;
esac
done
if [ -z "${ENV_NAME:-}" ] && [ -z "${ACTION:-}" ]; then
usage
exit 1
fi
#
ENV_NAME="${ENV_NAME,,}"
git_push() {
git add ../"${ENV_NAME}"
case ${1:-} in
create)
git commit -am "${ENV_NAME} environment was created"
git push origin HEAD:"$CI_COMMIT_REF_NAME" -o ci.skip
echo "Environment ${ENV_NAME} was created.";;
delete)
git commit -am "${ENV_NAME} environment was deleted"
git push origin HEAD:"$CI_COMMIT_REF_NAME" -o ci.skip
echo "Environment ${ENV_NAME} was deleted.";;
esac
}
create_env() {
#
if [ -d "${DEPLOY_DIR}/${ENV_NAME}" ]; then
echo "Environment ${ENV_NAME} exists..."
exit 0
else
mkdir -p ${DEPLOY_DIR}/"${ENV_NAME}"
fi
#
NET=$(sed -e 'a$!d' "${SCRIPT_DIR}"/subnets.txt)
sed -i /"$NET"/d "${SCRIPT_DIR}"/subnets.txt
echo "$NET" > ${DEPLOY_DIR}/"${ENV_NAME}"/subnets.txt
if [ -n "$NET" ] && [ "$NET" != "" ]; then
echo "Subnet: $NET"
SUBNET="${NET}/${CIDR}"
else
echo "There are no free subnets..."
rm -r "./${DEPLOY_DIR}/${ENV_NAME}"
exit 1
fi
pushd "${DEPLOY_DIR}/${ENV_NAME}" || exit 1
# main.tf terraform
cat > main.tf << END
module "base" {
source = "../../modules/azure"
env_name = "${ENV_NAME}"
vnet_address = "${SUBNET}"
subnet_address = "${SUBNET}"
}
END
# C backend.tf terraform , state
cat > backend.tf << END
terraform {
backend "azurerm" {
storage_account_name = "terraform-user"
container_name = "environments"
key = "${ENV_NAME}.tfstate"
}
}
END
}
delete_env() {
#
if [ -d "${DEPLOY_DIR}/${ENV_NAME}" ]; then
NET=$(sed -e '$!d' ./${DEPLOY_DIR}/"${ENV_NAME}"/subnets.txt)
echo "Release subnet: ${NET}"
echo "$NET" >> ./"${SCRIPT_DIR}"/subnets.txt
pushd ./${DEPLOY_DIR}/"${ENV_NAME}" || exit 1
popd || exit 1
rm -r ./${DEPLOY_DIR}/"${ENV_NAME}"
else
echo "Environment ${ENV_NAME} does not exist..."
exit 1
fi
}
case "${ACTION}" in
create)
create_env
git_push "${ACTION}"
;;
delete)
delete_env
git_push "${ACTION}"
;;
*)
usage; exit 1;;
esac
env.sh
- (\).
:
DEPLOY_DIR
.
scripts/subnets.txt .
Terraform.
git .
-
scripts/subnets.txt
scripts/subnets.txt:
172.28.50.0
172.28.51.0
172.28.52.0
...
. CIDR scripts/create_env.sh
We got the foundation that allows us to deploy a new stand by running pipline in Gitlab CI.
Reduced our company's infrastructure costs.
Developers are not idle and can create and delete stands when they need it.
We also got the opportunity to create virtual machines in any cloud by writing a new Terraform module and slightly modifying the script for creating / deleting environments
We can play with Gitlab triggers and deploy new stands from the pipelines of development teams by transferring versions of services.