It would seem that Terraform developers offer fairly convenient best practices for working with AWS infrastructure. Only there is a nuance. Over time, the number of environments increases, each one has features. Almost a copy of the application stack in the neighboring region appears. And the Terraform code needs to be carefully copied and edited according to the new requirements, or to make a snowflake.
My talk about patterns in Terraform for dealing with chaos and manual routine on large and long projects.
Video:
I am 40, I am 20 years in IT. I have been working for Ixtens for 12 years. We are engaged in ecommerce-driven-development. And I have been practicing DevOps practice for 5 years.
My story will be about experience in a project in a company, the name of which I will not say, hiding behind a non-disclosure agreement.
, . , , Amazon.
4 . , . , , . , - .
, , . , 4 .
.
, , time to market. , DevOps- . Terraform Puppet.
Terraform – open source HashiCorp. , , , .
– , - , , .
, . , .
Amazon. Terraform plan. Terraform plan : «, ». , , . .
, , Terraform apply Terraform instance, .
. - . instances, 53 .
. plan. , . . .
Terraform state-. . . , Amazon, , , , , Amazon. - , Terraform , Amazon.
state- . Git, . - , .
, . . Terraform bucket, state-. Terraform , state-, .
. . , .
Terraform , . . - .
, , Terraform apply . .
production - , instances, production instances .
. , . - , : , maintenance , , . , , .
, HashiCorp, - , .
production .
, - . , , , instance, , . . , .
Terraform . , IP-, , instance, IP- route53 .
, . , , 8 – .
. Jenkins job . pull request , , , , . performance , instances . , - , . . .
Jenkins shell-, Terraform. , . Terraform apply .
, .
- , , production, , , .
, , production. , production . , production operation . , , production, .
, , - . , , .
, Terraform – . . - , . .
- , pull request, , . , . , , -, . , Terraform-. , Terraform breaking chain , . . . pull request, , Terraform.
, , Terraform-, , - .
, operation, , pull request, . , . DevOps, , , . - .
, . , . : « , ».
, , - Terraform. , , , , Terraform-, , , .
, , Symlinks. Terraform . , , , . . , Symlinks .
, production assume role, - Amazon-. , , , , Terraform Amazon- .
Symlinks ? , Terraform state-. - . , Terraform . - , .
, - , . bucket. , sandbox sandbox-, production. , bucket production sandbox. , . - , - .
? , Terraform . Terraform . - . Terraform init, , .
shell-, , -, . Shell- . , -, . , state-, , , .
? JSON-. Terraform hcl (HashiCorp Configuration Language), JSON.
JSON shell-. , - bucket. bucket Terraform-, shell- .
bucket Terraform? remote state-. . . - , , Amazon: «, , instance», .
- . : «Terraform, , , state- ». environments.
state-. , VPC. Terraform-, VPC, VPC, , .
. . , VPC , , instance.
state- . , , , .
. Terraform ? , , 4 .
HashiCorp , Terraform-. Terraform fmt , . , , , HashiCorp, . .
– Terraform validate. , – , . ? . . Terraform validate.
, , , .
– , .
, Terraform, HashiCorp : « ? , – ?». .
Terraform , , Terraform : «, , , . , Terraform-».
, Terraform-, . , , , - , Terraform : « - , ».
– Terraform plan. , – . . , .
- , , - - - . Terraform plan , .
. , , Python, . , : Terraform- - , .
Terraform plan . , , .
, , , , – . pull request : «, ». . attach , .
– . , Terraform Amazon : « instance ? autoscale ?». , , refresh=false. , Terraform S3 state. , state , Amazon.
Terraform plan , state , . . -, - Terraform refresh. Terraform refresh , state , .
. . , Terraform, Terraform , . . . , , . pull request - , , . , Terraform plan.
, , user-data.
user-data? Amazon, instance, instance – -. instance , cloud init instances. Cloud init : «, – load balancer». - .
, , Terraform plan Terraform apply, user-data , . . . . , , - .
, Amazon, - .
, , template. : «, template». , Amazon.
– user-data. . . . , - user-data, : «, - – ».
, , Automate Terraform apply.
, Terraform apply , , , .
– . . . job, , , . , « » — , , , , - . , : «, , , ».
production, sandbox , , , , - . : autoscale-, security-, roles, route53 . , , .
, , , - persistent, , , - . jobs, .
Amazon Terminate protection. . . . Terraform Amazon : « instance, ». Amazon : «Sorry, . Terminate protection».
– . Terraform-, . , , - . , , , , .
. review. , - review , . , .
, . . . - , , - environment.
, , , . . , .
, Terraform, locals. output’ - , . .
, . . , ( ), , . : « , ». , , , , - , . , . , . : «, !». : «, . , ».
, , , , . , . . , , , , , .
:
- . , , - .
- . . . - , . , - - , Elasticsearch, Terraform plan, , . , .
- . , , . .
- production- . , - production - , . - .
- Terraform- , refactoring .
- Immutable infrastructure. AMI .
- route53, , .
- API rate limits. Amazon : «-, , , ». , .
- Spot instances. Amazon – spots . .
- IAM roles.
- , Amazone instances , . instances 100-150 – 1 000 . – .
- instances.
. Terraform – , . !
! state- S3, , state- ?
-, . -, flags, , - . . . , , , - - . – , state- Git . , - state-, , . . , Terraform . - , locks, , .
enterprise?
enterprise, . . , .
. . Amazon, instance . Terraform, Life Second , .
. .
. -, . - ? Test Kitchen. , - . Local Values. Input Variables? - Local Values? , - .
. – . , . , , -, , , , . , . . - , , - .
Local Values .
! ! . , . ?
, ! , , , , , . , , , , . , , , . , , . , . .
jsonnet -?
.
, . , .
– , , . . . , . . , , . .
. !
, . , Terraform . Ansible?
. Ansible , Puppet Amazon. Terraform .
Amazon?
, Amazon. Amazon. , Terraform . Ansible, : « 5 instances», , : « 3». Terraform : «, 2 », Ansible : «, 3». 8.
! ! Terraform. , Terraform - , Terraform .
. . . , , . ., .
. Remote backend, S 3. ?
?
Terraform Cloud .
?
4 .
4 , , , .
locks, state -. . .
, . .
, branch? ?
, . Terraform, Puppet, , - , . , , . branches, . , , -. , .
. . ?
.
branch . . . , , – , , . , , . . . .
! ! ! . , . , , ? - , ?
. , , - - . - .
. . ?
. . . , .
! ! . , . Puppet ?
User-data.
. . - ?
User-data – , . . , Daemon , , , load balancer.
. . - , ?
. .
! User — data. , , - - . - user — data Git, , User-data?
User-data template. . . - . Terraform . template , , , , , . – –-, -, , . , , - , . autoscale- , - instances autoscale- . , - . .
, – ?
, , . . . output’ . , , - – , User-data .
. , , Terraform .
.
, , , . . , tfvars, . . , tfvars ?
. . (: Production/environment/settings.tf) : domain = , domain vpcnetwork, vpcnetwork stvars – ?
. setting source, .
, tfvars. Tfvars testing-. tfvars instances, . . , . , , . , , tfvars.
, ?
, tfvars – . . tfvars . – . .
! , , Terraform ’? , - . ssh -. Google -, . Terraform , . , , , .
, – , , . . Terraform . . .
. . , , ?
, .
! . Mail. ru Group. …, ? , User — data, host name, Puppet ? . SG, . . SG, instances, ?
instances, , . , , , autoscale-. , .
, , . - , . , , , . , , , - .
?
SG instances, - ?
, . instance , , . , , , IP- . . -, Consul Discovery, , Kubernetes. Consul IP- instance.
. . IP, host name ?
host name, . . . instance – AE . . - , .
! , Terraform – , .
.
. , , Bare Metal instances? ? - , , Ansible, ?
Ansible . . . Ansible , instance . Terraform , instance . Bare Metal – .
, : «».
– , . Terraform-, - .
, – , . . , - : «, N , Amazon».
Terraform Front-End jobs, PagerDuty, data doc . . . .
! 4 Terraform. Terraform, , , , - - , plan. - . ? , ?
Mainly with our hands and eyes, if we see something strange in the report, then we analyze what is happening there, or we simply kill. In general, pull requests are commonplace.
If there is an error, are you doing rollback? Have you tried this?
No, this is a person's decision at the moment when he sees a problem.