Hello, Habr! My name is Sasha Kozlov, I have been developing infrastructure and system administration in Avito for the last three and a half years. I'll tell you how we scaled and modernized our work with infrastructure code over time and took it to a qualitatively new level.
We are responsible for the full cycle of equipment management: from procurement and installation in the DC to delivery to the end user. It's about the lessons we've learned over the past few years, working with thousands of pieces of equipment and several thousand pieces of configuration.
We will talk about the "first generation" IaC tools such as Ansible, Chef, Salt, Puppet. If you are dealing with on-premises infrastructure, and you do not have your own cloud on virtualization, then most likely you are already using one of these tools.
Puppet. - , , Puppetlabs . , , , , . , β .
, . , . - push-, - pull, ansible pull-. β Python, Jinja- YAML-, β DSL Ruby. : , PuppetDB, , .
, . git revert, , . , , . β β , 0 β β. .
, Puppet. - β , , . , :
- , ;
- ;
- CI-, ;
- ;
- , , ;
- «» Configuration Drift.
, , . β . .
. : , , .
Puppet , . , .. pull-. β . 30 , , , Puppet . , event-driven , Salt, .
, . stateful-, , . .
: Puppet stateful-, , . , DNS- Puppet, . , , , API.
, . , . , , git . . .
: control repo . git-. , Puppet-, "The roles and profiles method".
Control repo β . , CI- - . β , . β , Β« ?Β». : k8s- , Kafka , ClickHouse- ..
β , . - . Puppet -, , . , , , .
control repo, .
β , , , . control repo, semver . 50 : , , Kubernetes .
, , Puppet. . , , . , control repo?
, , . control repo , . , . , , . Docker, . control repo, . , .
, , :
- Building a Functional Puppet Workflow Part 2: Roles and Profiles
- Roles and Profiles in a Control Repo?
- Workflows Evolved: Even Besterer Practices
- Profiles and the Path to Hiera Data
- Puppetlabs Best Practices Docs
- control repo, . , , , :
External Node Classifier
, , β , ?
node definitions , , . , - :
node /^avi-ceph(2[1-9]|3[0-9]|4[0-9]|5[0-9]|6[0-9]|7[0-9]|8[0-9]|9[0-9])/ {
...
}. β , . , , . Puppetlabs External Node Classifier.
External Node Classifier β , , . , . ENC , top-scope variables. control repo , , , node definition:
node default {
include base # , control repo
if $::role != '' {
notify{ "Node ${::fqdn} has role ${::role}": loglevel => info }
include "role::${role}"
} else {
notify{ "Node ${::fqdn} has no role": loglevel => warning }
}
}ENC , . CMDB, . CMDB netbox Digital Ocean, . razor, , , -.
CMDB - , 15-20 , , .
«» ? , Configuration Drift, . Configuration Drift β , , . β , , , - Puppet.
, , , . , . , k8s-.
, API-. netbox API, . PXE netboot- Debian, preseed' , API, IPMI Redfish API. , , , .
, , . , .
. : , , , , .
, β , . β . . βwrite-onlyβ , .
3.7 6 Puppet, , . :
- .
- -.
:
- , ..
- , , duplicate resource declaration. : , , .
- Docker-, , inspec.
Kubernetes Puppet "Kubernetes The Hard Way", .
, , smoke-, . , , .
, Beaker , , -, . ~~ smoke-: ~~ , .
, , :
context 'application deployment' do
it 'can deploy an application into a namespace and expose it' do
shell('systemctl restart kubelet')
shell('count=0;
while [[ $(kubectl get pods -n tiller -l name=tiller -o \'jsonpath={..status.conditions[?(@.type=="Ready")].status}\') != "True" ]];
do
if [[ count -gt 180 ]]; then
break;
fi;
sleep 1 && ((count++));
done')
shell('kubectl create -f /tmp/nginx.yaml', acceptable_exit_codes: [0]) do |r|
expect(r.stdout).to match(%r{namespace/nginx created\nconfigmap/my-nginx-config created\ndeployment.apps/my-nginx created\nservice/my-nginx created\n})
end
end
it 'can access the deployed service' do
shell('count=0;
while [[ $(kubectl get pods -n nginx -l run=my-nginx -o \'jsonpath={..status.conditions[?(@.type=="Ready")].status}\') != "True" ]];
do
if [[ count -gt 180 ]]; then
break;
fi;
sleep 1 && ((count++));
done')
shell('curl --connect-timeout 1 --retry-delay 1 --retry-max-time 300 --retry 150 -s --retry-connrefused 10.100.10.5', acceptable_exit_codes: [0]) do |r|
expect(r.stdout).to match %r{Welcome to nginx!}
end
end
end, . CI .
CI PR. PR : . , , , CI PR, , .
, :
- puppet-syntax
- puppet-lint
- rspec-puppet
- puppetlabs-spec-helper
- test-kitchen, Puppet: kitchen-docker-puppet-example
- beaker
-:
Development kit
, CI/CD β , workflow.
- β PDK, . , , . PDK control repo Kitchen Docker. PDK Beaker, .
, :
- , : , CI .
- , , -.
- , -, .
- control repo, , .
- .
- .
development tool, CI , .
(puppet-rspec, puppet-linter, test-kitchen) , . , , .
, , Β« Β». , .
, , . , β . : Forge. Puppet Forge , .
Puppet . , : metadata.json Puppetfile. , - . . librarian-puppet, , r10k, . , . β , : acceptance- -.
. , git-, :
mod 'dba-clickhouse',
:git => 'ssh://git@github.com/iac/dba-clickhouse.git',
:ref => '1.2.2'
mod 'dba-kafka',
:git => 'ssh://git@github.com/iac/dba-kafka.git',
:ref => '1.2.0', , . Puppet Forge, , librarian-puppet.
semver, , . , , β :
# Puppetfile
mod 'arch-puppetserver', '0.20.5' #
mod 'arch-vault', '~> 2.1' #
mod 'si-lxc' # , . , , , . :
[22:39:43] in dba-control on ξ production via ruby-2.5.1 at οΈ unstable
$ iack dep show
[] Collecting modules metadata
FULL NAME | CURRENT VERSION | LATEST VERSION | OUT OF DATE?
---------------------|-----------------|----------------|---------------
si-lxc | latest | 0.3.2 | N/A
si-base | latest | 1.3.1 | N/A
petems-hiera_vault | v0.4.1 | | Major
arch-vault | 2.1.0 | 2.1.0 | No
dba-postgresql | 0.1.2 | 0.1.3 | Tiny
dba-pgbouncer | 0.4.0 | 0.5.1 | Minor
si-grub | 0.1.0 | 0.1.0 | No
si-collectd | 0.2.3 | 0.2.4 | Tiny
si-confluent | 0.3.0 | 0.3.0 | No
dba-redis | 0.2.3 | 0.2.3 | No
dba-collectd_plugins | latest | 0.2.0 | N/A
dba-mongodb | 0.2.1 | 0.2.1 | No
dba-patroni | 0.1.4 | 0.2.4 | Minor
dba-cruise_control | 0.1.1 | 0.1.2 | Tiny
dba-lxd | 0.7.0 | 0.7.0 | No
dba-clickhouse | 1.2.1 | 1.2.2 | Tiny
dba-zookeeper | 2.0.0 | 2.0.0 | No
si-td_agent | 0.1.0 | 0.1.0 | No
dba-kafka | 1.1.6 | 1.2.1 | Minor
arch-puppetserver | 0.20.1 | 0.20.2 | Tiny
pcfens-filebeat | 4.1.0 | 4.4.1 | Minor
KyleAnderson-consul | 5.0.3 | 6.0.1 | Major
puppetlabs-apt | 6.3.0 | 7.4.2 | Major
puppetlabs-stdlib | 5.2.0 | 6.3.0 | Major:
- Managing environment content with a Puppetfile
- puppet-forge-server β ruby Sinatra, Puppet Forge
- librarian-puppet
code style
code style ? β , . , , , . review β , .
Puppet . The puppet language style guide . Puppet-lint, CI, .
, . control repo:
development kit, «»:
β
, . Puppet Hiera Vault. hiera-backend, vault hiera_lookup.
-, , - . , :
$token_data = vault::secret_field('tokens.csv', 'data')token_data 'data' tokens.csv, Vault. Vault , , Hiera:
$ vault-util ls puppet/arch/
common/
nodes/
roles/ , , common. β roles/ nodes/.
β , -, , , . TLS- .
, , , . , Puppet:
Canary
Puppet , control repo. control repo , . ENC .
, . , . β - .
, , , .
Puppet
Puppet, , , . DSL β Ruby. Hiera β , . , . Hiera , , .
Puppet β , , . . , : , Hiera, . - , , .
Puppet Ruby , . , pet project. , , β . : , , , .
β . , pull-. , , . , , pull push. Bolt, Puppetlabs , , PuppetDB, .
, , :
- , .
- .
- .
- .
- CI .
- , workflow .
- External Node Classifier .
- .
- .