Clever Geek Handbook

Salt. Say a word about the glorious pillar

In one of our past articles on Just add some Salt, we covered how we migrated 700+ servers to Salt. We shared our experience with optimizing Salt: how to apply and customize it effortlessly. Then we just touched on the topic of pillars, but today we would like to dwell on it in more detail.



Different pillars are needed



Pillars are a secure (secure) data store inside Salt. Therefore, first of all, they are used to delimit access to critical data (certificates, logins, passwords).



In addition to the default pillars, Salt also has the ext_pillar module , which provides an interface for connecting to external data sources and generates pillars from these sources into one common dictionary.



For example, you can take data from mysql, postgres, redis, mongo, git , or even from the output of the script / command - cmd_json , cmd_yaml .



A complete list of modules is published on the official SaltStack website .



If you have a non-standard situation and the available modules do not suit you, you can write your own and put it in / usr / lib / python3 / dist-packages / salt / pillar /, after which you need to restart the wizard.



An example of such a module: 



#/etc/salt/master/conf.d/pillar.conf
ext_pillar:
  - dummy: dummy
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    




#/usr/lib/python3/dist-packages/salt/pillar/dummy.py
def ext_pillar(minion_id, pillar, *args, **kwargs):
    dummy = {'dummy': 'what u want mann?'}
    return dummy




Of all the available modules, pillarstack turned out to be the most interesting and relevant for our team. Now let's tell you why.



Introduction to PillarStack



Stack or pillarstack is "a simple and flexible YAML pillar that can read pillars from pillars," according to the official documentation on the site .



It allows you to use the pillars read earlier inside the pillars. This means that we can use the pillar from the previous config. And it's mega-convenient! Let's show how it works: 



,      :
#/etc/salt/conf.d/pillarstack.conf
ext_pillar:
  - stack:
    - /srv/pillar/stack1.cfg
    - /srv/pillar/stack2.cfg
    - /srv/pillar/stack3.cfg

#/srv/pillar/stack1.cfg
#    stack  "" ,
#    2      yml ,    

core.yml
common/*.yml
osarchs/{{ __grains__['osarch'] }}.yml
oscodenames/{{ __grains__['oscodename'] }}.yml
hosts/{{ minion_id }}/roles.yml

#/srv/pillar/stack2.cfg
#   stack      stack1.cfg
#      , , roles
#  stack   yml 

{% for role in stack.roles %}
roles/{{ role }}/*.yml
{% endfor %}

#/srv/pillar/stack3.cfg
#    stack ""   stack1.cfg  stack2.cfg
#       
creds/{{ stack.db.host }}/*.yml




Each config can be represented as a layer. In each such layer, we can use data from the previous layers.



The following variables are available when configuring pillars:



  • grains - targeting configs by grains
  • opts - targeting configs by options in config
  • pillar - pillars that were formed before processing the current ext_pillar: stack




# ,      stack

#      stack      grains   pillar
# stack.cfg   ,    pillar 
ext_pillar:
  - stack:
      grains:cpuarch:
        x86_64:
          - /srv/pillar/stack1.cfg
          - /srv/pillar/stack2.cfg
      pillar:environment:
        dev: /srv/pillar/dev/stack.cfg
        prod: /srv/pillar/prod/stack.cfg


#       stack:     grains,   pillar
#   pillar     stack1.cfg, stack2.cfg
#      'environment',  stack.cfg  
ext_pillar:
  - stack:
      grains:custom:grain:
        value:
          - /srv/pillar/stack1.cfg
          - /srv/pillar/stack2.cfg
  - stack:
      pillar:environment:
        dev: /srv/pillar/dev/stack.cfg
        prod: /srv/pillar/prod/stack.cfg




In yml files you can use:



  • __opts__: config options
  • __grains__: minion grains
  • pillar: pillars from other ext_pillar or from default pillars (those in top.sls)
  • stack: accumulated pillar stack, including the current config.




If you are just going to switch to pillarstack, then below are a couple of points worth paying attention to:



1. Recursive inclusion works only in pillar. Stack doesn't include directories, only files. 



# pillar
# top.sls
base:
  '*':
    - dir1.* #  /dir1/dir2/dir3/*

# stack    
# stack.cfg
/dir1/*
/dir1/dir2/*
/dir1/dir2/dir3/*




2. Default behavior when merging lists:



  • pillar - lists are overwritten
  • stack - the merge-last strategy is used (the lists are added).




Pillar fusion strategies allow for very flexible customization of pillars.

A detailed description with examples is in the documentation.



To summarize each strategy:



  • merge-last (default): recursive merge, last dictionary / list takes precedence
  • merge-first: recursive merge, priority is given to first dictionary / list
  • remove: remove the specified elements
  • overwrite: dictionary / list override.




Explanation: in each merge two dictionaries / lists are involved, let's call them the first and the last The



strategy is indicated as the first element of the dictionary / list to which it must be applied.

The main thing is not to get carried away with excessive use of strategies, as this will complicate the configuration. Perhaps, in this case, it is worth reconsidering the organization of the pillars.



Conclusion



Pillars allow you to restrict access to sensitive data. As the data grows, the scenarios for its use are becoming more sophisticated, so it is important to use a convenient tool for organizing it. At the moment for our team this is pillarstack, in the future we plan to deploy vault.



It seems to us surprising why pillarstack has not yet supplanted the default pillar, because it is much more convenient and flexible, and strategies are very helpful when it is necessary to change the behavior of the stack variable pointwise. What do you think? Do you use pillarstack in your work?


More articles:


All Articles