Multi-master SaltStack setup

Leave a reply

Multi-master SaltStack setup is quite easy to build out. There is no need for VIPs or DNS CNAMEs
(though they can be implemented) and all of the functionality is handled by Salt.
To have working masters, you need to keep the following directories in sync. You may use
clustering filesystems or rsync to do that. In this example we will use rsync which is more than
enough. This howto describes how to do this on CentOS, but the setup should be the same on any
other OS (such as FreeBSD).
Initial setup
First off, install salt-master package on two servers that are designated to be master servers.
RHEL/CentOS:
view source

print

1.$ sudo yum install salt-master

FreeBSD:
view source

print

1.$ cd /usr/ports/sysutils/py-salt

2.$ sudo make install clean

Set file_roots in master configuration file:
view source

print

1.file_roots:

2.- /srv/salt
Lets also run salt-master as a nonroot user. Change user option in the master configuration file:
view source

print

1.user: salt

Change permissions of all directories:

view source

print

1.$ sudo chown -R salt:salt /etc/salt /var/cache/salt /var/log/salt

/srv/salt /var/run/salt
2.$ sudo chmod 664 /var/log/salt/master
Add your regular user to the salt group so your regular user can execute salt commands against
the minions.
RHEL/CentOS:
view source

print

FreeBSD:

1.$ sudo usermod -G salt username

?view source
print

1.$ sudo pw usermod username -G salt

Well be using a default config for everything else.

Multi Master configuration
Lets jump to the multi master configuration.
Make sure your master keys are the same on all the masters. This needs to be done only once or
whenever you update the keys. You can sync the keys from one of the servers. Master keys are
located in /etc/salt/pki/master/master.{pem,pub}
Then, sync the minion public keys (obviously it would be beneficial to setup ssh keys for the salt
user before doing this). Make sure this directory is synced whenever you add a new minion.
view source

print

1.$ sudo -u salt /usr/bin/rsync -avz -e ssh /etc/salt/pki/master/

{minions_pre,minions,minions_rejected,minions_denied,minions_autosign}
salt@${masterhostB}:/etc/salt/pki/master/
Next, sync file_roots.
view source

print

1.$ sudo -u salt /usr/bin/rsync -avz -e ssh /srv/salt salt@$

{masterhostB}:/srv/salt
Were done with salt master setup. We should have two identical masters now. Lets setup the
minions now.
Minion setup
Install salt-minion package.
RHEL/CentOS:
view source

print

1.$ sudo yum install salt-minion

FreeBSD:
view source

print

1.$ cd /usr/ports/sysutils/py-salt

2.$ sudo make install clean

Edit the minion config file to point to the masters. We will use failover type scenario in which the
minion will connect to only master a time but will failover to another master if the first one goes
down:
view source

print

1.$ sudo cat /etc/salt/minion

2.master:
3.- masterhostA

4.- masterhostB
5.master_type: failover
6.master_alive_interval: 60
Master type is set to failover thus all minions will only connect to one master server. If we remove
that setting, minions will connect to all master servers but will not be able to determine which one
is live until they to contact them. We can adjust the availability check using the
master_alive_interval setting (which in this example is set to 60 seconds).
If you want to connect to a single, random master server, use the random_master: True setting.
Start the salt-minion service:
view source

print

1.sudo service salt-minion start

Watch the first salt master for minions keys to pop up:
view source

print

1.sudo salt-key -L

Since salt-master is running as salt user, we still need root perms to view keys that are available to
access. Word of caution: whenever you run salt-key, the daemon changes the permissions of
/etc/salt/pki/master directory to whatever the daemon is running as. If we ran the salt-master
daemon as root, the permissions of that directory would change to root:root which would be
problematic when rsyncing as salt user (you should never be ssh-ing as root user for securitys
sake). Running salt-master as salt user will keep these permissions as salt:salt after running saltkey.
You should see the minions public key pending under Unaccepted Keys:
view source

print

1.$ sudo salt-key -L

2.Password:
3.Accepted Keys:
4.Unaccepted Keys:
5.webserver01
6.Rejected Keys:
Accept the key with sudo salt-key -A. You will see this minions public key in
/etc/salt/pki/master/minions directory on only one salt master server. Now thats a problem so
lets sync that directory using one of the rsync scripts that we came up with:
view source

print

1.$ sudo -u salt /usr/bin/rsync -avz -e ssh /etc/salt/pki/master/

{minions_pre,minions,minions_rejected,minions_denied,minions_autosign}

salt@${masterhostB}:/etc/salt/pki/master/
This will keep the directories in sync on both master servers and when you run the sudo salt-key
-L command, you will see the minions key on both servers. You could also employ something like
glusterfs, but rsync scripts are simple and perfectly fine.
Test salt state modules or execution modules against a minion from one of the salt masters. You
will be able to call salt commands from only one master because the minion connects to only one
server (consequence of master_type: failover setting in master configuration file).
view source

print

1.$ salt 'webserver01' test.ping

2.webserver01:
3.True
You should see a successful ping test.