Prerequisites:

Have a salt master and a salt minion configured
Have data which you want gpg encrypted

Introduction

I presume that anyone reading this blog is aware that it is not done to paste plaintext passwords in code (commited to git) (albeit a prive repo or not). Therefore since the release of salt 2016.3.0 it is possible to encrypt sensitive data with gpg keys.

Short introduction to gpg:

Pretty Good Privacy (PGP) is an encryption program that provides cryptographic privacy and authentication for data communication. PGP is used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications. Phil Zimmermann developed PGP in 1991.[3]

PGP and similar software follow the OpenPGP, an open standard of PGP encryption software, standard (RFC 4880) for encrypting and decrypting data.

source: Wikipedia

All the following tasks will be performed on your salt master, unless specifically stated not so

Basic setup and creation of the keys

Perform the following tasks to store the gpg keys in:

sudo mkdir -p /etc/salt/gpgkeys
sudo chmod 0700 /etc/salt/gpgkeys

! A warning ahead: Please do not fill in the comment or email adress. Later on in this manual you will encrypt secrets with gpg data based on a unique identifier which is your "real name". Use something convenient like salt-master !

Now we are ready to generate the gpg keys.
This can be done in a short or a longer method:

Short method: gpg --gen-key --homedir /etc/salt/gpgkeys
Long method: gpg --full-generate-key --homedir /etc/salt/gpgkeys

The short method will only ask you for the following two questions:

your real name
your email address ! You do not need to answer this ! Beter yet, just don’t use it

The short method will provide you with this output upon completion:

gpg --gen-key --homedir /etc/salt/gpgkeys
gpg (GnuPG) 2.2.12; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Vuurvoske
Email address: # DO NO ENTER THIS #
You selected this USER-ID:
    "vuurvoske"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key D4B3D31C777DD911 marked as ultimately trusted
gpg: revocation certificate stored as '/etc/salt/gpgkeys/openpgp-revocs.d/9E623C33F36CEEFBB395366AD4B3D31C777DD911.rev'
public and secret key created and signed.

pub   rsa3072 2021-10-27 [SC] [expires: 2023-10-27]
      9E623C33F36CEEFBB395366AD4B3D31C777DD911
uid                      vuurvoske
sub   rsa3072 2021-10-27 [E] [expires: 2023-10-27]

The following dirs and files will be created in the /etc/salt/gpgkeys directory:

ls -lh /etc/salt/gpgkeys
total 20K
drwx------ 2 vuurvoske vuurvoske 4.0K Oct 27 20:06 openpgp-revocs.d
drwx------ 2 vuurvoske vuurvoske 4.0K Oct 27 20:06 private-keys-v1.d
-rw-r--r-- 1 vuurvoske vuurvoske 3.9K Oct 27 20:06 pubring.kbx
-rw-r--r-- 1 vuurvoske vuurvoske 2.0K Mar 26  2021 pubring.kbx~
-rw------- 1 vuurvoske vuurvoske 1.3K Oct 27 20:06 trustdb.gpg
ls -lh openpgp-revocs.d/ private-keys-v1.d/
openpgp-revocs.d/:
total 8.0K
-rw------- 1 vuurvoske vuurvoske 1.6K Mar 26  2021 5DDC0B5689E1DCAF667D4A2B1D2375B73AD74CBF.rev
-rw------- 1 vuurvoske vuurvoske 1.6K Oct 27 20:06 9E623C33F36CEEFBB395366AD4B3D31C777DD911.rev

private-keys-v1.d/:
total 16K
-rw------- 1 vuurvoske vuurvoske 1.4K Mar 26  2021 0EF447AA98AD58E4FE34E4711242D33D45B96191.key
-rw------- 1 vuurvoske vuurvoske 1.4K Mar 26  2021 2380134059E78A3679048C992BC679AB42506452.key
-rw------- 1 vuurvoske vuurvoske 1.6K Oct 27 20:06 2D5AA1C2CBE17425B548595DD0FB05ABDC72DC68.key
-rw------- 1 vuurvoske vuurvoske 1.6K Oct 27 20:06 8DA09ED3A3E7C5506B68E802D2C2D19021C96B96.key

The long method will give you this dialog:

gpg --full-generate-key --homedir /etc/salt/gpgkeys
gpg (GnuPG) 2.2.12; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) RSA and Elgamal
   (3) RSA (sign only)
   (4) RSA (sign only)
Your selection?

Your choice does matter. Salt requires you to choose for RSA. I test it thorougly and only with RSA keys i was able to decrypt my messages. For example RSA + elgamal 3072 does not work. RSA with 3072 or 4096 bits however do.

We choose 1 here

The following question is about key bit length:

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)

Let’s just choose 4086 here for max protection.

output:

Requested keysize is 4086 bits

Now we’re asked how long the key should be:

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0

Zero is the default, which i’m going to choose aswell. This is for my homelab environment.

Key does not expire at all
Is this correct? (y/N)

Confirm your answer here

Next up, we’re giving the system our real name

GnuPG needs to construct a user ID to identify your key.

Real name:

Which is ofcourse Harry Potter

In addition to your name, also your email and a comment will be asked.
You can choose whatever fits your glove. But it’s better not to use them

Real name: vuurvoske
Email address: # do not enter this ! #
Comment: # do not enter is ! #
You selected this USER-ID:
    "vuurvoske"

Confirm by typing O

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?

After doing this we can provide a passphrase for our key and we will not:

Keep confirming these kind of screens (4 times in total)

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: WARNING: some OpenPGP programs can't handle a RSA key with this digest size
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key D2E3X085ACED4C72 marked as ultimately trusted
gpg: revocation certificate stored as '/etc/salt/gpgkeys/openpgp-revocs.d/0EF447AA98AD58E4FE34E4711242D33D45B96191.rev'
public and secret key created and signed.

pub   RSA4096 2021-10-27 [SC]
      0EF447AA98AD58E4FE34E4711242D33D45B96191
uid                      vuurvoske
sub   rsa4096 2021-10-27 [E]

Note: I am not using the keys generated as shown in this tutorial

Importing to keys into the master

When they already exists and you just need them imported

gpg --homedir /etc/salt/gpgkeys --import /path/to/private.key
gpg --homedir /etc/salt/gpgkeys --import /path/to/pubkey.gpg

remarks about the keyrings

In this howto well be creating the gpg keys and importing them to /etc/salt/gpgkeys.
This brings some challenges. Just for quick reference, if you want to list your keys or delete them. Please refer to the commands underneath. (hopefully it will save you alot of time)

salt gpg db:

# listing the keys
gpg --homedir /etc/salt/gpgkeys --list-keys
# listing the secret (private) keys
gpg --homedir /etc/salt/gpgkeys --list-secret-keys
# deleting a secret key:
gpg --homedir /etc/salt/gpgkeys --delete-secret-keys "Vuurvoske (1337 h@x0r) <email at adress>"
# or # 
gpg --homedir /etc/salt/gpgkeys --delete-secret-keys "4CACC95DFAC3611BD843FE90D2D9E067ACED3C71"
# deleting the public key:
gpg --homedir /etc/salt/gpgkeys --delete-key "Vuurvoske (1337 h@x0r) <<email at adress>>"
# or # 
gpg --homedir /etc/salt/gpgkeys --delete-key "4CACC95DFAC3611BD843FE90D2D9E067ACED3C71"

your own user’s gpg db:

#listing the keys:
gpg --list-keys
# listing the secret (private) keys
gpg --list-secret-keys
# deleting a secret key:
pg --delete-secret-keys "Vuurvoske (1337 h@x0r) <email at adress>"
# or #
gpg --delete-secret-keys "4CACC95DFAC3611BD843FE90D2D9E067ACED3C71"
# deleting a public key:
gpg --delete-key "Vuurvoske (1337 h@x0r) <email at adress>"
# or #
gpg --delete-key "4CACC95DFAC3611BD843FE90D2D9E067ACED3C71"

–> please be aware that if you delete these keys, all your previously created secrets can not be used by salt anymore. You will have to re-encrypt them with a new key <–

**Extra remark: From this point I will just use the <username> reference in my gpg commands. Just be aware you can also use the HEX-like code

Altering keys:

For salt you should specify the –homedir, for other purposes use your own gpg keyring. With this command you can also change the level of trust for the key

gpg --homedir /etc/salt/gpgkeys --edit-key <key_id>

Importing the public key to the keyring of your user.

So it can be used for encryption of secrets without much hassle Because during our generation of the keys we specified /etc/salt/gpgkeys only this command will show these keys: gpg --homedir /etc/salt/gpgkeys --list-keys. However when we want to encrypt secrets with the new keys, this will not work. Because the keyring uses a diffrent default dir. Therefor we should import the public key to our own user so that we can easily use the identifier we choose during generation. But first, we have to export our own key

gpg --homedir /etc/salt/gpgkeys --armor --export "vuurvoske" > vuurvoske.gpg

This step is optional, just depends if you want to encrypt secrets on your salt-master or not
Import it to your local user’s gpg db:

gpg import /path/to/key/vuurvoske.pgp

Exporting the keys to a different location

If you want to store them remotely or on a removable disk:

Use the option gpg_keydir for this as seen in the example:

gpg --homedir /etc/salt/gpgkeys --armor --export <KEY-NAME>.key > /run/media/usb/exported_private_key.gpg

note: This can also be done for the public key in the same manner

Encrypting secrets!

Securely copy the pubkey to whatever VM or physical machine you want to use the key on.

Import the key as follows:
As a clarification. You should do this either on your salt master as a local user or on a different machine. Transfer your key first, before trying to import

gpg --import exported_pubkey.gpg

When done so, use the following command to perform the encryption:

echo -n "supersecret" | gpg --armor --batch --trust-model always --encrypt -r "vuurvoske"

You can also just encrypt entire files:

cat supersecretfile.txt | gpg --armor --batch --trust-model always --encrypt -r "vuurvoske"

note: echo -n makes sure your history does not “record” your value entered at “supersecret”

To use these secrets in pillar data. Please place #!yaml|gpg at the first line of each pillar data file (where you will use secrets). In example to encrypt a ssh private key:

echo -n "test123" | gpg --armor --batch --trust-model always --encrypt -r "vuurvoske"

Gives output:

In a pillar file you can use it as follows:

/srv/pillar/files/a-secret.yaml

#!yaml|gpg
# this is a gpg encrypted ssh private key file for user X

private-keys:
    a-secret: |
        -----BEGIN PGP MESSAGE-----

        hQGMAxF9pl+zYwPPAQwAsFa28f2ReTaURzVjSdMQ8+dPfSdYNRcVCLWkKJvbrY1F
        GJRJ3HL1g5MXRE81ptqVDfhXKcjty8bjp8+NLKITat0+osRd85gBqCFc/kbHYRtL
        YK8f91WN0PUS6zys/ask3435saAGSDGsfasfa6FYSWLTOuOHr/eb0aQjUoQB6N7r
        6TzC9bnImYKp8yqS6NowT35eLV59hfEjBQHVuWMYXpIivzLwjM5wE20GTAS8gfyY
        sjcC77AS43gDAS/V2nO3rzbIJZH/KqniBDvB7RpdQu6FdyUH18F5NqafW8r28w3ofK
        d/J1sR+2xaP+sv7VFo8DV3yu7gX9/ZP0G8IlbM7Q1cXcpo9CVyWugh2maPRQ2R4D
        ZcsqZZrzZllBEM7SfVRJyEm5lfza6bgI37KcgCnA14+/OGbiyKFECVq06dQAaQfL
        DE6dDOdturpwuXpMiCb/7gAvk1kSGcpc7M7musLVWb1BBz2yrpbYU0iSNw02cGCy
        OEuetSQMabfxGXTheBgg0kIB4U69F+LTG0jD49B1t8Z4KtpxjxDSUaR6GoUeZa8D
        6/YWAt+gHxa1xfvmqIzbssZM1NNToBkuVLr+vCEc1H4IOCI=
        =0Rvw
        -----END PGP MESSAGE-----

please note: the private-keys indentifier is optional, when using only one secret i recommend not using it

Example on pillar data location:

mkdir /srv/pillar/secure
touch /srv/pillar/secure/a-secret.sls

In the top.sls configure this as follows:

base:
  'hades*':
    - secure.a-secret

note: make sure that in your pillar data top.sls your minion can ‘reach’ this pillar file.

To see if your secret is read use the following command:

salt 'hades*' pillar.item a-secret
hades.domain.local:
    ----------
    a-secret:
        test123

using the gpg encrypted secret in states

Ofcourse we also want to use these nice gpg encrypted secrets in salt
Please refer to the examples underneath

I know pubkeys don’t necessarily have to be encrypted, but i like the example ;)

You can use it in a state / highstate as follows:

{%- set user = pillar['user'] %}

/home/{{ user }}/.ssh/keys/a-secret:
  file.managed:
    - user: {{ user }}
    - group: {{ user }}
    - mode: 600
    - attrs: a
    - contents_pillar: private-keys:a-secret

Apply the state like underneath to apply the pillar data to the minion:

salt 'hades*' state.apply ssh.privkeys test=true
hades.domain.local:
----------
          ID: /home/vuurvoske/.ssh/keys/home
    Function: file.managed
      Result: True
     Comment: File /home/vuurvoske/.ssh/keys/home updated
     Started: 22:22:13.528167
    Duration: 34.272 ms
     Changes:   
              ----------
              attrs:
                  ----------
                  new:
                      a
                  old:

If you would rather just use it on the cli, that can also be performed:

First set a variable behind - ssh_keys:

ssh_deploy:
  ssh_auth.present:
    - user: {{ user }}
    - ssh_keys: {{ secretvariable }}
    - config: '%h/.ssh/authorized_keys'

Then apply the state as follows:

salt 'hades*' state.apply pillar="{'secretvariable': 'paste your encrypted value here'}"

note: when unsure, append test=true at the end

Used documentation:

Encrypting private pillar data on your salt master

And thus you can push your code to git without exposing your secrets