Software inventory with Salt on FreeBSD
Introduction
Software inventory is one of the 20 CIS Controls. The CIS Controls provide prioritized cybersecurity best practices. They are a recommended set of of actions for cyber defense that provide specific and actionable ways to stop today’s most pervasive and dangerous attacks.
Up to recently I was not doing software inventory (and control) for the SoCruel.NU platform. The platform is (almost) completely based on FreeBSD and all hosts (physical, virtual, laptop) are managed with SaltStack, so it would be nice if these can be used for this purpose. And it can!
SaltStack has a pkg module with which you can manage and control the FreeBSD software packages running on your Salt Minion.
This post requires the setup of SaltStack on FreeBSD. For more information on this see my SaltStack on FreeBSD series posts.
The basics explained
The SaltStack pkg module (for FreeBSD) is documented here. Please be aware that this is old documentation and that now for FreeBSD you just can use pkg instead of pkgng for calling the module!
To do an inventory of the software running on a FreeBSD system, we have to list the packages installed on that system. We can do this by using the Salt pkg module and calling the list_pkgs option. To list the packages on a system called minion.intra.domain.tld run the command from the Salt Master:
$ sudo salt 'minion.intra.domain.tld' pkg.list_pkgs
With Salt it is also possible to call multiple systems / Salt Minions:
$ sudo salt '*.domain.tld' pkg.list_pkgs
The output of the above commands in not really structured. But wait, Salt supports the JSON format for its output. To utilize just use the --out option:
$ sudo salt '*.intra.domain.tld' pkg.list_pkgs --out=json
The JSON output can be queried with the lighweight and flexible command-line JSON processor jq, which is available in the FreeBSD ports. to install jq use the command:
$ sudo pkg install jq
These are all the needed ingredients to be able to do some software inventory on FreeBSD based Salt Minions!
The inventory script(s)
The code in this chapter is all written in Bash. Bash is also available in the FreeBSD ports and can be installed easily:
$ sudo pkg install bash
Query host(s) for their packages
The first requirement I had was to be able to query a host (or hosts) and see which packages are installed on that particular host (or hosts) and format the output in a nice way. This can be done based on the basics described above and some Bash code.
We call this script si.sh and it needs the host(s) to be queried as input, like i.e. sudo si.sh host1.intranet.domain.tld or sudo si.sh '*.intranet.domain.tld'.
First we state that we use Bash and declare some program variables of tools we need:
#!/usr/bin/env bash # Commands varSalt="/usr/local/bin/salt" varSed="/usr/bin/sed" varCat="/bin/cat" varTee="/usr/bin/tee -a" varTouch="/usr/bin/touch" varGrep="/usr/bin/grep" varWc="/usr/bin/wc" varChmod="/bin/chmod" varRm="/bin/rm -rf" varJq="/usr/local/bin/jq"
Next we declare some exit states:
# Exit states varStateOK=0 varStateWARNING=1 varStateCRITICAL=2 varStateUNKNOWN=3
Now we can write the main part of the script:
varHostsOutFile="/tmp/_hosts.out"
${varRm} ${varHostsOutFile} >/dev/null 2>&1
${varTouch} ${varHostsOutFile}
${varChmod} 0640 ${varHostsOutFile}
varHosts="$1"
if [ -z "$1" ]if the input
then
echo "This function requires (a) hostame(s) as input, i.e. 'tst*.intranet.domain.tld'!"
exit $varStateWARNING
else
echo "The packages installed on ${varHosts} are:" | ${varTee} ${varHostsOutFile}
echo "" | ${varTee} ${varHostsOutFile}
${varSalt} ${varHosts} pkg.list_pkgs --out=json --static | ${varSed} '1d' | ${varSed} '/}/d' | ${varSed} 's/{//' | ${varSed} 's/"//g' | ${varSed} 's/....//' | ${varSed} '/retcode/d' | ${varTee} ${varHostsOutFile}
fi
So what do these 14 lines of code do:
- The first 4 lines create an output file (
/tmp/_hosts.out) - Line 5 declares the variable
varHostswhich is taken as an input parameter - Lines 6 until 14 define an
if then else fistatement - Line 6 checks if the $1 input parameter exists
- Lines 7 until 9 states what happens if the input parameter does NOT exist
- Lines 10 until 13 states what happens if the input parameter DOES exist, here is where it all happens (!)
- Line 11 echo’s a text to standard output and the output file (
/tmp/_hosts.out) - Line 12 echo’s a blank line to standard output and the output file (
/tmp/_hosts.out), just to make the output nicer - In line 13 you have the
saltcommand which queries thevarHostsfor their packages. The output is made nicer with a couple ofsedcommands and copied to standard output and the output file (/tmp/_hosts.out)
The output then looks like the below:
$ sudo ./si.sh test.intranet.domain.tld
The packages installed on test.intranet.domain.tld are:
test.intranet.domain.tld:
ca_root_nss: 3.53,
curl: 7.71.0,
db5: 5.3.28_7,
dialog4ports: 0.1.6,
entr: 4.5,
expat: 2.2.8,
fping: 4.2,
gdbm: 1.18.1_1,
gettext-runtime: 0.20.2,
gmp: 6.2.0,
gnuls: 8.30,
indexinfo: 0.3.1,
libev: 4.33,1,
libffi: 3.2.1_3,
libiconv: 1.16,
libinotify: 20180201_2,
libmaxminddb: 1.4.2,
libxml2: 2.9.10,
libzmq4: 4.3.1_1,
lowdown: 0.7.0,
minio: 2020.05.16.01.33.21,
minio-client: 2020.06.20.00.18.43,
nagios-check_ports: 0.7.4,
nagios-plugins: 2.3.3,1,
net-snmp: 5.7.3_20,1,
nginx: 1.18.0_15,2,
norm: 1.5r6_1,
nrpe3: 3.2.1,
openpgm: 5.2.122_6,
openssl: 1.1.1g,1,
p5-Crypt-CBC: 2.33_1,
p5-Crypt-DES: 2.07_1,
p5-Digest-HMAC: 1.03_1,
p5-Digest-SHA1: 2.13_1,
p5-Net-SNMP: 6.0.1_1,
pcre: 8.44,
perl5: 5.30.3,
pkg: 1.14.6,
portmaster: 3.19_25,
py37-Jinja2: 2.10.1,
py37-MarkupSafe: 1.1.1,
py37-asn1crypto: 1.3.0,
py37-certifi: 2020.6.20,
py37-cffi: 1.14.0,
py37-chardet: 3.0.4_3,
py37-cryptography: 2.6.1,
py37-distro: 1.4.0_1,
py37-idna: 2.8,
py37-libcloud: 3.1.0,
py37-msgpack: 0.6.2,
py37-openssl: 19.0.0,
py37-progressbar: 2.5,
py37-psutil: 5.7.0,
py37-pycparser: 2.20,
py37-pycrypto: 2.6.1_3,
py37-pycryptodomex: 3.9.7,
py37-pyinotify: 0.9.6,
py37-pysocks: 1.7.1,
py37-pyzmq: 19.0.1,
py37-requests: 2.22.0,
py37-salt: 3001_1,
py37-setuptools: 44.0.0,
py37-six: 1.14.0,
py37-tornado4: 4.5.3,
py37-urllib3: 1.25.7,1,
py37-yaml: 5.3.1,
python37: 3.7.7_1,
readline: 8.0.4,
rsync: 3.1.3_1,
sudo: 1.9.1,
Query if a package is installed somewhere
Often, I find out through e.g. Freshports that a certain package has a vulnerability. It is nice to know quickly if and on what hosts you have this package running.
This is my second requirement and again we do this with the basics discussed above and some Bash code.
We call this script si.sh (again) and it needs both the host(s) which we want to query as well as a package name to be queried as input, like i.e. sudo si.sh test.intranet.domain.tld apache24 or sudo si.sh '*.domain.tld' vim-console.
First we state that we use Bash and declare some program variables of tools we need:
#!/usr/bin/env bash # Commands varSalt="/usr/local/bin/salt" varSed="/usr/bin/sed" varCat="/bin/cat" varTee="/usr/bin/tee -a" varTouch="/usr/bin/touch" varGrep="/usr/bin/grep" varWc="/usr/bin/wc" varChmod="/bin/chmod" varRm="/bin/rm -rf" varJq="/usr/local/bin/jq"
Next we declare some exit states:
# Exit states varStateOK=0 varStateWARNING=1 varStateCRITICAL=2 varStateUNKNOWN=3
Now we can write the main part of the script:
varPackageOutFile="/tmp/_package.out"
${varRm} ${varPackageOutFile} >/dev/null 2>&1
${varTouch} ${varPackageOutFile}
${varChmod} 0640 ${varPackageOutFile}
varLiveServersOutFile="/tmp/_live-servers.out"
${varRm} ${varLiveServersOutFile} >/dev/null 2>&1
${varTouch} ${varLiveServersOutFile}
${varChmod} 0640 ${varLiveServersOutFile}
varHosts="$1"
varPackage="$2"
if [ -z "$1" ]
then
echo "This option requires a hostname as argument, i.e. test.intranet.domain.tld or '*.domain.tld'!"
exit $varStateWARNING
else
if [ -z "$2" ]
then
echo "This option requires a package name as argument, i.e. apache24!"
exit $varStateWARNING
else
${varSalt} ${varHosts} test.ping --out=json | grep true | ${varSed} 's/\true\>//g' | ${varSed} 's/,//g' | ${varSed} 's/://g' | ${varSed} 's/"//g' | ${varSed} 's/....//' > ${varLiveServersOutFile}
echo "The package ${varPackage} is installed on:" | ${varTee} ${varPackageOutFile}
echo "" | ${varTee} ${varPackageOutFile}
${varCat} ${varLiveServersOutFile} | while read varServerName
do
varPackageVersion=$(${varSalt} ${varServerName} pkg.list_pkgs --out=json --static | ${varJq} '.[]' | ${varJq} --arg k "$varPackage" '.[$k]')
if [ ${varPackageVersion} != "null" ]
then
echo ${varServerName} | ${varTee} ${varPackageOutFile}
fi
done
fi
fi
${varRm} ${varLiveServersOutFile}
Here we check if the hosts are actually live or not at the moment we issue this script. We do this with the line:
${varSalt} ${varHosts} test.ping --out=json | grep true | ${varSed} 's/\true\>//g' | ${varSed} 's/,//g' | ${varSed} 's/://g' | ${varSed} 's/"//g' | ${varSed} 's/....//' > ${varLiveServersOutFile}
All hosts which are live are put in the file ${varLiveServersOutFile}. This file is then read line by line (the while do done loop) and for each line (is host) then is checked if the given package is installed:
varPackageVersion=$(${varSalt} ${varServerName} pkg.list_pkgs --out=json --static | ${varJq} '.[]' | ${varJq} --arg k "$varPackage" '.[$k]')
if [ ${varPackageVersion} != "null" ]
then
echo ${varServerName} | ${varTee} ${varPackageOutFile}
fi
An example output of this script in action (with a query for the package vm-bhyve) looks like:
sudo ./si.sh '*.intranet.domain.tld' vm-bhyve The package vm-bhyve is installed on: host3.intranet.domain.tld host1.intranet.domain.tld host2.intranet.domain.tld host4.intranet.domain.tld
Wrap up
More can be done than just the examples above. The Salt pkg module has more options and capabilities than the ones shown. The module has e.g. also an audit option, which queries installed packages against known vulnerabilities. It is left to the reader to fill this one in.
I very much like that you query the actual situation on your hosts with the script explained above. It provides the situation as is now. And I find this powerfull. A script as documented above however does not provide historical information and/or a nice web interface. You could export the SaltStack queries to (a) text file(s) or even a database!
The SaltStack pkg module is Operating System independent. So if you have a mixed environment of FreeBSD, OpenBSD and even some Linux based distibutions, this would al work nicely!
For me this bare basic solution fits my needs for now! If you need a more comprehensive software inventory solution including a web interface / application, please take a look at OCS INVENTORY. It is fully supported on FreeBSD and is available in FreeBSD ports! I might implement it one day.
Resources
Some (other) resources about this subject: