First Zeek queries on FreeBSD

Introduction

This is the third blog post in a series of posts about running Zeek on FreeBSD. The blog posts so far in this Zeek on FreeBSD series are:

In this third blog post we do some basic queries on some of the Zeek logfiles.

Technical prerequisites

The following technical prerequisites have to be in place to be able to implement what is described in this post:

have Zeek implemented as desribed in Implement Zeek on FreeBSD

The Zeek logfiles

Zeek is not an active protective security device, like a firewall or intrusion prevention system. Rather, Zeek runs on a “sensor”, a hardware, software, virtual, or cloud platform that quietly and unobtrusively observes network traffic. Zeek interprets what it sees and creates compact, high-fidelity transaction logs.

In our case these logs reside in the /var/zeek/logs and /var/zeek/spool directories. If we list the files in the /var/zeek/spool/zeek directory (or /var/zeek/logs/current) you will see all the current Zeek logfiles.

We look at the following log files in this blog post:

conn.log: the log file with all the connections
dns.log: the log file ith all the DNS activity
ssl.log: the log file with a record of SSL sessions, including certificates being used

These are not all the logfiles produced by Zeek! More information about the logfiles can be found in the Zeek User Manual.

Query the connection log

The Zeek connection log conn.log provides a lot of information about all your TCP/UDP/ICMP connections. Our first query on conn.log shows the top 10 source IP addresses that generated the most network traffic:

First we go the the directory where the logs are:

$ cd /var/zeek/logs/current

Then we issue the query on the log file:

$ cat conn.log | jq -j '.["id.orig_h"], " ", "\n"' | sort | uniq -c | sort -rn | head -n 10

The above command is now explained in more detail:

cat conn.log | jq -j '.["id.orig_h"], selects the id.orig_h (is the source IP address of the connection) column from the conn.log log file
" ", set a space between output columns
"\n"' puts a newline character at the end of the line
| sort uses the sort command to organize the rows in alphabetical order
| uniq -c uses the uniq command with the -c option to remove duplicates while returning unique instances and their counts
| sort -rn uses the sort command with the -rn option to organize the rows in reverse numerical order
| head -n 10 uses the head command with the -n option to display the 10 top most values.

We can also do this for destination UDP/TCP ports (column id.resp_p from the log file):

$ cat conn.log | jq -j '.["id.resp_p"], " ", "\n"' | sort | uniq -c | sort -rn | head -n 10

Each log entry in the conn.log log file has also a service attribute, which we can query as well:

$ cat conn.log | jq -j '.["service"], " ", "\n"' | sort | uniq -c | sort -rn

Query the DNS log

Another log file is dns.log. Zeek logs all DNS activity it sees in this log file. And we can query this log file! As first example we can make an overview of which domains have been queried with DNS (we are still in the /var/zeek/logs/current directory):

$ cat dns.log | jq -j '.["query"], " ", "\n"' | sort | uniq | rev | cut -d '.' -f 1-2 | rev | sort | uniq -c | sort -nr | head

Here is a breakdown of the above command:

cat dns.log | jq -j '.["query"], : selects the query field, which tells us what domain was requested
" ", set a space between output columns
"\n"' puts a newline character at the end of the line
| uniq : remove all duplicate queries
| rev : Takes each query and reverses the string, so that www.google.com becomes moc.elgoog.www. The reason we do this is to strip the query down to the top level domain (TLD), like .com or .net, and the next portion of the domain.
| cut -d '.' -f 1-2 : Split the full query on every period and keep the first and second elements (e.g moc.elgoog.www -> moc.elgoog).
| rev : Reverse the string again to bring it back to normal.
| sort | uniq -c : Remove and count duplicates.
| sort -nr | head : Output the entries with the most duplicates.

Another query we can do on the dns.log log file is the number of unique queries for each DNS query type (i.e. A, NS, MX, TXT, etc.) using the qtype_name field in the dns.log log file:

$ cat dns.log | jq -j '.["qtype_name"], " ", "\n"' | sort | uniq -c | sort -rn

The breakdown of this command is left to reader and can be done using the above breakdowns.

Query the SSL log

The ssl.log log file provides SSL/TLS handshake information. Each record has some valueble fields which we can query. The first field which we query is the server_name which is the URL connected to and we list the 10 most connected URLs:

$ cd /var/zeek/logs/current
$ ssl.log | jq -j '.server_name, " ", "\n"' | sort | uniq -c | sort -rn | head -n 10

Another interesting field is the SSL/TLS version used. To list them with the amounts used we can issue the command:

$ cat ssl.log | jq -j '.version, " ", "\n"' | sort | uniq -c | sort -rn | head -n 10

The breakdown of these commands is left to reader and can be done using the above breakdowns.

Wrap up

This blog post covers some basic queries on some current Zeek logfiles. The same queries can be done on the archived log files of Zeek. In our configuration these are in the /var/zeek/logs directory. The archived log files are in .gz format zo these can be read using zcat or gzcat.

This is just the top of a very big iceberg. Zeek provides much more log files (see below for link to an overview) and capabilities!

In the next, fourth, blog post of this series we will add some capabilities using Zeek Packages!

Resources

Some (other) resources about this subject:

Updated: January 26, 2021

SoCruel.NU