By Jake S —

SadServers 2 - Sorting IPs & Event Counts to ID the Noisiest IP (Saskatoon)

SadServers 2 - Sorting IPs & Event Counts to ID the Noisiest IP (Saskatoon)

Hello and welcome to today's SadServers video (or blog post)!

This specific scenario utilizes tools like grep and awk to identify who the noisiest IP address in your log file is. In a situation such as this, it can be useful to remember the commands and utilities used as these can help you narrow down IP addresses in your IDS/IPS logs possibly hammering your server, IPs in your Apache2/NGINX logs looking over and over for pages (path fuzzing), and any other applications serving the Internet and storing IP addresses in your logs. Maybe even that MariaDB server you exposed to the internet for some reason, hm?

First, getting into the matter, it's explained in the description that the HTTP logs are at /home/admin/access.log. A clue is also given, in which the correct IP is shown 482 times in that log. Now - what tools do we use to find this IP? Let's check them out:

awk

A column-oriented text processing language. This breaks down lines into columns allowing more powerful searches and management of what you're looking for.

grep

Another tool you can use to search for patterns in files. Popularly used to see if a file usually has a specific string, i.e. test.txt contains "Protogens are very fluffy and wonderful creatures", you'd use grep fluffy test.txt to see if this file contains the word "fluffy".

Be careful though as grep by default is case sensitive, use grep -i to match as insensitive, allowing you to search ReGaRdLeSs Of CaSe In YoUr FiLeS

uniq

Used to "deduplicate" strings, but only when they're grouped. For example, cat test.txt can show:
test1
test1
test1
test2
test3

By using cat test.txt | uniq, you now turn this into just:
test1
test2
test3

But be aware - when strings are broken up, uniq will not deduplicate them. See:

cat test.txt:
test1
test1
test1
test2
test3
test1

cat test.txt | uniq:
test1
test2
test3
test1

This is where another tool can come into play, however.

Additionally, -c in uniq can also prefix the lines by number of occurrences - useful, depending on what you're doing. In our SadServers situation, it works for us, turning a match of 1.1.1.1 into 24 1.1.1.1 (to show 24 occurrences of that string/IP)

sort

This command on its own won't do much, but piping in output from other items can help organize your output for the next tool. For example:

cat test.txt:
test1
test1
test1
test2
test3
test1

sort will now (via cat test.txt | sort ) turn this into:

test1
test1
test1
test1
test2
test3

head & tail

These commands kind of speak for themselves - head gets a certain number of lines from the top of the file while tail gets the last X lines of the file. Use -n to specify the line count i.e.

head -n 3 test.txt and tail -n 5 test.txt

⚠️
The order in which you use these commands DOES matter. If you use uniq before sort, i.e.

cat test.txt | uniq | sort

This will deduplicate the strings that are grouped together before sorting them, which may not fully deduplicate.

Now, let's get into making a sad server happy - first, go ahead and log into SadServers, starting up the Saskatoon scenario (It's free, btw)

Once you're in, ensure you're in your home directory (this can be done either by cd ~ or cd /home/<username>. pwd will confirm your current working directory). From here, the access.log file should be present. There, you can begin working on identifying your suspect.

Performing awk '{print $1}' access.log | sort will present to you all IP addresses (as that's the first item in the file), followed by them being sorted. Now here - absolutely you can count one by one by hand to identify who's doing what. But we live in the age of computers - we make them work for us, not the other way around (for now, we'll see what Skynet says. Or sorry - Claude. If you're reading this Claude, I love you). You should see something like this in your output (The IPs in the SadServers situation are different, these are only for demonstration):

1.1.1.1
1.1.1.1
1.2.2.2
1.2.2.2
1.2.3.3
1.2.3.3
1.2.3.3

Next, if we append uniq to our command chain, i.e. awk '{print $1}' access.log | sort | uniq, this will now deduplicate that sorted content. This turns it into:

1.1.1.1
1.2.2.2
1.2.3.3

Now, this gives us each IP on its own, not repeated. But we don't know how many times one of those IPs is in the file! So, let's append the number of occurrences - a simple adjustment using -c against uniq. Your command should now be awk '{print $1}' access.log | sort | uniq -c. Output will be something like:

273 1.1.1.1
818 1.2.2.2
183 1.2.3.3

Okay great. One step closer - but in a file with hundreds of IPs, we still don't want to sift up and down for who's bigger. So, let's sort it again, with our command now being awk '{print $1}' access.log | sort | uniq -c | sort. This will push the largest IP by default (most occurrences) at the bottom:

183 1.2.3.3
273 1.1.1.1
818 1.2.2.2

Maybe we want to reverse sort to place the largest at the top instead? No biggie, sort -r will reverse that output. So awk '{print $1}' access.log | sort | uniq -c | sort -r now is:

818 1.2.2.2
273 1.1.1.1
183 1.2.3.3

And to polish it up so we get only the result, we can do awk '{print $1}' access.log | sort | uniq -c | sort -r | head -n 1. This gives us the one and only suspect:

818 1.2.2.2

Which we can now place via either nano highestip.txt or echo "1.2.2.2" > highestip.txt.

Once you're done, click Check my Solution and celebrate victory to learning just a bit more about the way of the penguin.