Internet traffic real-time detection using Arch Linux

So the last post got off to a bit of a rocky start.  After doing a bit more research, I discovered that the Snort IDS system was very finite.  There needed to be better integration.  That is where Suricata comes in.  Suricata is meant as a replacement for Snort in certain situations.

Unfortunately, Suricata doesn't appreciate all of Snort's rules.  This can quickly go from frustrating to something else.  Thankfully, many basic commands allow to ease the situation down to a few simple commands.

Let's take a look at what's actually going on in the logs:

[10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php lastname ASCII"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"lastname="; nocase; uricontent:"SELECT"; nocase; pcre:"/ASCII\(.+SELECT/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006007; classtype:web-application-attack; sid:2006007; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2273       [10392] 27/6/2015 -- 16:12:37 - (detect-reference.c:159) <Error> (DetectReferenceParse) -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "cve". Supported keys are defined in reference.config file.  Please have a look at the conf param "reference-config-file"                                                                                                                                                                                                     [10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php lastname UPDATE"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"lastname="; nocase; uricontent:"UPDATE"; nocase; pcre:"/UPDATE.+SET/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006008; classtype:web-application-attack; sid:2006008; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2274          [10392] 27/6/2015 -- 16:12:37 - (detect-reference.c:159) <Error> (DetectReferenceParse) -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "cve". Supported keys are defined in reference.config file.  Please have a look at the conf param "reference-config-file"                                                                                                                                                                                                     [10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php firstname SELECT"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"firstname="; nocase; uricontent:"SELECT"; nocase; pcre:"/SELECT.+FROM/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006009; classtype:web-application-attack; sid:2006009; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2275       

Here, I've pulled 3 rejections out of a list of over 10,000.


Yes, over 10,000.

Here is where some basic commands and simple math come into play in order to take a problem the size of Mount Everest and bring it down to the size of a grain of sand.  You can easily notice that each rule being rejected has its own SID.  An SID is an identification token used to identify a specific rule.  Each rule must have an SID for it to be considered by the program.  The problem comes that many of such rules from Snort are not properly formatted to handle Suricata's system.  I am not sure why this is the case, so while it would be proper to fix the rules, it is simpler for the time being to just move the offending rules out of the way.  We're not actually moving them out of the way, but they are being ignored so that the rest of the program may function as it is supposed to.  Once the program is able to run, then fixing the broken rules can begin and things can still run smoothly.  However, there has to be a clear understanding that this is what is going to happen, as many rules are absolutely critical to things running in the desired direction.  Therefore, getting those other rules fixed ASAP will become the next priority, rather than just leaving them to rot in an inefficient and disrespectful manner.

From the Snort manual:

This example is a rule with the Snort Rule ID of 1000983.

    alert tcp any any -> any 80 (content:"BOB"; sid:1000983; rev:1;)

 So here we can see that there is some adherence to the basic format.  Again, getting down to the root cause will be left for another time.  First of all, let's find out just how may offending rules we have:

[root@mebion rules]# grep sid /var/log/suricata.log | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g' | wc -l                                                                                                  11511                                                                                                                                                                                                                                       

Well now, 11,511 bad rules.  It probably could have been performed a bit more elegantly, but drawing it out to this degree appears to work just perfectly.  Next, we need to know how many rules there are altogether, so that way we can be sure that we aren't throwing ourselves under the bus.  After all, what if there are only 11,511 rules total?

 [root@mebion rules]# grep sid *.rules | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g'  | wc -l                                                                                                               46520                                                                                                                                                                                                                                        

Okay, 46,520 rules total as of this time.  Now, math:

[root@mebion rules]# bc

 bc 1.06.95 Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
 This is free software with ABSOLUTELY NO WARRANTY.
 For details type `warranty'.
 46520-11511
 35009                                                                                                                                                                                                                  
Good, we've got 35,009 good rules to work with.  11,511 rules doesn't seem so significant now, does it?

Now, according to the documentation, it is possible to ignore those rules:

If you disable a rule in your rule file by putting a # in front of it, it will be enabled again the next time you run Oinkmaster. You can disable it through Oinkmaster instead, by entering the following:

cd /etc/suricata/rules

and find the sid of the rule(s) you want to disable.

Subsequently enter:

sudo nano /etc/oinkmaster.conf

and go all the way to the end of the file.
Type there:

disablesid 2010495

Instead of 2010495, type the sid of the rule you would like to disable. It is also possible to disable multiple rules, by entering their sids separated by a comma.

 Here is where it gets interesting.  We can put the list of all the SID's into a file:

grep sid /var/log/suricata.log | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g' > /etc/suricata/bad-sids.txt

 Which, of course, gives us each SID on each line.  That's not separated by a comma.  Therefore, we need to reformat.

cat bad-sids.txt | tr '\n' , > fixed-sids.txt

 Okay, now that they have been properly formatted, they can simply be copy & pasted into the configuration file all on one line.  This solves that problem, enjoy!

Firewall with the Raspberry Pi

Figuring out what kind of firewall works best for your situation is never a sweatless task.  For a[n] [un]lucky few, it's a no-brainer.  However, in my case I decided that a firewall needs to do more than just block a few ports.  Nowadays, with the advent of malware and hackers getting into systems via web browsers, it's easier than ever for identity thieves to strike.  So, we need something that will actually work by not only blocking, but alerting us to violations that slip past through ports 80 and 443.  We need a way to identify web traffic, as it can pass through any port.  We also need to determine if the information passing through it is of a certain nature.

If little Johnny is researching his school project, it wouldn't be very productive if he got distracted by visiting other websites.  Indeed, a parent walking in on a situation should have some foresight to what's really going on.  These days, teenagers have more than enough stress to worry about.  The less stressed the parents are, the easier it is on the kids.  But, we also want to make sure that they know who is in charge.  Kids should be able to feel a form of freedom to explore and express themselves.  It is easy for that to become a  problem.  Even if the problem has already occurred, it's not too late to get a remedy in place.  Being pro-active is one of the hardest things a parent can do, since the situation can go sideways in a split second.

So, in my previous blog post, I introduced the concept of using a Raspberry Pi computer to act as a firewall.  Many iterations of this have been successfully implemented.  One of my favorites is called the PiWall.

EDIT: It turned out the instructable was just a copy & paste ripoff of the original blog, and so won't be linked to any further.

In this case, I've made some modifications to the base system.  Since I use flash drives in lieu of hard drives in order to save energy and produce less heat, I've replaced all instances of the Extended Filesystem with Samsung's Flash Friendly Filesystem, or F2FS.  Furthermore, some easy filesystem performance tests prove beyond a shadow of a doubt of Samsung's commitment to the Open Source Community.  In practice, some sequences that once produced lag in EXT4 now provide results instantaneously.  Your Mileage May Vary (YMMV), but overall I am happy with the results.

Until the next update...

Roll the updates

In the coming months, stay tuned for a series regarding traffic monitoring and some detail to creating a system that generates near real-time alerts.

The start of this project began with a simple Linux installation on a Raspberry Pi.  Instead of listing everything out here, I will simply link on to a post that better describes the process of that setup here.

Once that step is finished, the next step will be ready.  Stay tuned this summer for more!