Internet traffic real-time detection using Arch Linux

So the last post got off to a bit of a rocky start.  After doing a bit more research, I discovered that the Snort IDS system was very finite.  There needed to be better integration.  That is where Suricata comes in.  Suricata is meant as a replacement for Snort in certain situations.

Unfortunately, Suricata doesn't appreciate all of Snort's rules.  This can quickly go from frustrating to something else.  Thankfully, many basic commands allow to ease the situation down to a few simple commands.

Let's take a look at what's actually going on in the logs:

[10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php lastname ASCII"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"lastname="; nocase; uricontent:"SELECT"; nocase; pcre:"/ASCII\(.+SELECT/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006007; classtype:web-application-attack; sid:2006007; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2273       [10392] 27/6/2015 -- 16:12:37 - (detect-reference.c:159) <Error> (DetectReferenceParse) -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "cve". Supported keys are defined in reference.config file.  Please have a look at the conf param "reference-config-file"                                                                                                                                                                                                     [10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php lastname UPDATE"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"lastname="; nocase; uricontent:"UPDATE"; nocase; pcre:"/UPDATE.+SET/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006008; classtype:web-application-attack; sid:2006008; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2274          [10392] 27/6/2015 -- 16:12:37 - (detect-reference.c:159) <Error> (DetectReferenceParse) -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "cve". Supported keys are defined in reference.config file.  Please have a look at the conf param "reference-config-file"                                                                                                                                                                                                     [10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php firstname SELECT"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"firstname="; nocase; uricontent:"SELECT"; nocase; pcre:"/SELECT.+FROM/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006009; classtype:web-application-attack; sid:2006009; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2275       

Here, I've pulled 3 rejections out of a list of over 10,000.


Yes, over 10,000.

Here is where some basic commands and simple math come into play in order to take a problem the size of Mount Everest and bring it down to the size of a grain of sand.  You can easily notice that each rule being rejected has its own SID.  An SID is an identification token used to identify a specific rule.  Each rule must have an SID for it to be considered by the program.  The problem comes that many of such rules from Snort are not properly formatted to handle Suricata's system.  I am not sure why this is the case, so while it would be proper to fix the rules, it is simpler for the time being to just move the offending rules out of the way.  We're not actually moving them out of the way, but they are being ignored so that the rest of the program may function as it is supposed to.  Once the program is able to run, then fixing the broken rules can begin and things can still run smoothly.  However, there has to be a clear understanding that this is what is going to happen, as many rules are absolutely critical to things running in the desired direction.  Therefore, getting those other rules fixed ASAP will become the next priority, rather than just leaving them to rot in an inefficient and disrespectful manner.

From the Snort manual:

This example is a rule with the Snort Rule ID of 1000983.

    alert tcp any any -> any 80 (content:"BOB"; sid:1000983; rev:1;)

 So here we can see that there is some adherence to the basic format.  Again, getting down to the root cause will be left for another time.  First of all, let's find out just how may offending rules we have:

[root@mebion rules]# grep sid /var/log/suricata.log | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g' | wc -l                                                                                                  11511                                                                                                                                                                                                                                       

Well now, 11,511 bad rules.  It probably could have been performed a bit more elegantly, but drawing it out to this degree appears to work just perfectly.  Next, we need to know how many rules there are altogether, so that way we can be sure that we aren't throwing ourselves under the bus.  After all, what if there are only 11,511 rules total?

 [root@mebion rules]# grep sid *.rules | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g'  | wc -l                                                                                                               46520                                                                                                                                                                                                                                        

Okay, 46,520 rules total as of this time.  Now, math:

[root@mebion rules]# bc

 bc 1.06.95 Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
 This is free software with ABSOLUTELY NO WARRANTY.
 For details type `warranty'.
 46520-11511
 35009                                                                                                                                                                                                                  
Good, we've got 35,009 good rules to work with.  11,511 rules doesn't seem so significant now, does it?

Now, according to the documentation, it is possible to ignore those rules:

If you disable a rule in your rule file by putting a # in front of it, it will be enabled again the next time you run Oinkmaster. You can disable it through Oinkmaster instead, by entering the following:

cd /etc/suricata/rules

and find the sid of the rule(s) you want to disable.

Subsequently enter:

sudo nano /etc/oinkmaster.conf

and go all the way to the end of the file.
Type there:

disablesid 2010495

Instead of 2010495, type the sid of the rule you would like to disable. It is also possible to disable multiple rules, by entering their sids separated by a comma.

 Here is where it gets interesting.  We can put the list of all the SID's into a file:

grep sid /var/log/suricata.log | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g' > /etc/suricata/bad-sids.txt

 Which, of course, gives us each SID on each line.  That's not separated by a comma.  Therefore, we need to reformat.

cat bad-sids.txt | tr '\n' , > fixed-sids.txt

 Okay, now that they have been properly formatted, they can simply be copy & pasted into the configuration file all on one line.  This solves that problem, enjoy!