Time drift in Arch Linux

Have you ever looked at the clock on your computer, then looked at the clock on your mobile device and wondered why they differed?  The easy answer is time drift.  The realtime clock (RTC) in your computer, as well as mobile devices like smartphones, tablets, laptops, watches, etc operates on its own timeframe.  It does not conform to any universal standard.  This is why utilities such as NTP and timedatectl were created.

The timedatectl utility is specifically designed for Arch Linux as an easy-to-use utility for the Network Time Protocol (NTP), but NTP is generally controlled a bit more directly in Ubuntu, Redhat and other distributions.  I'm going to focus on timedatectl for this particular entry.

First, open up a terminal.

Next, let's get an idea of what's currently going on with the time and date situation.  Just run the following command:

timedatectl status

This tells you the various times, which timezone you're set to, whether the network time is on, if NTP is turned on and whether or not the RTC is set to the local TimeZone.  Since we want to correct for time drift, let's get NTP synchronized.  In order to do that, we need to turn Network time on.  Let's start by checking out our options.

There is a set-ntp option.  It states that it needs a BOOL, or boolean, which can be one of two different entries.  In this case, 'yes' and 'no' are a pair of very human conditions that we can work with.  It is important to perform this function as a standard user.  It will ask you to authenticate.  This is for security purposes.

Once that is completed, you can check to see the current status.

Now, the Network time is set to 'on', and all three clocks are set to the current time.  This should reflect from other devices as well.

By maintaining the current time on your desktop or portable computer, you can avoid delays.  Happy time!

Compilation of OpenCV programs

Compiling programs written in C++ is usually a fairly easy and non-complex task to complete.

% g++ -o program program.cpp

However, OpenCV (Open Source Computer Vision) employs a much more complex set of libraries that are necessary to reference object compilation.  While a good amount of documentation exists to manipulate and generate a useful program to generate desired outcomes, some of it remains an exercise to the programmer to know how to get it setup and running.  Just about anyone that has that level of knowledge can their code to work just fine.  However, sometimes the obvious slips by.

So, you're working on your program and it's based off of some known, good working code.  You set your parameters and hit the compilation string.

/tmp/ccSv9hPY.o: In function `MatchingMethod(int, void*)':

/home/user/Project/strimaker/template/MatchTemplate_Demo.cpp:87: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'

/home/user/Project/strimaker/template/MatchTemplate_Demo.cpp:88: undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'

/tmp/ccSv9hPY.o: In function `main':

/home/user/Project/strimaker/template/MatchTemplate_Demo.cpp:37: undefined reference to `cv::namedWindow(cv::String const&, int)'

/home/user/Project/strimaker/template/MatchTemplate_Demo.cpp:38: undefined reference to `cv::namedWindow(cv::String const&, int)'

/home/user/Project/strimaker/template/MatchTemplate_Demo.cpp:42: undefined reference to `cv::createTrackbar(cv::String const&, cv::String const&, int*, int, void (*)(int, void*), void*)'

/home/user/Project/strimaker/template/MatchTemplate_Demo.cpp:46: undefined reference to `cv::waitKey(int)'

collect2: error: ld returned 1 exit status

Now you've got a slew of problems and a non-existent binary.

Linking the libraries is fairly simple, as long as you know what to link against.

 g++ -o MatchTemplate_Demo MatchTemplate_Demo.cpp -g -O2 -lopencv_imgcodecs -lopencv_imgproc -lopencv_core -lopencv_highgui

Here, I'm linking against anything the headers require.  One way to figure out what libraries to use is through the use of the Autotools program.  Unfortunately, OpenCV relies on CMake, and so one must be able to distinguish between them in order to avoid wasting time on a problem that may not need to exist.

Getting MRTG to work in ArchLinux

It's been a long time since I ran MRTG, the Multi Router Traffic Grapher.  So, I figured it was time to run it again.  Using Arch Linux, it's pretty simple to get it installed and configured.  However, the current model does not come with a mrtg.service file.

In fact, no one has written one!  It's supposed to be ran by hand.  This kind of dulls things, since a server ought to be able to be brought back online and it should "just work" with little human intervention.  So, I'm a bit of a perfectionist when it comes to making things work just right, out of the box.

Creating service files for systemd is not very difficult.  I just used another .service file as a template to work from.

The only other type of service file that I could find was a CentOS oriented control file, and by the looks of it, someone didn't realize that you could run MRTG in daemon mode and not have to worry about rerunning the process every so often.  Still, it was helpful to remember that daemonizing isn't always the necessary way.

[Unit]



Description=Multi Router Traffic Grapher



After=network.target




[Service]



PIDFile=/run/mrtg/mrtg.pid



User=mrtg



Group=mrtg



ExecStart=/usr/lib/systemd/scripts/mrtg.sh



ExecReload=/usr/bin/kill -USR2 $MAINPID



KillSignal=SIGQUIT



KillMode=mixed






[Install]



WantedBy=multi-user.target

You'll notice that I generated a shell script to handle actually starting the daemon.  I was getting errors while trying to load an environment variable into it, so I set it to run its own process:

#!/bin/sh
LANG=C /usr/bin/mrtg --pid-file=/run/mrtg/mrtg.pid --user=mrtg --group=mrtg --daemon /srv/http/mrtg/html/mrtg.cfg

Create /run/mrtg:

# mkdir /run/mrtg

Set the appropriate permissions:

# chown mrtg.mrtg /run/mrtg

When all is said and done, MRTG will run in the background, unattended, and produce beautiful graphs in Arch Linux.

Internet traffic real-time detection using Arch Linux

So the last post got off to a bit of a rocky start.  After doing a bit more research, I discovered that the Snort IDS system was very finite.  There needed to be better integration.  That is where Suricata comes in.  Suricata is meant as a replacement for Snort in certain situations.

Unfortunately, Suricata doesn't appreciate all of Snort's rules.  This can quickly go from frustrating to something else.  Thankfully, many basic commands allow to ease the situation down to a few simple commands.

Let's take a look at what's actually going on in the logs:

[10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php lastname ASCII"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"lastname="; nocase; uricontent:"SELECT"; nocase; pcre:"/ASCII\(.+SELECT/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006007; classtype:web-application-attack; sid:2006007; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2273       [10392] 27/6/2015 -- 16:12:37 - (detect-reference.c:159) <Error> (DetectReferenceParse) -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "cve". Supported keys are defined in reference.config file.  Please have a look at the conf param "reference-config-file"                                                                                                                                                                                                     [10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php lastname UPDATE"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"lastname="; nocase; uricontent:"UPDATE"; nocase; pcre:"/UPDATE.+SET/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006008; classtype:web-application-attack; sid:2006008; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2274          [10392] 27/6/2015 -- 16:12:37 - (detect-reference.c:159) <Error> (DetectReferenceParse) -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "cve". Supported keys are defined in reference.config file.  Please have a look at the conf param "reference-config-file"                                                                                                                                                                                                     [10392] 27/6/2015 -- 16:12:37 - (detect.c:357) <Error> (DetectLoadSigFile) -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS The Address Book SQL Injection Attempt -- user.php firstname SELECT"; flow:established,to_server; uricontent:"/user.php?"; nocase; uricontent:"firstname="; nocase; uricontent:"SELECT"; nocase; pcre:"/SELECT.+FROM/Ui"; reference:cve,CVE-2006-4575; reference:url,www.securityfocus.com/bid/21870; reference:url,doc.emergingthreats.net/2006009; classtype:web-application-attack; sid:2006009; rev:5;)" from file /etc/suricata/rules/ET-emerging-web_specific_apps.rules at line 2275       

Here, I've pulled 3 rejections out of a list of over 10,000.


Yes, over 10,000.

Here is where some basic commands and simple math come into play in order to take a problem the size of Mount Everest and bring it down to the size of a grain of sand.  You can easily notice that each rule being rejected has its own SID.  An SID is an identification token used to identify a specific rule.  Each rule must have an SID for it to be considered by the program.  The problem comes that many of such rules from Snort are not properly formatted to handle Suricata's system.  I am not sure why this is the case, so while it would be proper to fix the rules, it is simpler for the time being to just move the offending rules out of the way.  We're not actually moving them out of the way, but they are being ignored so that the rest of the program may function as it is supposed to.  Once the program is able to run, then fixing the broken rules can begin and things can still run smoothly.  However, there has to be a clear understanding that this is what is going to happen, as many rules are absolutely critical to things running in the desired direction.  Therefore, getting those other rules fixed ASAP will become the next priority, rather than just leaving them to rot in an inefficient and disrespectful manner.

From the Snort manual:

This example is a rule with the Snort Rule ID of 1000983.

    alert tcp any any -> any 80 (content:"BOB"; sid:1000983; rev:1;)

 So here we can see that there is some adherence to the basic format.  Again, getting down to the root cause will be left for another time.  First of all, let's find out just how may offending rules we have:

[root@mebion rules]# grep sid /var/log/suricata.log | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g' | wc -l                                                                                                  11511                                                                                                                                                                                                                                       

Well now, 11,511 bad rules.  It probably could have been performed a bit more elegantly, but drawing it out to this degree appears to work just perfectly.  Next, we need to know how many rules there are altogether, so that way we can be sure that we aren't throwing ourselves under the bus.  After all, what if there are only 11,511 rules total?

 [root@mebion rules]# grep sid *.rules | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g'  | wc -l                                                                                                               46520                                                                                                                                                                                                                                        

Okay, 46,520 rules total as of this time.  Now, math:

[root@mebion rules]# bc

 bc 1.06.95 Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
 This is free software with ABSOLUTELY NO WARRANTY.
 For details type `warranty'.
 46520-11511
 35009                                                                                                                                                                                                                  
Good, we've got 35,009 good rules to work with.  11,511 rules doesn't seem so significant now, does it?

Now, according to the documentation, it is possible to ignore those rules:

If you disable a rule in your rule file by putting a # in front of it, it will be enabled again the next time you run Oinkmaster. You can disable it through Oinkmaster instead, by entering the following:

cd /etc/suricata/rules

and find the sid of the rule(s) you want to disable.

Subsequently enter:

sudo nano /etc/oinkmaster.conf

and go all the way to the end of the file.
Type there:

disablesid 2010495

Instead of 2010495, type the sid of the rule you would like to disable. It is also possible to disable multiple rules, by entering their sids separated by a comma.

 Here is where it gets interesting.  We can put the list of all the SID's into a file:

grep sid /var/log/suricata.log | sed -e 's/.*sid//g' | sed -e 's/rev.*//g' | sed -e 's/://g' | sed -e 's/;//g' > /etc/suricata/bad-sids.txt

 Which, of course, gives us each SID on each line.  That's not separated by a comma.  Therefore, we need to reformat.

cat bad-sids.txt | tr '\n' , > fixed-sids.txt

 Okay, now that they have been properly formatted, they can simply be copy & pasted into the configuration file all on one line.  This solves that problem, enjoy!

Firewall with the Raspberry Pi

Figuring out what kind of firewall works best for your situation is never a sweatless task.  For a[n] [un]lucky few, it's a no-brainer.  However, in my case I decided that a firewall needs to do more than just block a few ports.  Nowadays, with the advent of malware and hackers getting into systems via web browsers, it's easier than ever for identity thieves to strike.  So, we need something that will actually work by not only blocking, but alerting us to violations that slip past through ports 80 and 443.  We need a way to identify web traffic, as it can pass through any port.  We also need to determine if the information passing through it is of a certain nature.

If little Johnny is researching his school project, it wouldn't be very productive if he got distracted by visiting other websites.  Indeed, a parent walking in on a situation should have some foresight to what's really going on.  These days, teenagers have more than enough stress to worry about.  The less stressed the parents are, the easier it is on the kids.  But, we also want to make sure that they know who is in charge.  Kids should be able to feel a form of freedom to explore and express themselves.  It is easy for that to become a  problem.  Even if the problem has already occurred, it's not too late to get a remedy in place.  Being pro-active is one of the hardest things a parent can do, since the situation can go sideways in a split second.

So, in my previous blog post, I introduced the concept of using a Raspberry Pi computer to act as a firewall.  Many iterations of this have been successfully implemented.  One of my favorites is called the PiWall.

EDIT: It turned out the instructable was just a copy & paste ripoff of the original blog, and so won't be linked to any further.

In this case, I've made some modifications to the base system.  Since I use flash drives in lieu of hard drives in order to save energy and produce less heat, I've replaced all instances of the Extended Filesystem with Samsung's Flash Friendly Filesystem, or F2FS.  Furthermore, some easy filesystem performance tests prove beyond a shadow of a doubt of Samsung's commitment to the Open Source Community.  In practice, some sequences that once produced lag in EXT4 now provide results instantaneously.  Your Mileage May Vary (YMMV), but overall I am happy with the results.

Until the next update...

Roll the updates

In the coming months, stay tuned for a series regarding traffic monitoring and some detail to creating a system that generates near real-time alerts.

The start of this project began with a simple Linux installation on a Raspberry Pi.  Instead of listing everything out here, I will simply link on to a post that better describes the process of that setup here.

Once that step is finished, the next step will be ready.  Stay tuned this summer for more!

Google Now woes

Do you own an Android powered tablet?  If it's running the latest KitKat, perhaps you've also come across Google's new "Google Now" app that's being advertised.  In theory, I think it has promise.  However, at this time it doesn't really do much to enhance my usage other than to increase my frustration level. 

As a result, I uninstalled it.

This left my icons in a weird state: Most were still there, but many had been replaced with a generic green Android icon.  Some labels had also been stripped away so only the process name that the underlying Linux system knows them as was showing (Such as com.zygote or com.motorola.something!).  How to bring them all back without going through each one?!  Shut the tablet off.  Yup, then turn it back on.  After I did so, my tablet was back to its original setup that I had made for it.

I'm sure someone will come across the same situation, so there you have it!