Your log capacity

In this post I want to talk about log capacity. The reason why I want to talk about this is because we noticed that quite a lot of people understand that logs are pretty handy in incident response to figure out what happend but not always have an idea how and what.

A lot of information is produced in your computer system. During an incident response situation, the analyst needs to sift through these log to figure out what happend. Since we live in networked times, this means you got to get these logs from multiple nodes in the network. These can be anything if you want to. To handle this it is important to create a central log server. This makes the attacker his or her life more difficult because now the logs need to be changed at two places.

When setting up a log server one has to take into account that this is traffic over a network, thus you need to make sure that that the protocol used for that log shipping is secure.

A question I sometimes get is what to log, and there the answer is the classic "that depends". Depending on the operating system and the running services and applications the answer depends. The internet is your friend (try log analysis + your subject), but usually is default setting not enough.

Once you are getting a nice amount of data on your log server you might run into storage capacity. One of the important things to know about this is that it used to take more than one year before an organization would discover they got compromised and nowadays it is a bit less than a year.

When you look at an attack campaign like a supply chain model, things have to happen in certain order. Let's say you discover that data is being stolen from your organization, this means that the attacker is at the end of the campaign and if you want to learn about how the bad person got in you got to find it in your logs. When you have only the log capacity of 1 week or 1 month chances are that most information is already gone.

I know disks for things like SANs are not the cheapest things in the world, this means choices have to be made. Depending on your situation a cheaper solution like a NAS or a couple of terabytes of USB/firewire to store offline might be a solution. In this case it is better to have something, than to have nothing because an incident handler can't magically make logs appear.

Facebook and security

I discovered something interesting recently. I knew that Facebook checked if it was a know connection before giving you access to your homepage and otherwise it tells you it is an unknown device. Well in my recent experience they made quite a lot of mistakes in the process.

I made a connection from abroad with my laptop so it was the same device but it triggered the same mechanism. To prove I was myself I had to identify people in pictures they uploaded.

The interesting thing is that some of the people I know have put family pictures up. One series of photos was the dad in one picture, the mom in another one and a daughter in the last one. Looking at the names I could figure out which name I had to select but it was interesting that it was 3 times a different person.

While doing the procedure I thought of how I would attack it and it would be actually quite easy. You just have to do some homework like figuring out family and friends which is not that hard. I guess facial recognition software would be a possibility too.

A simple OpenVPN setup with Zentyal

The other day I needed to come up with a VPN solution for somebody with no networking or IT knowledge whatsoever. The question was of course not formulated like that. The original question was "I need a way to surf the Internet from anywhere in the world and be sure I can do everything that requires security like online banking ect."

I recently ran into Zentyal, a modified Ubuntu, and that looked like an actual tool for this job. So I ran a test the other day and I must say I was pretty impressed by the ease of setup.

The first step was the regular OS install. Once installed logged into the management console which is completely web based which is good because that would take care of the simple part for the person I was building this solution. The good part is that I still have a command line when I need something, it is still a full blown linux.

As a second step I configured the network, gave the machine a static IP address and the IP address of the gateway. Did a ping to www.google.com to test name resolution and network connectivity and it worked so I was up and running for some basic testing.

My first test was to check how the software installation from the web interface worked and I must say it was pretty slick. I installed the ClamAV module as a first test. It installed it, downloaded the latest virus definitions and ran. The next day there was an update for ClamAV on Ubuntu (I know this because CERT.be published it in its advisories). When I got home, I saw it didn't update ... I was not happy of course because. It was my mistake, I found that there is this auto-update which I tested and it works fine.

Then it was time for the real test, setting up the OpenVPN (We worked with dynamic DNS for the OpenVPN server). According to the manual, it looked pretty straight forward. When selecting the package from the inventory it said you also need the certificate authority. After creating the certificates an configuring the VPN with the certificates it was just a click to download the configuration file with the correct certificates in a tar.gz and copying them on the other machine. The files can be produced for Windows, Linux and MacOS X.

I installed Tunnelblick on the Mac, dumped the contents of the tar.gz in the appropriate directory. Last step was to configure the gateway to allow the Tunnelblick to connect to the OpenVPN server and I was ready to run a test. It worked like a charm.

I must say I am pretty impressed because the GUI allowed me to explain everything in a simple way to the end user. How to create other users, track there activities etc. I would recommend it to check it out if you are looking for a solution in a small environment.

Update: I forgot to mention I needed an extra route in my router because the VPN is a different IP range.

May 2013 ISSA-BE Wrap Up

This week there was another ISSA-BE chapter meeting. The whole evening was themed around forensics.

The first talk was given by Sally Trivino and was called Forensics Technology Solutions for Litigation Support.

First topic at hand was "what is evidence". Sally pointed out that two things come into play. The validation of suspicions and the legal facts admissible in court. A little side note I want to make here is that I learned from my legal department that to prosecute somebody, you need to be able to show you suffered damage.

Life would be simple if there was just evidence but there are different kinds of evidence. The first type is rather straight forward, direct evidence. The best example of this is "the smoking gun". You have actual direct proof of what happend and who caused it. Of course this is not the case in computer forensics so you need to find circumstantial evidence. This means that you must correlate different sources to confirm a hypothesis.

For your circumstantial evidence to be admissible in court you have to follow forensic procedures. These are a set of procedures you must follow or you can't build a case. Which forensic tools you can use depends on the jurisdiction. By using scripts, you use a tool that everybody can read thus it is most likely to be acceptable. Standard tools like EnCase, FTK, and SANS SIFT are usually part of the acceptable tools but it is better to check then to be sorry.

As for any craft you will pick the tool based upon the job you need to do. Depending if the data is structured or unstructured, volatile or static, direct or indirect you will need other tools and the field of forensics has a different name.

To explain to the audience how forensics take place Sally showed us as example the EDRM general approach. Some things are obvious other things are less. It is hard to go into detail what she said but there is one thing I think is important to take away from this is that when you write your report you need to write it for non-techies a.k.a "normal" people, not lawyers. It is your job to explain what happend to a judge. To explain that story it is good to have an investigation trail.

As last part of the talk Sally highlighted a couple of best practices in forensics:
- make a safeguard as soon as possible
- make sure you have minimal impact on the system
- look at the different legal aspects
- make sure you have a chain of custody
- make sure you have an evidence trail
- make sure you have pristine copies, only work on copies.
- make sure you have reputable tools
- make sure your files are cryptographically verifiable
- factor in data skew (check against the atomic clock)
- correlate logs.

Finally we briefly touched upon the challenges in the forensics field.

First of all there is wiretapping. In Belgium this is according to the law only allowed for certain purposes by law enforcement. If you thus sniff traffic on your company/private network you might be committing a crime against the privacy laws. I recently assisted a workshop and this was one of the topics addressed during the workshop. From what I remember you can actually sniff but before you do talk to the legal department because the matter is rather complex.

Of course forensics with the whole cloud computing story becomes more complicated. A piece of advise on this was that you have to be sure where your data is because
a) you can't "export" certain types of data
b) the rules of other countries may apply to that data

When you outsource tasks you are still liable for everything, you can't outsource legal responsibility

Finally Sally gave us a heads up on the data breach notification act from the EU. If you are not familiar how it works. It is rather simple the EU makes directives and then the EU countries have a certain period to implement a law about it. On the matter of data breach notification I can recommend ENISA's text about the subject.

The second talk was by Didier Stevens on his network device forensics. Unless you have been living under a rock lately or don't read Didier's blog you must have heard about the new tool that Didier wrote recently.

His research was done on an CISCO ASA device but it is not his intention to limit it to this type of devices. He wanted to figure out what information he forensically could retrieve from a network device.

Like a regular computer system the golden advice nowadays is "do not shutdown". The devices have a small disk but everything runs in memory. It might be a good idea to disconnect the network if possible because the data coming in and out will change the memory.

The first step in forensics Didier made confirmed what Sally said in her talk. You got to have logs, and not just stored locally but centralized with a syslog-like solution. What to log is important too. By default you get some but not all information, so it is recommended that you change that in your configuration.

An example of events to log:

When you connect a laptop or a desktop to a switch is a “switch port state” change. It logs the physical connection and the logical connection. This could be useful since you now have a physical location where the person was when he/she plugged-in.

Some devices have special security features like NAC/NAP or DHCP snooping. When you use this in monitoring mode, it creates a log but no policy is not enforced. This log can then be used in forensic analysis.

Compromising a network device can be done on multiple levels:

A (running) configuration change can be made. Therefore it is important to have configuration and release management. You can dump the running configuration to a file and compair the hashes. You must know what is “running” on a CISCO device and the “written” configuration are not the same. You got to specifically say “store on disk”.

Scripts change the behavior of the device configuration. Again when you use them you must be able to compair them with the script you've put in by doing configuration and release management.

You can compromise the OS image. Didier explained how he manipulated the function that calculates the hash for the image and it always came back that the hashes were ok.

Then we were in for a treat, a little demo of NAFT. Right now only NAFT and CIR are the only open source tools for doing this kinds of forensic analysis as far as Didier is aware. The demo where you see passwords being dumped from the image was pretty cool.

It was an interesting evening, the next ISSA-BE chapter meeting will be in June. Check the website if your are interested in joining.

Reverse engineering some javascript

A couple of days ago I had to reverse engineer some javascript. The classics in the encoding are hexadecimal or decimal encoding but this time the was something else. It looked basically like this:

a = ["3f","15","f4","22","4o","4g","17", ... ]

I changed the actual values to make sure that I don't give anything away on the actual investigation, so yes, if you try it on these values it will not make any sense.

It was clear that the nasty stuff was hidden in that array. At the end of the code I found a function called parseInt that actually interacted with the array. It looked like this:

parseInt(a[i],36)

Again this is altered code, but the real value that was there instead of 36 was hard coded and the i was a variable used in a loop to run over the array. What is interesting is that the loop was written like this:

for(i=0;i-value!=0;i++){...}

The value was hard coded but the classical way of looping which you get thought in programming classes was not used.

I had a look at parseInt() and what it basically does is that it takes a string (the value in the array) and turns it into an integer. But there are things that resemble nothing like integers so something had to be up with that hard coded value 36. 36 represents a radix. The radix represents the numerical system you are working in. If it would have been 16, it would have been hexadecimal, a 10 would represent the decimal system, an 8 the octal system. To make things for a human a bit more complicated our attacker chose 36. The radix parameter has to be a value between 2 and 36.

I wrote some code and made it appear to me as an array of decimal numbers. After this transformation the next step was the String.fromCharCode(). This javascript function transforms the character code to a string. Thus i ordered my system to do that and transformed my array into characters.

When I made the program print the characters one after the other the content of the obfuscated malicious javascript code revealed itself and I could go on with the investigation.

Blocking Phishing Sites

The other day I got the question how the popular browsers block you from going to malicious websites. Interesting, that you had to connect back to a database seems logical but the actual inner workings were a mystery to me so I looked it up.

Firefox, Safari and Chrome all use the same technology called the Google Safe Browsing API. Microsoft Internet Explorer uses a technology called SmartScreen Filter.

Google Safe Browsing API is a complete website with the technical details. Basically it works like this:

You type in the website you want to visit in your browser:
http://www.somehost.com/path/page.html?args

Your browser will need to check the following paths against the database:

http://www.somehost.com/path/page.html?args

http://www.somehost.com/path/page.html

http://www.somehost.com/path/

http://www.somehost.com/

http://somehost.com/path/page.html?args

http://somehost.com/path/page.html

http://somehost.com/path/

http://somehost.com/

As you can see this is quite a lot of strings and looking up strings in a database is usually a slow thing. The trick used here is that a hash is calculated and a 4-byte prefix is sent to the database. In case of a match the database will return all the matches and the client can then calculate the full hash to see if it is in the returned list. 

If the full hash matches the end user is informed about this else your page gets loaded.

As sources for their database they mention Antiphishing.org en Stopbadware.org but you can be pretty sure that it would not be the only sources.

At the Windows Live blog I could find more information on how the Microsoft SmartScreen Filter works. The example Microsoft gives is the following:

Let's say a malicious website is hosted at canada-pharmacy.us. This IP gets marked in the database as "bad", besides the URL the IP address is marked as "bad" too. SmartScreen will generalize this to IP's in the neighborhood. This is done based on ASN blocks, the way IP addresses are split up by owner.

DNS server rating are also part of the SmartScreen technology. The DNS servers that seems to know just a little too much about abusive domains is given a lower rating according to the blog. Unfortunately for a techie this is a meaningless description.

SmartScreens telemetry comes from reports from end users, third parties, traffic from URLs showing up in e-mails and logs. The feeds are fed into machine learning algorithms to either flag or pass a URL. When the algorithm is in doubt, the information is given to an analyst who will do the necessary research.

As a conclusion I would say make sure you have a browser that has such a technology enabled, it is not perfect but it is free and better then having nothing.

Changing your SSH port and configuring SVN

A while ago I wrote an post on SSH. Yesterday I was discussing the brute force attacks you get and my conversation partner said that he now systematically configures SSH on another port and rarely sees any attacks.

Just for fun tonight I've switched my port 22 to another port and will monitor my logs to see if I get the same findings. What I actually did is alter the port forwarding scheme so that I keep the config of the server rather standard. I expect to see the same thing as my conversation partner.

After the change I had to figure out how this works out with SVN which I do with svn+ssh. Without the necessary modifications that fails of course since port 22 is not open anymore.

On the server side you don't have to change anything. On the client side there is in ~/.subversion/config in the section [tunnels] a line to add:
sshtunnel = ssh -p port_number -q

You can put in any name instead of sshtunnel but I personally like clear naming for when I am tired. The -q at the end is important otherwise you will get the message "Killed by signal 15."

A last hurdle was that I had already a number of check out projects and created a little problem. I simply renamed the directory, downloaded a new copy this time using
svn co svn+sshtunnel://server/path/dir dir

To merge the contents of both (the old one and the new one) without overwriting the new svn config it was simply
cp -R -n old_dir new_dir
rm -R -f old_dir