Sysmon ... digging for gold

When things are bizarre, weird and strange often people come and see their incident response team. This incident wasn't different some process wasn't doing what the admin was expecting it would do but he didn't knew what it was doing.

He knows I got a nice bag of little tools and thus I introduced the sysadmin to sysmon. I would recommend to install on each and every Windows system. It logs much more than what a standard windows system logs and is thus a treasure chest for any incident responder.

Download
You can download the 32-bit and the 64-bit version from sysinternals. I prefer to make my sysinternals tools from http://live.sysinternals.com.

Installation
The installation is pretty straight forward. You open a command prompt with Administrator privileges and go to the directory where you've downloaded sysmon. I will reference during the rest of this post to sysmon.exe depending on your platform you will need to reference the 32-bit or 64-bit version.

To install it run sysmon.exe -i --accepteula. This outputs

System Monitor v4.1 - System activity monitor
Copyright (C) 2014-2016 Mark Russinovich and Thomas Garnier
Sysinternals - www.sysinternals.com

Sysmon installed.
SysmonDrv installed.
Starting SysmonDrv.
SysmonDrv started.
Starting Sysmon..
Sysmon started.

Configuration
Software needs to be configured. I like my logs verbose so lets go over the the options:

-c   Update configuration of an installed Sysmon driver or dump the current configuration if no other argument is provided. Optionally take a configuration file.
-h   Specify the hash algorithms used for image identification (default is SHA1). It supports multiple algorithms at the same time. Configuration entry: HashAlgorithms.
-i   Install service and driver. Optionally take a configuration file.
-l   Log loading of modules. Optionally take a list of processes to track.
-m   Install the event manifest (done on service install as well).
-n   Log network connections. Optionally take a list of processes to track.
-r   Check for signature certificate revocation. Configuration entry: CheckRevocation.
-u   Uninstall service and driver.


I configure my systems the following way:
sysmon -c -l -n -r

I like my hash to be sha1 because that makes it easy to submit to websites like virustotal.

The Logs
You can find the logs created by sysmon in the event viewer (you need administrative privileges).

1. Open the event viewer
2. Go to Applications and Services logs
3. Go to Microsoft
4. Go to Windows
5. Go to Sysmon
6. Go to Operational

Remember that it is a good practice to split off your event logs to a separate disk if the I/O is a bottle neck. When you right click on operational and request the properties you can change the log path and the log size. Since I like verbose logs I've set mine to at least 250 MB (249984 KB) and cyclical.

Now that everything is configured it is time to restart the service. Open a powershell prompt with elevated privileges and do:

restart-service sysmon

Digging for Gold
The last step to figure out what is going on is of course log analysis. There are a couple of event IDs

EventID 1 shows you process creation
Process Create:
UtcTime: 2016-08-01 14:24:12.390
ProcessGuid: {ddfd1a0f-5b8c-579f-0000-0010f4d2d004}
ProcessId: 7204
Image: C:\Windows\System32\mmc.exe
CommandLine: "C:\WINDOWS\system32\mmc.exe" "C:\WINDOWS\system32\eventvwr.msc" /s
CurrentDirectory: C:\WINDOWS\system32\
User:
LogonGuid: {---}
LogonId: 0x4d0c45f
TerminalSessionId: 1
IntegrityLevel: High
Hashes: SHA1=F5DC12D658402900A2B01AF2F018D113619B96B8
ParentProcessGuid: {ddfd1a0f-62f2-579c-0000-0010f1060400}
ParentProcessId: 2940
ParentImage: C:\Windows\explorer.exe
ParentCommandLine: C:\WINDOWS\Explorer.EXE


Event ID 2 shows you when a file was created


File creation time changed:
UtcTime: 2016-08-01 14:24:22.358
ProcessGuid: {ddfd1a0f-3a92-579f-0000-0010c31a2804}
ProcessId: 2996
Image: C:\Users\\Desktop\portable\firefox\FirefoxPortable\App\firefox\firefox.exe
TargetFilename: C:\Users\\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations\85YY0LRHLX4IUPLTXXXZ.temp
CreationUtcTime: 2015-12-18 08:35:35.991
PreviousCreationUtcTime: 2016-08-01 14:24:22.343

Event ID 3 shows you the network connections
Network connection detected:
UtcTime: 2016-08-01 14:24:19.240
ProcessGuid: {ddfd1a0f-62d5-579c-0000-0010eb030000}
ProcessId: 4
Image: System
User: NT AUTHORITY\SYSTEM
Protocol: udp
Initiated: false
SourceIsIpv6: false
SourceIp: 10.127.255.255
SourceHostname:
SourcePort: 137
SourcePortName: netbios-ns
DestinationIsIpv6: false
DestinationIp: 10.127.12.160
DestinationHostname:
DestinationPort: 137
DestinationPortName: netbios-ns

Event ID 5 shows you when a process is terminated
Process terminated:
UtcTime: 2016-08-01 14:24:17.398
ProcessGuid: {ddfd1a0f-5b8c-579f-0000-00103dcfd004}
ProcessId: 5684
Image: C:\Windows\System32\dllhost.exe

As you can see there is a tremendous amount of info available for an incident responder. If you want some cool ideas what you can do with the data I recommend you to read this excellent post by CrowdStrike will help you get amazing value out of the collected data.

Your log capacity

In this post I want to talk about log capacity. The reason why I want to talk about this is because we noticed that quite a lot of people understand that logs are pretty handy in incident response to figure out what happend but not always have an idea how and what.

A lot of information is produced in your computer system. During an incident response situation, the analyst needs to sift through these log to figure out what happend. Since we live in networked times, this means you got to get these logs from multiple nodes in the network. These can be anything if you want to. To handle this it is important to create a central log server. This makes the attacker his or her life more difficult because now the logs need to be changed at two places.

When setting up a log server one has to take into account that this is traffic over a network, thus you need to make sure that that the protocol used for that log shipping is secure.

A question I sometimes get is what to log, and there the answer is the classic "that depends". Depending on the operating system and the running services and applications the answer depends. The internet is your friend (try log analysis + your subject), but usually is default setting not enough.

Once you are getting a nice amount of data on your log server you might run into storage capacity. One of the important things to know about this is that it used to take more than one year before an organization would discover they got compromised and nowadays it is a bit less than a year.

When you look at an attack campaign like a supply chain model, things have to happen in certain order. Let's say you discover that data is being stolen from your organization, this means that the attacker is at the end of the campaign and if you want to learn about how the bad person got in you got to find it in your logs. When you have only the log capacity of 1 week or 1 month chances are that most information is already gone.

I know disks for things like SANs are not the cheapest things in the world, this means choices have to be made. Depending on your situation a cheaper solution like a NAS or a couple of terabytes of USB/firewire to store offline might be a solution. In this case it is better to have something, than to have nothing because an incident handler can't magically make logs appear.

How long do I need to keep logs?

Today I talked to a guy at an ISP where I do the SQL maintenance and I asked him what they keep in their logs about what people do with their services (telephony and Internet access for companies and private persons).

For the telephony part, the law in Belgium asks them to keep which number called which number and matching them with the contracts of the customers of the telephony providers the law officers can trace your calls. I asked him if this is still the case if you use something like Skype out and according to him there is no way to trace this for the telephony provider, the cops have to have an agreement with Skype (who are based in Luxembourgh).

For the internet behavior he told me that they just keep the IP address leases for the dynamic IP customers and they don't care to what websites you go or what chatrooms you frequent. The only thing that the law requires them to do is to give the name and address of who owned that IP at that particular point in time.

I asked him what the most common case for requesting the users identity and he said that it is usually a case of copyright violation.

He wasn't aware of the TOR network and when I explained to him how it works, he said that it becomes a very difficult task for the cops to trace your particular visit to a website back to you.

One particularity he told me is that the public (companies and private persons) are responsible for keeping their own router logs and should be able to show them to the men of law in case of an investigation. For how long you have to keep them if your not an ISP he couldn't tell me.

If anybody can tell me more about this subject please post a reaction. I think that it is important for the public to know this.