Blog: SANS Digital Forensics and Incident Response Blog

Blog: SANS Digital Forensics and Incident Response Blog

Is Anti-Virus Really Dead? A Real-World Simulation Created for Forensic Data Yields Surprising Results

One of the biggest complaints that many have in the DFIR community is the lack of realistic data to learn from. Starting a year ago, I planned to change that through creating a realistic scenario based on experiences from the entire cadre of instructors at SANS and additional experts who reviewed and advised the attack "script". We created an incredibly rich and realistic attack scenario across multiple windows-based systems in enterprise environment. The attack scenario was created for the new FOR508: Advanced Forensics and Incident Response course. Our main goal was to place the student in the middle of a real attack that they have to response to.

The purpose is to give attendees of the new FOR508 real filesystem and memory images that they will examine in class to detect, identify, and forensicate APT-based activities across these systems in class. The goal is to give students who attend the course "real world" data to analyze. The goal was to create attack data to use in our courses at SANS so our students could have a direct feel for what it is like to investigate advanced adversaries.

This past week, we ran through the exercise. I had a team of attackers mimic the actions of an advanced adversary similar to the APT. Having seen APT tactics first hand, I scripted the exercise but also wanted to create a realistic environment that would mimic many organizations home enterprise networks. My attack team (John Strand and Tim Tomes) learned quickly the difference between a penetration test and a test that duplicated typical APT actions.

Over the week, I learned some very valuable lessons by being able to observe the attack team first-hand. More in future blog articles, but the first question I had on my list was: "Is A/V really dead?"

Is A/V really dead?


Over the years, I knew that it can be circumvented, but until I helped plan out and execute this exercise I was exposed to the truth first hand. In many incidents over the years (including many APT ones), we and other IR teams have found that A/V detected signs of intrusions, but they were often ignored. I expected at least some of those signs to exist this past week while running through the exercises we were creating. I had hoped differently, but after a week of exploiting a network using the same APT techniques that we have seen our adversaries use, I think it paints a very dark picture for how useful A/V in stopping advanced and capable adversaries. This isn't an anti-AV or HIDS write-up but to give you something to think about when it comes to what we blindly are looking for. I would never recommend someone go without it as AV still stops very basic and simple attacks, but it is clear that in order to find and defend against advanced adversaries we need to do more than rely on A/V. Having A/V state that they can defeat advanced adversaries such as the APT has been a challenge for them and they anti-virus industry has not been very open about this very clear fact.

To be honest, I actually had some hope for some of the enterprise level A/V and HIDS products to catch some of the more basic techniques we used (as I wanted the artifact to be discovered by attendees), but A/V proved easy to circumvent by my team. While I'm sure all many of these products stop low-hanging fruit attacks, we found that we basically did whatever we wanted without our enterprise managed host-based A/V and security suite sending up a flare.

What? Nothing?


What is bundled into this suite? -->Anti-virus, Anti-spyware, Safe surfing, Anti-spam, Device Control, Onsite Management, Host Intrusion Prevention System HIPS bundled in McAfee Endpoint Protection Suite — http://shop.mcafee.com/Catalog.aspx. I also separately purchase their desktop host intrusion prevention piece and built that into McAfee EPO and deployed that across the environment as well.

To help understand how this might have happened, many have asked for the details of the network and the attack.

The Windows Based Enterprise Network:


The network was setup to mimic a standard "protected" enterprise network using standard compliance checklists. We did not include any additional security measures that usually are implemented post APT incidents. This was supposed to mimic a network at a "Day 0" compromise, not actively hunting, using threat intelligence, white listing, and more. However, we did have a substantial firewall, A/V, host based IDS, patching automatically done, and more. We also had fairly restrictive users on it, but included some bad habits found in most enterprise settings (poor admin password policy, local admin accounts w/same password, XP user with admin rights locally)

  • Full auditing turned on per recommended guidelines
  • Users are restricted to being a users (cannot even install a program if they wanted to)
  • Windows DC set up and configured by Jason Fossen (our Windows Security Course) he didn't tighten down the network more than what is expected in real enterprise networks
  • Systems installed and have real software on it that is used (Office, Adobe, Skype, Tweetdeck, Email, Dropbox, Firefox, Chrome)
  • Fully patched (Patches are automatically installed)
  • Enterprise Incident Response agents (F-Response Enterprise)
  • Enterprise A/V and On-Scan capability (McAfee Endpoint Protection —Anti-virus, Anti-spyware, Safe surfing, Anti-spam, Device Control, Onsite Management, Host Intrusion Prevention (HIPS) )
  • Firewall only allowed inbound 25 and outbound 25, 80, 443 only.
  • The "APT actors" has hit 4 of the systems in this enterprise network network. (Win2008R2 Domain Controller, Win764bit, Win732bit, WinXP).
  • Users have been "using" this network for over a year prior to the attack. That way it has the look and feel of something real. These users have setup social media (yes they are on twitter... you might be friends with them), email, skype, etc. Each character user has a backstory and a reason to be there working.

Bad habits we included and commonly see in most enterprise networks:

  • Local Admin User (SRL-Helpdesk) found on each system w/same password
  • A regular user with local admin rights on an XP machine.

Malware Used (non-public):

  • C2 Beacon — Port 80 C2 channel encoded in XMLRPC traffic. Meterpreter backend - What this malware did was beacon out every 32 seconds to a specific IP address over port 80 looking like traditional web traffic. The command and control channel was embedded into XMLRPC traffic. The command and control shell we decided to use a meterpreter backend as developing a new one was too costly and we found that it was still "good enough" for our requirements. Malware detected on Microsoft Security Essentials due to payload, but not in McAfee's products (I know -- odd!).
  • C2 Channel — Custom Meterpreter Backed based executable. Will connect out over port 80. It doesn't have persistence or a beacon interval. Must be started to connect.

Malware Used (Public):


The evasion technique is pretty simple, wrap the executable into a python script (you can also use perl and Ruby) then insert it into a good executable or export to a new one.
  • Poison Ivy - Straight export to Python Array. Pretty sad that it worked actually. This is where I had hoped to create some alerts that I would have had to suppress.
  • Psexec - Not malware
  • Radmin - No encoding needed. Apparently this backdoor is OK?
  • mimikatz - No encoding. Again another place hoping to suppress some alerts so we could find them in the "system forensics" piece of the exercise.

APT Attack Phases


This exercise and challenge will be used to show real adversary traces in network captures, host systems, memory, hibernation/pagefiles etc. Hopefully the malware will be used in FOR610: (malware analysis) for additional exercises. The network captures will hopefully be used in FOR558 (network forensics).

And through the week none of the defenses we had put in place did not matter what-so-ever. It was quite simple to evade any detection. Our APT "team" consisted of John Strand and Tim Tomes.

  • Phase 1 — Spearphsing attack (w/signed Java Applet attack — public) and malware C2 beacon installation (custom malware - encapsulated port 80 http traffic and POISON IVY)
  • Phase 2 — Lateral movement to other systems, malware utilities download, install additional beacons, and obtain domain admin credentials
  • Phase 3 — Search for intellectual propery, profile network, dump email, dump enterprise hashes
  • Phase 4-- Collect data to exfiltrate and copy to staging system. Rar up data using complex passphrase
  • Phase 5 — Exfiltrate rar files from staging server, perform cleanup on staging server
In the end, we will have created authentic memory captures on each box, network captures, malware samples, in addition to full disk images w/Restore Points (XP) and VSS for (Win7 and Win2008) machines.

Why did we choose McAfee's product?


I have seen a lot of enterprise managed A/V and HIPS suites and none of them have fared well against the APT actors and malware. It is too easy to obscure the malware to avoid detection so any A/V choice here (McAfee, Symantec, etc) would have yielded similar results. And that matches what myself and many others have witnessed in seeing these products at locations where APT adversaries roamed freely for months before detection. In the end, it really would not have mattered to choose product X over Y. We wanted to select a product where most attendees of FOR508 would feel at home when performing incident response.

In order to set up a realistic environment I wanted to go with one of the product choices that was implemented in more environments so that attendees could easily "identify" with their own enterprise networks through the lens of this exercise. I asked the SANS advisory board for their recommendations in late August 2011 and found that most seemed to lean toward McAfee EPO. This agreed with me as the DoD Host Based Security System HBSS also implemented similar functionality found in the A/V products we ended up using.

Furthermore, when we installed the product we did not tighten the configuration options beyond what the default settings had chosen for itself. I literally mimicked an admin purchasing a product, installing it, and crossing my fingers -- hoping it worked correctly. In most environments we have investigated, this was typical (w/ standard out-of-box settings.) We also came to find out that the .dat files were not automatically updated. The system apparently needed some care and feeding a bit. However, having said that, the attack team verified during the test that the malware used (public and proprietary) evaded detection using the latest .dat files as of week of 2 April 2012.

We uploaded all the log files for their team to review to get a better sense of what we had installed, if anything was incorrectly applied, and for additional feedback. We used their analysis tool called WebMer found here: http://mer.mcafee.com/enduser/downloadlatestmer.aspx to collect all the log files, operational parameters, and more from each system. McAfee verified via a conference call that the system was installed correctly and operational during the attack, but that some settings could have been implemented that would have slowed the speed of the advance, but not stopped it.

Their team determined we could have implemented several key things to have help slow the attack:

From McAfee --- snip ---

Old DAT

SANS system's show they are using DAT version 6498, this DAT is more than 200 days old and we recommend they update the DAT. Standard deployment is daily update of DATs.


End From McAfee ----snip-----


(Author note about the old DAT: We were not able to verify the old DAT as the product was install with all defaults enabled including automatic updates. When we tested the malware -- the attack team told me that they tested on systems with 100% up to date on the test systems with McAfee installed on it. So even it was older, it didn't matter as it didn't detect anything either way... new or older DAT. McAfee even admitted after seeing the malware used in this attack that we shared with them that their current DAT would not have hit on those samples in the way we had them configured.)


From McAfee --- snip ---


VSE's Access Protection

VSE's Access Protection can mitigate the threat that non-public malware poses. There is a large list of pre-defined Access Protection Rules that an administrator would selectively enable as part of the deployment process depending on the systems and environment VSE is being deployed to.


Here are some of the pre-defined Access Protection Rules that would mitigate or at least alert on the threat posed by the approach SANS used. The first one is recommended for all systems. The latter two are recommended for servers and desktops where the usage scenario is more controlled or specific. In other systems they would typically just be marked for "report" and not "block".

  • Prevent Windows Process spoofing — enable for BLOCK/REPORT
  • Prevent all programs from running files from the Temp folder — enable for REPORT
  • Prevent creation of new executable files in the Windows folder — enable for REPORT
End From McAfee ----snip-----

If we enabled these protections, it would not have really been a stopping point but it would have created logs. I actually regret not having implemented these prior to the test exercise as it would have been great to have someone see the logs during the investigation.

Overall, the point of the exercise was not to embarrass anyone. I wanted to come as close to "real" as I could get. As a result, we knew we had to include real world implementations of some of the best tools money can buy. In the end, this isn't about trying to shame anyone or pick on a A/V vendor. It is about reporting "What happened?" and "What did we notice?" Hopefully everyone learns something in the end about the exercise and benefits from it.

Some IR/Forensic Results from the Attack


Some results from doing some quick Forensic Analysis of these machines

Timeline Analysis of Spearphishing Attack



 

Memory Analysis (Quick hits using Redline without analysis - Yes... for in-depth analysis I would also use Volatility)



Conclusion:


We used a combination of custom crafted malware and well-known malware such as Poison Ivy, metasploit, and more. We used simple A/V evasion to get around it and we NEVER turned it off. RESULT-> NOT A PEEP from A/V. Yes it was installed correctly as it did detect the un-armored metasploit payload quickly and killed it (a test to make sure it DID in fact work as I became worried it really didn't work or was setup wrong). I would gladly let anyone from McAfee look at our setup to make sure we didn't make a mistake, but I followed their guide to the letter and used recommended settings when installing the product (They took us up on that, and we sent in the logs from all 4 systems). I also have found a lot of clients with incorrect installed Enterprise products, so it is clearly possible I mundged something up during the install. If we are wrong, then we are wrong and we can go back and do run through it again after we apply their suggestions as we have it snapshotted inside an ESX server. I was actually anticipating it would find at least ONE thing we did. Nothing was found.

If anyone needs just a little proof that you are using A/V products to mainly defend against low-skilled attackers, then there it is. I asked that the attack team use skills learned in most "Penetration Testing" courses. They didn't use anything really advanced, which is one of the reasons many argue that even the "Advanced Persistence Threat" isn't really that advanced. We even made many mistakes during the attack. Even then... nothing was found and nothing was automatically blocked. If this were a real compromise, we could have been on this network for months or years prior to anyone finding us. Just like in the real world.

17 Comments

Posted April 10, 2012 at 1:01 AM | Permalink | Reply

ven

Hi,

Just wondering how you were able to compose the super timeline? I mean, did you get all the logs and put it in one container/tool/software that can sort the date? Can you please tell us the tool you use for creating this super timeline? I believe this is an interesting post since most people dont understand APT in actual scenario.

Posted April 10, 2012 at 3:33 AM | Permalink | Reply

robtlee

Thanks for asking. On the blog we have a good article on creating it posted here: http://computer-forensics.sans.org/blog/2011/12/07/digital-forensic-sifting-super-timeline-analysis-and-creation# --Rob

Posted April 10, 2012 at 3:08 PM | Permalink | Reply

robtlee

The exact sequence of events (you can also see this in the timeline)

1. Java Applet Attack
2. Via Java Applet Attack -> Upload of stub meterpreter and execute
3. Upload C:\windows\system\32\dllhost\svchost.exe (executable with the persistent meterpreter embedded in it)
4. Set Registry for execution of svchost.exe (interval and connect IP)
5. Set persistence of svchost

Note: The svchost is a meterpreter based executable that encapsulated the traffic into normal looking http traffic (RPC over XML). The svchost also "beacons" out every X seconds which is set in a registry key (IP address and Interval are set).

Posted April 10, 2012 at 2:39 PM | Permalink | Reply

Mark Lachniet

Hi Rob,

Could you clarify on the "Meterpreter backend" executable and how it was gotten on the targets? Was it embedded in another executable? Did you use the packing and crypting functions that are normally used for Meterpreter (i.e. using one of the msfencode routines)?

Posted April 11, 2012 at 1:58 AM | Permalink | Reply

1995virus

This confirms what I discovered from my experience with viruses... you used to be able to disassemble a virus, swap a couple instructions at a key point and you could evade the anti-virus detection with this "new" virus that does exactly the same thing

Posted April 12, 2012 at 5:47 PM | Permalink | Reply

Mascalz1

Hi Rob,
pls can you tell me if DEP were configured and if were not, could it change this malware behaviuor?

Thanks a lot

Antonio

Posted April 13, 2012 at 8:43 PM | Permalink | Reply

robtlee

Out of box settings for XP, Win7, and Server2008R2. We set auto-patching and updating. We turned on auditing and that was it. We didn't retard the system security at all.

Posted April 16, 2012 at 12:57 PM | Permalink | Reply

Liam Randall

Great article Rob; did you guys take any pcaps during the process? I would love to replay them to Snort

Posted April 16, 2012 at 2:45 PM | Permalink | Reply

robtlee

We did. We are going to be using those in the updated FOR558 - Network Forensics in the future.

Posted April 17, 2012 at 6:53 PM | Permalink | Reply

Bart Lewis

Hi Rob,

You know what would have caught this immediately? File integrity monitoring. As soon as any of the critical system files were changed you would have received an alert. Anti-Virus is part of a layered approach to security but FIM is a true last line of defense. I would love to take this conversation further if you want to reach out. Maybe even include it as part of your next test?

Bart

Full Disclosure: I work for a vendor that sells a FIM solution.

Posted April 18, 2012 at 9:49 AM | Permalink | Reply

Ilya Rabinovich

Hi!

In fact, the way you set the question is definitely wrong. AV is not dead, it's miss-advertised and not effective in terms of modern threat's model. The real problem is not "AV is dead", but:
1. Anti-virus industry is too way established and build all the products with blacklisting whitelisting scheme, ignoring all the rest.
2. Users are trust in "big names" and ignoring all the innovations made by small companies/startups.

Unfortunately, no signs the situation will change to good in nearest future.

Posted June 08, 2012 at 5:37 PM | Permalink | Reply

Matt

Some of the newer A/V products from Symantec (and I though McAfee) appear to take a white-listing approach based on file reputation. The new version of SEP messed up one of our regular incident exercises because the custom binaries we use hadn't been seen before and SEP wouldn't let them run. How would this functionality have impacted this exercise?

Posted December 03, 2012 at 1:37 PM | Permalink | Reply

John S

The biggest complaint from PC users about security suites is annoyance of the suite itself. Anything from slowing the PC to having too many annoying features that are almost as bad as malware itself.
Its why I only use a "free" AV solution. The slightly added protection to me is simply not worth the added annoyances and costs. The bottom line is most all security suites operate the same way. Filter any downloaded data and compare it against a list of know malware.
As someone pointed out this solution has always had a flaw. Because in order to have it on a list it must have been discovered already. In the end a properly updated operating system and a user familiar with how malware is passed through their PC can eliminate a lot of risk in itself. Of course Security application makers prefer to use scare tactics to sell their software and it does work for many who are paranoid and unfamiliar with protecting themselves. All it takes is one bad piece of malware on your PC and your hooked to go out and spend some money on Security. But are you really solving anything? I have done my share of cleaning PC's with malware that have some of the most complex Security software installed and updated. It simply proves that a well engineered piece of malware has no trouble getting past security and the unsuspecting PC user. The only thing that eventually indicates to the user of any problem is a slow PC or a freezing PC.
By that time its very possible the damage has already been done.
The trouble is that their is a short list of weak areas in PC's these days where malware is targeted. Its focused on certain plug ins, apps and email. Much of operating system and browser security issues are dealt with efficiently most of the time. In the end most people who find security suites a requirement will also find they only solve part of the problem. The rest of us know that a educated user is far more effective and depends on a security suite only as a backup.

Posted February 05, 2013 at 3:52 AM | Permalink | Reply

Chris

do you have any additional info on:

" Port 80 C2 channel encoded in XMLRPC traffic. Meterpreter backend"
?

Posted February 08, 2013 at 8:38 AM | Permalink | Reply

Kevin Bishop

Since your testing in an Enterprise Environment would using a technology such as Microsoft's AppLocker prevented the attack?

Posted August 22, 2013 at 1:49 PM | Permalink | Reply

Maria S. Walker

Hi Rob,

I'm just wondering what is the difference with malware and anti-virus software. Which one should be used? I have a laptop that runs with Windows 7 OS. Anyway, thanks for posting and sharing this information.

Posted September 12, 2013 at 6:45 PM | Permalink | Reply

Danny D. Clark

Security software can be helpful especially when keeping your computer safe from internet threats. However, it can sometimes be annoying as well. It keeps asking on things like your computer needs to be backed up with stronger protection shield, or the pc is working slow, etc etc. These features is sometime irritating. But nevertheless, we need antivirus really.

Post a Comment






Captcha

* Indicates a required field.