KEYNOTE TALK AVerify: AN OPEN-SOURCE ANTI-VIRUS - Eicar

AVerify
Towards verifiable anti-virus testing
2010-05-10 EICAR 2010

What’s this presentation about?
� Mostly AV testing, of course
� Just my personal point of view (not my
employer’s!)
� The reasons behind AVerify, and the project
goals
2010-05-10 EICAR 2010

A philosophical note
� The bad guys are helping each other
– sharing/selling techniques, codes, …
� Yet the AV ecosystem is fragmented, highly
competitive
– Non disclosure of information/samples/… gives an
edge (temporary)
– Cooperation exists on a small scale
2010-05-10 EICAR 2010

A philosophical note
� Information sharing is a good thing, in general
– Security through obscurity never works
� See: Mifare (RFID), A5/1 (GSM), …
– I started with ProView, thanks to McAfee :)
� Case in point: WEP, 2004
– Wireless security started improving after tools
became readily available
– Studying attacks techniques should be encouraged
2010-05-10 EICAR 2010

(not my fault)
2010-05-10 EICAR 2010

� AV comparisons can be found everywhere
� A few problems:
– Does it still make sense to test with 16-bit DOS
infector samples?
� Nope. But look at that shiny 99.8% detection rate!
– Results cannot be reproduced, no clear methodology
given. “Independent” ?
– Scanning 10000+ files a realistic usage scenario?
– Ad-hoc, black-box scoring
� “My My AV must be better than yours, it’s got a 4-star GOLD
award!”
2010-05-10 EICAR 2010

Let’s search “antivirus reviews” on
Google. The winner is…
2010-05-10 EICAR 2010

A sound methodology!
� Apparently based on checking boxes:
2010-05-10 EICAR 2010

“Race Race To Zero” (2008) Challenge
� Defcon 16, Las Vegas, August 2008
– Aim: modify known samples to bypass signaturesignaturebased detection
– Best time: nine samples obfuscated in 3 hours
� Lots of media coverage
� Precise technical details not shared with the
public
2010-05-10 EICAR 2010

“Race Race To Zero” (2008) Challenge
� Signature-based detection will fail eventually
– Cf. Fred Cohen
� What good is this challenge?
– Precise results and samples not available
– Not a set of single tests, more of an ad-hoc method
(let’s nop/pack/obfuscate that sample until it evades
the AV)
– Cannot be reproduce exactly, obfuscation technique
not shared
� Not worthless, but not earth-shattering either
2010-05-10 EICAR 2010

The firewall “leak tests”
� Guillaume Kaddouch, 2007
� Single tests, mostly network evasion &
keyloggers
– Disclosed exploit techniques & executables
– Published the results for each test
� Initially, most vendors didn’t pass
– But they improved their products
� Sadly, the test suite is no longer maintained
– And source code is not accessible
2010-05-10 EICAR 2010

“A A study of anti-virus’ response to
unknown threats (EICAR, 2009)”
� N. Richaud and myself
� Twelve anti-virus products tested
� 21 single tests, oriented towards “proactive”
(HIPS-like) detection
� Tests run by hand in Dec. 2008
– Some tests did require admin rights, but not all
� Results published at EICAR 2009
2010-05-10 EICAR 2010

� Low-level access
EICAR 2009
2008 versions MBR (modified
bootroot)
avast!
AVG
Avira Detected
BitDefender
ESET
F-Secure
Kaspersky Detected
McAfee
Norton
Panda
Sophos
TrendMicro
2010-05-10 EICAR 2010
Device\PhysicalMemory

� Keyloggers:
EICAR 2009
2008 versions WH_KEYBOARD_LL GetRawInputData
avast!
AVG
Avira
BitDefender
ESET
F-Secure
Kaspersky Detected
McAfee
Norton
Panda
Sophos
TrendMicro Detected
2010-05-10 EICAR 2010

� Code injection:
2008 versions CreateRemoteThrea
d
avast!
AVG
Avira Detected
BitDefender
ESET
F-Secure
EICAR 2009
SetWindowsHookE
x
Kaspersky Detected
McAfee
Norton
Panda
Sophos
TrendMicro Detected Detected
2010-05-10 EICAR 2010
QueueUserAPC

� Lessons learned:
EICAR 2009
– 12 AV x 21 tests = a full week of work (non-stop!
except the coffee breaks ;)
– Coding is fun, testing is very repetitive & boring
– A partial view, limited to HIPS-based detection
– Tests were run on a specific configuration; Windows
Vista was ignored
� No real winner(s), even basic techniques were
barely detected
2010-05-10 EICAR 2010

The iAWACS 2009 Challenge
� Goal: to disable several anti-virus programs
without the user noticing
� Windows XP SP3 with admin account
� Results:
– Almost all AV could be disabled thanks to ring0
access
– Took a few minutes to a few hours (tops)
2010-05-10 EICAR 2010

The iAWACS 2009 Challenge
� Lessons learned:
– Disabling is easy with admin rights ;)
� Wait, we knew that already.
� But should it be?
� HIPS-like detection leads to the problem of false positives…
– Many classic techniques still not blocked
� Including access to the PhysicalMemory device
� First published in Phrack, 2002
– Once again, there was no clear winner or loser
2010-05-10 EICAR 2010

The iAWACS 2010 Challenge
� Different rules:
– Windows 7, user account (no admin rights at all)
– Every possible attack was considered fair game
– Time limit: four hours
� Seven attacks proposed:
– Many denial-of-service ones
� One was .bat-powered ;)
– Ransomware
2010-05-10 EICAR 2010

The iAWACS 2010 Challenge
� Judging the attacks, a challenge in itself
– Some were quite sophisticated
– Some very basic but still effective
� Reiterates that signature-based detection not a
silver bullet
� Although a few good surprises:
– Example: KAV’s warning of an non-signed program
launched
– Creation of a key in CurrentVersion\Run detected by
two other anti-viruses
2010-05-10 EICAR 2010

The Guillermito case
� 2001, Tegam vs. Guillermito (a French hacker)
– Tegam claimed to protect against all known and
unknown viruses
– Guillermito then disclosed several flaws in Tegam’s
product, ViGuard
� Condemned (2006) to pay 15000 euros
– Reverse-engineering cited as a reason
� Tegam then went bankrupt…
� Preventing reverse-engineering only helps the
bad guys!
2010-05-10 EICAR 2010

Some difficulties of testing
� Reverse-engineering: to be avoided?
– Or just claim to have been very lucky ;)
– Reading the fine print, tedious but required
� Evaluation versions can be hard to find
– Some AV companies make it easy, others not
– Could be considered a single test in itself
� Finding new threats is a challenge in itself
Finding new threats is a challenge in itself
– Even AV companies have trouble obtaining samples
2010-05-10 EICAR 2010

More difficulties of testing
� Can the test environment be considered
realistic?
– Many malware detect VMware and other vm
– Single tests may run for a few minutes, real users use
their computer for hours
� A better setup would require:
– One physical machine per AV in parallel with identical
hardware
– “dummy” dummy” robots mimicking users
– Knowing which malware are the most common
– Gaining access to new samples near release time
2010-05-10 EICAR 2010

Testing, but what?
� Anti-viruses are complex products with heaps
of features
1. What exactly are they supposed to be
protecting against?
– Malware, rootkits, spyware,
, …
1. What are security mechanisms are
implemented? What should be tested exactly?
� More features not always a good thing
– Software less resilient, greater attack surface
2010-05-10 EICAR 2010

Does CC Certification make sense?
� (CC = Common Criteria, an evaluation process)
� Pros:
– Full review of documentation and source code
– In theory, best known attacks are applied
– Higher EAL levels offer (semi) formal proof
� Cons:
– TOE/ST is often reduced to save time
– Applies to a single version, but AV evolve fast
– No common reference for the attacks
�� depends on the evaluator’s competence
– Just another marketing gimmick?
2010-05-10 EICAR 2010

A note on CSPN Certification
� (CSPN = “Certification Sécurité Premier Niveau”)
� Similar to the Common Criteria, however:
– “Single Single shot” evaluation, much shorter (1 month)
– Mostly focused on the attacks themselves
� Cons:
– Like CC, nothing is made public
(except the eventual certificate)
– Perimeter is restricted as well
– Cert. applies to a single version
– Testing depends on the evaluator
2010-05-10 EICAR 2010

AVerify
� Inspired by the EICAR anti-virus test file
� But this project is independent from EICAR itself!
� Follows the EICAR code of conduct
� Provide a set of simple tests to other researchers
– with ith the full description & source code
– w
� This ensures:
– Independent reproducibility
– Based on the original experimental desc.
– Reliably repetition of these experiments
� Allows fact-based reviewing of AV programs
Allows fact-based reviewing of AV programs
– Instead of the hand-waving one often sees
2010-05-10 EICAR 2010

The basics
� Define a common platform
– Windows XP 32-bit still widely used
– Windows 7 64-bit gaining momentum
– Which software to install? Firefox/IE8/Chrome…?
� Define a base privilege level
– Admin or not admin? Or better, both.
� Define common usage scenarios and attack
vectors
– Opening malicious links from email / IM
– Running a malware run from an external drive:
USB key / CD-ROM / network drive
– Machine already infected, try do disinfect
� Evaluate the success of disinfection with a LiveCD
2010-05-10 EICAR 2010

Planned tests #1: HIPS features
� Very similar to our 2008 tests
� Every test should only exercise one aspect
– Old-school persistence through CurrentVersion\Run
– WH_KEYBOARD Windows hook
– StartService driver loading
– This includes firewall bypassing techniques
� Run each test identically on all AV
Run each test identically on all AV
– Define a passed/not passed check
� Combination of tests to simulate real
threats, evaluate AV’s threshold level
2010-05-10 EICAR 2010

Planned tests #2: resilience
� Try to cause faults in the AV itself
– Fuzzing: file formats, IOCTLs
� Instrumenting AV scanners may be tricky
– Example: avast! looks simple, just call ashQuick.exe on each file
– But exception catching requires removing SSDT hooks
– Checking and modifying ACLs
� Files, pipes, handles, …
– Attempts to disable/kill the anti-virus
� hosts file, connection blocking
� On-the-fly code patching
� On-disk signature database corruption
� Network updates corruption
2010-05-10 EICAR 2010

Planned tests #3: real-world
� This is slightly harder:
malware
– Find a new threat after it is released
– Repeatedly scan with each anti-virus to determine the
reaction time of AV companies
� Or use any older but common threat
� Install the threat, create a snapshot
– Determine how it installs and stays persistent
– Attempt to disinfect with each AV
– Use a LiveCD to check for successful disinfection on
disk
2010-05-10 EICAR 2010

Planned tests #4: other stuff
� Usage monitoring:
– Size on disk after installation
– Size on disk after x months of updating
– Amount of real memory usable in idle
– CPU consumed when scanning a threat
– Bandwidth consumed
� UI features
– Usefulness of error messages
– Logging and access to log files
� Support
2010-05-10 EICAR 2010

An interesting attack
� “KHOBE KHOBE – 8.0 earthquake for Windows desktop
security software”
– They might be over-hyping it a little ;)
– Researchers provided full details
– Could be re-implemented in AVerify
2010-05-10 EICAR 2010

Automation ftw
� Mostly based on VMware scripting (VIX)
– C/VB API bundled with VMware Workstation
– Also nicely documented
– Avoids errors due to manual testing
– Lots of useful APIs:
� VixVM_RevertToSnapshot
� VixVM_RunProgramInGuest
� Cannot send raw keyboard/mouse input
– However, VMware offers a VNC server
– VNC protocol open and easily scriptable
2010-05-10 EICAR 2010

Scripting a virtual machine
� Checking for success or failure:
– Depends on the test (eg. capturing keys, …)
– VixVM_CopyFileFromGuestToHost
– VixVM_CaptureScreenImage
� VirtualBox
– VBoxManage is kinda like VIX
� Start, stop, revert to snapshot…
– No remote code execution & VNC server though
� Installing one might interfere with the tests
– One possible alternative: enable remote desktop
� Scripting by modification of an OSS rdp client
2010-05-10 EICAR 2010

On providing source code
� Is providing attack code illegal in France?
– A somewhat muddy subject
– See French ministry of interior vs. VUPEN
� 2009: Author of exploit condemned to pay 1000 E
� I didn’t get jailed for publishing aircrack
– But that was in 2004/2005
– It did include win32 binaries
� AVerify will provide source code
– But on a per-request basis
– No ready-to use binaries!
2010-05-10 EICAR 2010

On providing samples
� Distributing samples == distributing malware?
– Yes but…
� Very useful for other security researchers
– vx.netlux.org
– offensivecomputing
� Real-world malware samples from AVerify to be
made available on offensivecomputing
– Or any other site willing to host them
2010-05-10 EICAR 2010

A simple example
� Automation of the EICAR anti-virus test file
through VMware VIX
� Current problem: detecting that it worked
– Screenshots not reliable
– AV logs?
2010-05-10 EICAR 2010

The future of AVerify?
� Lots of ideas, but not much code yet!
– No demo quite yet, sorry
– As always, time is not on our side
– Right now, it’s a one-person project
� First initial target: November 2010
– Source code and initial results made public
– Hopefully, together with a submission for the next
EICAR conference :)
� Anyone can contribute! You can too.
� Please check averify.org in a few months
2010-05-10 EICAR 2010

A final note
During the installation of a major AV (2010):
Scaring your users is not the solution. Educating them is.
2010-05-10 EICAR 2010

Q&A
Thank you for your attention!
2010-05-10 EICAR 2010