The Irrelevancy of Industry Accepted Malware Testing Standards
Writing or presenting about AV testing and performance is a great way to draw the collective ire of the AV industry. This is a hot button subject that I, personally, have received a lot of grief on. The primary reason that the AV industry is so sensitive about their software is because it is not as effective as they would like you to believe. Case in point is the recent Anti-Malware Testing Standards Organization’s document titled Issues involved in the ‘creation’ of samples for testing. If you want to find a document listing all the hot-button issues that particularly perturb the AV community, here it is.
Without taking a particular side, the document seeks to “frame the debate” of the issue of “creating” malware samples. What follows is a 19 page exploration of all the ways new malware can be created. Here is a short list of modifications that they address:
- Archiving samples using ZIP or tar
- Packing / repacking with a new packer (think UPX or ASPack)
- Using a malware generation kit
- Server-side polymorphic samples - the sample is slightly modified every time it is downloaded from a public website
- Patched versions of an existing file, including PE modifications and actual code changes
- Writing a custom packer
- Writing a new sample using existing techniques
- Writing new samples using unknown techniques
Specifically prohibited is public dissemination of malware samples. These might actually encourage people to test AV software before buying it.
The pros and cons of each are presented, followed by a way to frame your debate afterwards. What all of these miss is the central point that malware authors are using every single one of these techniques with spectacular success. The other terrible secret is that these techniques are extremely easy. Continued debate on whether or not these tests are ethical is moot because malware authors are already using them. In order to protect against real threats, you must use the techniques that are being used to evade your protection software.
Consider the NHTSA talking about testing crash performance, but not actually ever smashing any of the cars into a wall. There’s no substitute for the real thing unless you’re trying to hide something. In the case of the AV industry, that thing is their technological irrelevance to the modern malware threat.