Skip navigation.

The Irrelevancy of Industry Accepted Malware Testing Standards

Writing or presenting about AV testing and performance is a great way to draw the collective ire of the AV industry. This is a hot button subject that I, personally, have received a lot of grief on. The primary reason that the AV industry is so sensitive about their software is because it is not as effective as they would like you to believe. Case in point is the recent Anti-Malware Testing Standards Organization’s document titled Issues involved in the ‘creation’ of samples for testing. If you want to find a document listing all the hot-button issues that particularly perturb the AV community, here it is.

Without taking a particular side, the document seeks to “frame the debate” of the issue of “creating” malware samples. What follows is a 19 page exploration of all the ways new malware can be created. Here is a short list of modifications that they address:

  1. Archiving samples using ZIP or tar

  2. Packing / repacking with a new packer (think UPX or ASPack)
  3. Using a malware generation kit
  4. Server-side polymorphic samples - the sample is slightly modified every time it is downloaded from a public website
  5. Patched versions of an existing file, including PE modifications and actual code changes
  6. Writing a custom packer
  7. Writing a new sample using existing techniques
  8. Writing new samples using unknown techniques

Specifically prohibited is public dissemination of malware samples. These might actually encourage people to test AV software before buying it.

The pros and cons of each are presented, followed by a way to frame your debate afterwards. What all of these miss is the central point that malware authors are using every single one of these techniques with spectacular success. The other terrible secret is that these techniques are extremely easy. Continued debate on whether or not these tests are ethical is moot because malware authors are already using them. In order to protect against real threats, you must use the techniques that are being used to evade your protection software.

Consider the NHTSA talking about testing crash performance, but not actually ever smashing any of the cars into a wall. There’s no substitute for the real thing unless you’re trying to hide something. In the case of the AV industry, that thing is their technological irrelevance to the modern malware threat.



I believe you when you say that you have attracted the "ire" of the AV industry. Heh, a number of them are pretty crusty. But I don't agree with you that it is because they are hiding irrelevance or a lack of efficacy.

And your opinion is a funny one for a group that was bought by an AV vendor a couple of years ago for 13 million. :)

Here are a few thoughts on your post...

The document does not appear to be about obtaining or validating appropriate sample sets. It is a set of arguments about the value of creating malware. Even the term and concept of "creation" itself is arguable and discussed. You seem to be talking mainly about something else -- selecting and validating appropriate sample sets.

Malware is "malicious". I would prefer that every Tom Dick and Harry is not on my network handling and running malware of any kind, or on the same networks as SCADA, whether the malware is new ITW, prevalent but old ITW, or their own creation. And not because of some vague ethical issue. I also would prefer that individuals do not twirl a loaded pistol on their finger in public, just to test the efficacy of their homemade gun's trigger safety (the drama is swell).

"In order to protect against real threats, you must use the techniques that are being used to evade your protection software."
That's true, and you already said that malware authors are releasing this stuff, so no need create your own, right? Testers currently use volumes of malware that is currently ITW. Running useful, accurate real world tests is a large effort. The dynamic tests seem to be the most relevant out there, because they are replicating attacked users' scenarios over the course of months, like this one -- for the purpose of this discussion, the crash performance tests are being completed:

Finally, it would be great if you could elaborate on the "modern malware threat" that you suggest the entire AV industry is irrelevant to and hiding from (seriously, I am interested). It seems that a number of the suites do a pretty good job (see the link above). I would agree that many of the vendors' marketing messages out there, as in every industry, do not help clarify the effectiveness of any one solution in relation to others.


I'll try to touch on

I'll try to touch on everything you said, forgive me if I miss some of it.

They are either not running real attacks or suppressing them due to incompetence or choice, but both are bad. I'm not sure why they would advocate against performing real-world testing.

13 million? You're thinking of someone else.

The very act of arguing against creating malware for testing purpose is the bothersome point of the document. Creating new sample sets based on modifications that virus authors can and do perform are valid.

The malware and firearm debate is pretty old and has been debated at length. Without starting that up again and repeating myself, they have nothing in common. If you don't want malware on your network there very last thing you should be worrying about is someone getting samples and releasing them there. You should be worried about people hacking into your network and dropping custom malware on it.

The primary threat I was referring to in the "modern malware threat" is any sample that is slightly modified (like the colors in an icon resource in the PE) that evades antivirus scanners.

I think ...

... mugg is referring to your 1 April 2006 post ;-)

It's always best to check

It's always best to check the date around that time of year. :)

thoughts on thoughts


Yes, the April Fool's comment was meant to be an oh-so clever joke on my part. Oh well. Next time instead of a smile emoticon, I'll look for a dry stupid humor emoticon.

"They are either not running real attacks or suppressing them due to incompetence or choice, but both are bad."
I'm not sure why you say that. The PCMag/AV-Test link that I included above is as real-world as they come. Dynamic tests are supposed to be "real world". The testers seem to be a competent group, are you connected with them and know something else about their work?

So what do you think should replace the products being sold?

My bad on the joke, my own

My bad on the joke, my own joke even. :)

I looked at the PCMag/AV-Test article but didn't really get the sense of what kinds of tests they were doing. The article says "up-to-the-minute" threats but that doesn't really address what they did. The fact that they have such high detection number (98% in some cases) seems to indicate their test was bad. I could be wrong though.

The one testing organization I like that seems to do a decent job is AV Comparatives. I was actually able to replicate their results from the "Proactive/Retrospective" tests they perform. Here's a report from May of 2009:

That test lines up with what my testing has shown.

As far as what products should replace the ones being sold? Unfortunately the ones that are there are the only solution available. Aside from running no antivirus there's not really an option.

Yeah, I liked that one. The

Yeah, I liked that one.

The AV-Test run seemed to have a crew testing with 10 "fresh" ITW malware each day for three months against each suite (I hate it when they call "undetected" malware "0day"). I'm not sure that they used P2P, usb-autorun and email delivery vectors. It seems only web sites were visited, although I don't see methodology details online, and am somewhat reading between the lines).
Another description here and here: (chrome with google translate is your friend)
I think that the numbers are high in part because suites were used against web based attacks (which reflects ITW prevalence fairly well, imho) with the behavioral stuff, Url filtering, packer-focused scanner heuristics and cloud stuff all together.

I dunno, it seems to me that targeting just AV file scanners isn't necessarily "real-world", because it is testing one piece (cuz it's often the cheapest and easiest test when testers take shortcuts) of the suites out there. So I would agree that often (but not necessarily reflecting ITW prevalence accurately) scanners may have the results you linked to. But I really wouldn't consider that "real-world". The vendors aren't hiding from real world results. They are building on top of the file scanners with better stuff (and hey, the file scans are just going to be around, it's what people understand -- click on update, scan a file, report good or bad, I can buy it!).


Using just the file based

Using just the file based scanner is definitely taking a shortcut for testing, but I would argue it's the one piece that the entire AV system depends on. I would tend to not trust tests that didn't do a statistically significant number of tests. 10 per day is not statistically significant.

Also many of the real world tests would depend on the quality of web addresses being tested. These are all valid concerns, and many AV companies rightfully get picky about them.

Accurately recording the

Accurately recording the results from visiting ten unique and fresh sites per day with a dozen fresh machines over three months is a very large task. There are many "grey area" situations. Anyways, imho, this task of visiting almost 1000 sites is statistically significant for ITW attacks and far more accurately measures the impact of attacks on a set of systems and their anti-malware than scanning any measly directory of potentially invalid files. And "anti-malware" products are no longer dependent on AV file scans (they employ url filtering, "cloud" stuff, HIPS, spam filters, topology-based heuristics, sandboxes, anti-BoF exploitation, DEP, ASLR, etc), in many cases, it is their weak link.

I will be very interested to

I will be very interested to read more about your research on this topic then.

Do you know...

...what did it for me was sitting in on the AMTSO meetings (as a relative outsider) and watching representatives of 30-40 AV companies disagreeing with each other on virtually every subject on the agenda, and how finely the money-making aspect of things is interwoven in the security agenda as a whole. I remember thinking, no frigging wonder the bad guys have one up on us, with the big guys arguing like a bunch of kids on the playground, because Peter has taken away 1 extra marble from Steve's pile. And the funny thing is, some of the guys own up to how ridiculous it all has become these days, trying to do a job, between lay-offs, profit margins and the marketing hype.


you went to some or their

you went to some or their meetings? And you expected to see agreement and hand holding between competitors in a saturated industry? That's funny.
Say, you ever been to an IETF meeting? It's amazing that routers talk with each other at all. It's pretty naive to think that these organizations, in addition to the underground markets, have no bottom line. Neither side is a charity.

You should think, gee whiz, attackers usually have one up because that is the nature of offense v. defense. Not because vendor reps, testers, journalists disagree in discussion. See DailyDave...