AV Testing Standards: Don't Like the Results of the Tests? Change the Rules
There were good responses, mostly from people in the AV industry, to my blog post about the malware testing standards. Overlooking my error linking to their original paper (sorry) there were some points I would like to address.
At the heart of this whole process, is exactly how dangerous a collection of malware is. For the consumers, I would argue, it's not dangerous at all. The malware industry is the only one who has to fear from it. Notice I didn't say just the AV vendors, but also the producers of the malicious software. In large part the authors depend on a closed, inside group of people unwilling to collaborate openly on the problem. If you look at the major sources of malware in academic research prior to the creation of large open collections, you'll see that there were some big problems. First, the samples were old and not representative of current threats. Second, those samples either did not work or were not malicious in nature. Finally, the samples are traded as something of value.
I'm no different, of course. I derive value both from the collection and from consulting. I do, however, go out of my way to support those doing open research as much as I can. If someone in academia needs access to samples, just contact me and I'll work something out. Likewise we have helped innumerable small businesses get their start in the malware world before they could enter the "circle of trust" mentioned by David Harley.
The "circle of trust" is often cited when discussing who can and cannot gain access to these samples. Over the course of the years I've joined four of these groups. While the vetting is done as best as possible, there's very little outside of an email address, and a recommendation keeping someone from joining. Antivirus vendors exchange malware with themselves at a much higher volume, but there is still a perceived difficulty of entering this area. Malware exists on the Internet in a freely available manner as a function of its being. Limiting sample access to a certain set of privileged people fundamentally hurts innovation and response by everyone.
There was also some allusion that I did not support malware testing at all. That is not the case. Malware defense systems should be heavily tested against a range of threats. The basis for my problems with the AMTSO is that it should *not* be composed of anyone in the AV industry. Consumer Reports did an excellent job exposing the ineffectiveness of AV vendors by producing new samples. Due to the very nature of the threat, there are going to be new samples that are discovered for the first time. If an AV software can't respond to this threat, it should not be given a favorable review.
The current set of players in the malware testing arena are profit driven. In and of itself that's ok, I'm all for capitalism, but in fairness there needs to be an independent authority. AV testing companies that publish open information on the effectiveness of scanning results are not independent. Without naming names, there is a prominent one claiming to provide results for the public, but instead is backed by every AV vendor in the industry. This testing company takes in new samples, scans them with all the products, then tells the vendors how their performance rates. What is not acceptable, in my view, are the shoddily written reports intended for consumers that report unethically high detection rates.
Finally I would like to address the ethics of the malware tester. One thing I agree with David Harley on is the need to represent the full scope of the testing process to the consumer. One of the things that the academic world does well is to produce research which can be recreated by other researchers. That's the intent, at least. AV testing standards advocated by the vendors cannot and will not provide the latest samples to malware authors. What this ends up doing is providing all the methods of testing, but not the actual data to test on. For those of us able to use new samples, it's not a problem. Others who have older data and are unable to acquire new malware (due to cost, time involved, etc.) are left with only one viable option: Synthesize new samples using the exact same methods available to the authors.