AV-meter

An evaluation of antivirus scans and labels

Aziz Mohaisen, Omar Alrawi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

34 Citations (Scopus)

Abstract

Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection. In this paper, we set out to answer several questions concerning the detection rate, correctness of labels, and consistency of detection of AV scanners. Equipped with more than 12,000 malware samples of 11 malware families that are manually inspected and labeled, we pose the following questions. How do antivirus vendors perform relatively on them? How correct are the labels given by those vendors? How consistent are antivirus vendors among each other? We answer those questions unveiling many interesting results, and invite the community to challenge assumptions about relying on antivirus scans and labels as a ground truth for malware analysis and classification. Finally, we stress several research directions that may help addressing the problem.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer Verlag
Pages112-131
Number of pages20
Volume8550 LNCS
ISBN (Print)9783319085081
DOIs
Publication statusPublished - 1 Jan 2014
Event11th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, DIMVA 2014 - Egham, United Kingdom
Duration: 10 Jul 201411 Jul 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8550 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other11th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, DIMVA 2014
CountryUnited Kingdom
CityEgham
Period10/7/1411/7/14

Fingerprint

Malware
Labels
Scanner
Evaluation
Countermeasures
Classification Algorithm
Labeling
Baseline
Correctness
Disinfection
Attack
Family
Truth

Keywords

  • Automatic Analysis
  • Evaluation
  • Labeling
  • Malware

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Mohaisen, A., & Alrawi, O. (2014). AV-meter: An evaluation of antivirus scans and labels. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8550 LNCS, pp. 112-131). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8550 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-08509-8_7

AV-meter : An evaluation of antivirus scans and labels. / Mohaisen, Aziz; Alrawi, Omar.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8550 LNCS Springer Verlag, 2014. p. 112-131 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8550 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Mohaisen, A & Alrawi, O 2014, AV-meter: An evaluation of antivirus scans and labels. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 8550 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8550 LNCS, Springer Verlag, pp. 112-131, 11th International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, DIMVA 2014, Egham, United Kingdom, 10/7/14. https://doi.org/10.1007/978-3-319-08509-8_7
Mohaisen A, Alrawi O. AV-meter: An evaluation of antivirus scans and labels. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8550 LNCS. Springer Verlag. 2014. p. 112-131. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-08509-8_7
Mohaisen, Aziz ; Alrawi, Omar. / AV-meter : An evaluation of antivirus scans and labels. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8550 LNCS Springer Verlag, 2014. pp. 112-131 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{d35d1e284d914772903a4dc5bb26161d,
title = "AV-meter: An evaluation of antivirus scans and labels",
abstract = "Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection. In this paper, we set out to answer several questions concerning the detection rate, correctness of labels, and consistency of detection of AV scanners. Equipped with more than 12,000 malware samples of 11 malware families that are manually inspected and labeled, we pose the following questions. How do antivirus vendors perform relatively on them? How correct are the labels given by those vendors? How consistent are antivirus vendors among each other? We answer those questions unveiling many interesting results, and invite the community to challenge assumptions about relying on antivirus scans and labels as a ground truth for malware analysis and classification. Finally, we stress several research directions that may help addressing the problem.",
keywords = "Automatic Analysis, Evaluation, Labeling, Malware",
author = "Aziz Mohaisen and Omar Alrawi",
year = "2014",
month = "1",
day = "1",
doi = "10.1007/978-3-319-08509-8_7",
language = "English",
isbn = "9783319085081",
volume = "8550 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "112--131",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - AV-meter

T2 - An evaluation of antivirus scans and labels

AU - Mohaisen, Aziz

AU - Alrawi, Omar

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection. In this paper, we set out to answer several questions concerning the detection rate, correctness of labels, and consistency of detection of AV scanners. Equipped with more than 12,000 malware samples of 11 malware families that are manually inspected and labeled, we pose the following questions. How do antivirus vendors perform relatively on them? How correct are the labels given by those vendors? How consistent are antivirus vendors among each other? We answer those questions unveiling many interesting results, and invite the community to challenge assumptions about relying on antivirus scans and labels as a ground truth for malware analysis and classification. Finally, we stress several research directions that may help addressing the problem.

AB - Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection. In this paper, we set out to answer several questions concerning the detection rate, correctness of labels, and consistency of detection of AV scanners. Equipped with more than 12,000 malware samples of 11 malware families that are manually inspected and labeled, we pose the following questions. How do antivirus vendors perform relatively on them? How correct are the labels given by those vendors? How consistent are antivirus vendors among each other? We answer those questions unveiling many interesting results, and invite the community to challenge assumptions about relying on antivirus scans and labels as a ground truth for malware analysis and classification. Finally, we stress several research directions that may help addressing the problem.

KW - Automatic Analysis

KW - Evaluation

KW - Labeling

KW - Malware

UR - http://www.scopus.com/inward/record.url?scp=84904135251&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84904135251&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-08509-8_7

DO - 10.1007/978-3-319-08509-8_7

M3 - Conference contribution

SN - 9783319085081

VL - 8550 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 112

EP - 131

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

PB - Springer Verlag

ER -