Unsupervised learning based distributed detection of global anomalies

Junlin Zhou, Aleksandar Lazarevic, Kuo Wei Hsu, Jaideep Srivastava, Yan Fu, Yue Wu

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Anomaly detection has recently become an important problem in many industrial and financial applications. Very often, the databases from which anomalies have to be found are located at multiple local sites and cannot be merged due to privacy reasons or communication overhead. In this paper, a novel general framework for distributed anomaly detection is proposed. The proposed method consists of three steps: (i) building local models for distributed data sources with unsupervised anomaly detection methods and computing quality measure of local models; (ii) transforming local unsupervised local models into sharing models; and (iii) reusing sharing models for new data and combining their results by considering both quality and diversity of them to detect anomalies in a global view. In experiments performed on synthetic and real-life large data set, the proposed distributed anomaly detection method achieved prediction performance comparable or even slightly better than the global anomaly detection algorithm applied on the data set obtained when all distributed data set were merged.

Original languageEnglish
Pages (from-to)935-957
Number of pages23
JournalInternational Journal of Information Technology and Decision Making
Volume9
Issue number6
DOIs
Publication statusPublished - Nov 2010
Externally publishedYes

Fingerprint

Unsupervised learning
Communication
Experiments

Keywords

  • combining models
  • Distributed anomaly detection
  • global anomalies

ASJC Scopus subject areas

  • Computer Science (miscellaneous)

Cite this

Unsupervised learning based distributed detection of global anomalies. / Zhou, Junlin; Lazarevic, Aleksandar; Hsu, Kuo Wei; Srivastava, Jaideep; Fu, Yan; Wu, Yue.

In: International Journal of Information Technology and Decision Making, Vol. 9, No. 6, 11.2010, p. 935-957.

Research output: Contribution to journalArticle

Zhou, Junlin ; Lazarevic, Aleksandar ; Hsu, Kuo Wei ; Srivastava, Jaideep ; Fu, Yan ; Wu, Yue. / Unsupervised learning based distributed detection of global anomalies. In: International Journal of Information Technology and Decision Making. 2010 ; Vol. 9, No. 6. pp. 935-957.
@article{0e1445d858494324aef556f231ed2cd1,
title = "Unsupervised learning based distributed detection of global anomalies",
abstract = "Anomaly detection has recently become an important problem in many industrial and financial applications. Very often, the databases from which anomalies have to be found are located at multiple local sites and cannot be merged due to privacy reasons or communication overhead. In this paper, a novel general framework for distributed anomaly detection is proposed. The proposed method consists of three steps: (i) building local models for distributed data sources with unsupervised anomaly detection methods and computing quality measure of local models; (ii) transforming local unsupervised local models into sharing models; and (iii) reusing sharing models for new data and combining their results by considering both quality and diversity of them to detect anomalies in a global view. In experiments performed on synthetic and real-life large data set, the proposed distributed anomaly detection method achieved prediction performance comparable or even slightly better than the global anomaly detection algorithm applied on the data set obtained when all distributed data set were merged.",
keywords = "combining models, Distributed anomaly detection, global anomalies",
author = "Junlin Zhou and Aleksandar Lazarevic and Hsu, {Kuo Wei} and Jaideep Srivastava and Yan Fu and Yue Wu",
year = "2010",
month = "11",
doi = "10.1142/S0219622010004172",
language = "English",
volume = "9",
pages = "935--957",
journal = "International Journal of Information Technology and Decision Making",
issn = "0219-6220",
publisher = "World Scientific Publishing Co. Pte Ltd",
number = "6",

}

TY - JOUR

T1 - Unsupervised learning based distributed detection of global anomalies

AU - Zhou, Junlin

AU - Lazarevic, Aleksandar

AU - Hsu, Kuo Wei

AU - Srivastava, Jaideep

AU - Fu, Yan

AU - Wu, Yue

PY - 2010/11

Y1 - 2010/11

N2 - Anomaly detection has recently become an important problem in many industrial and financial applications. Very often, the databases from which anomalies have to be found are located at multiple local sites and cannot be merged due to privacy reasons or communication overhead. In this paper, a novel general framework for distributed anomaly detection is proposed. The proposed method consists of three steps: (i) building local models for distributed data sources with unsupervised anomaly detection methods and computing quality measure of local models; (ii) transforming local unsupervised local models into sharing models; and (iii) reusing sharing models for new data and combining their results by considering both quality and diversity of them to detect anomalies in a global view. In experiments performed on synthetic and real-life large data set, the proposed distributed anomaly detection method achieved prediction performance comparable or even slightly better than the global anomaly detection algorithm applied on the data set obtained when all distributed data set were merged.

AB - Anomaly detection has recently become an important problem in many industrial and financial applications. Very often, the databases from which anomalies have to be found are located at multiple local sites and cannot be merged due to privacy reasons or communication overhead. In this paper, a novel general framework for distributed anomaly detection is proposed. The proposed method consists of three steps: (i) building local models for distributed data sources with unsupervised anomaly detection methods and computing quality measure of local models; (ii) transforming local unsupervised local models into sharing models; and (iii) reusing sharing models for new data and combining their results by considering both quality and diversity of them to detect anomalies in a global view. In experiments performed on synthetic and real-life large data set, the proposed distributed anomaly detection method achieved prediction performance comparable or even slightly better than the global anomaly detection algorithm applied on the data set obtained when all distributed data set were merged.

KW - combining models

KW - Distributed anomaly detection

KW - global anomalies

UR - http://www.scopus.com/inward/record.url?scp=78149346488&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78149346488&partnerID=8YFLogxK

U2 - 10.1142/S0219622010004172

DO - 10.1142/S0219622010004172

M3 - Article

AN - SCOPUS:78149346488

VL - 9

SP - 935

EP - 957

JO - International Journal of Information Technology and Decision Making

JF - International Journal of Information Technology and Decision Making

SN - 0219-6220

IS - 6

ER -