A new study of two divergence metrics for change detection in data streams

Abdulhakim Qahtan, Suojin Wang, Raymond Carroll, Xiangliang Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Streaming data are dynamic in nature with frequent changes. To detect such changes, most methods measure the difference between the data distributions in a current time window and a reference window. Divergence metrics and density estimation are required to measure the difference between the data distributions. Our study shows that the Kullback-Leibler (KL) divergence, the most popular metric for comparing distributions, fails to detect certain changes due to its asymmetric property and its dependence on the variance of the data. We thus consider two metrics for detecting changes in univariate data streams: a symmetric KL-divergence and a divergence metric measuring the intersection area of two distributions. The experimental results show that these two metrics lead to more accurate results in change detection than baseline methods such as Change Finder and using conventional KL-divergence.

Original languageEnglish
Title of host publicationECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings
PublisherIOS Press
Pages1081-1082
Number of pages2
ISBN (Electronic)9781614994183
DOIs
Publication statusPublished - 1 Jan 2014
Externally publishedYes
Event21st European Conference on Artificial Intelligence, ECAI 2014 - Prague, Czech Republic
Duration: 18 Aug 201422 Aug 2014

Publication series

NameFrontiers in Artificial Intelligence and Applications
Volume263
ISSN (Print)0922-6389

Other

Other21st European Conference on Artificial Intelligence, ECAI 2014
CountryCzech Republic
CityPrague
Period18/8/1422/8/14

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Qahtan, A., Wang, S., Carroll, R., & Zhang, X. (2014). A new study of two divergence metrics for change detection in data streams. In ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings (pp. 1081-1082). (Frontiers in Artificial Intelligence and Applications; Vol. 263). IOS Press. https://doi.org/10.3233/978-1-61499-419-0-1081

A new study of two divergence metrics for change detection in data streams. / Qahtan, Abdulhakim; Wang, Suojin; Carroll, Raymond; Zhang, Xiangliang.

ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings. IOS Press, 2014. p. 1081-1082 (Frontiers in Artificial Intelligence and Applications; Vol. 263).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Qahtan, A, Wang, S, Carroll, R & Zhang, X 2014, A new study of two divergence metrics for change detection in data streams. in ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings. Frontiers in Artificial Intelligence and Applications, vol. 263, IOS Press, pp. 1081-1082, 21st European Conference on Artificial Intelligence, ECAI 2014, Prague, Czech Republic, 18/8/14. https://doi.org/10.3233/978-1-61499-419-0-1081
Qahtan A, Wang S, Carroll R, Zhang X. A new study of two divergence metrics for change detection in data streams. In ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings. IOS Press. 2014. p. 1081-1082. (Frontiers in Artificial Intelligence and Applications). https://doi.org/10.3233/978-1-61499-419-0-1081
Qahtan, Abdulhakim ; Wang, Suojin ; Carroll, Raymond ; Zhang, Xiangliang. / A new study of two divergence metrics for change detection in data streams. ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings. IOS Press, 2014. pp. 1081-1082 (Frontiers in Artificial Intelligence and Applications).
@inproceedings{27c13100ddc14bca9b0957370269f4e8,
title = "A new study of two divergence metrics for change detection in data streams",
abstract = "Streaming data are dynamic in nature with frequent changes. To detect such changes, most methods measure the difference between the data distributions in a current time window and a reference window. Divergence metrics and density estimation are required to measure the difference between the data distributions. Our study shows that the Kullback-Leibler (KL) divergence, the most popular metric for comparing distributions, fails to detect certain changes due to its asymmetric property and its dependence on the variance of the data. We thus consider two metrics for detecting changes in univariate data streams: a symmetric KL-divergence and a divergence metric measuring the intersection area of two distributions. The experimental results show that these two metrics lead to more accurate results in change detection than baseline methods such as Change Finder and using conventional KL-divergence.",
author = "Abdulhakim Qahtan and Suojin Wang and Raymond Carroll and Xiangliang Zhang",
year = "2014",
month = "1",
day = "1",
doi = "10.3233/978-1-61499-419-0-1081",
language = "English",
series = "Frontiers in Artificial Intelligence and Applications",
publisher = "IOS Press",
pages = "1081--1082",
booktitle = "ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings",
address = "Netherlands",

}

TY - GEN

T1 - A new study of two divergence metrics for change detection in data streams

AU - Qahtan, Abdulhakim

AU - Wang, Suojin

AU - Carroll, Raymond

AU - Zhang, Xiangliang

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Streaming data are dynamic in nature with frequent changes. To detect such changes, most methods measure the difference between the data distributions in a current time window and a reference window. Divergence metrics and density estimation are required to measure the difference between the data distributions. Our study shows that the Kullback-Leibler (KL) divergence, the most popular metric for comparing distributions, fails to detect certain changes due to its asymmetric property and its dependence on the variance of the data. We thus consider two metrics for detecting changes in univariate data streams: a symmetric KL-divergence and a divergence metric measuring the intersection area of two distributions. The experimental results show that these two metrics lead to more accurate results in change detection than baseline methods such as Change Finder and using conventional KL-divergence.

AB - Streaming data are dynamic in nature with frequent changes. To detect such changes, most methods measure the difference between the data distributions in a current time window and a reference window. Divergence metrics and density estimation are required to measure the difference between the data distributions. Our study shows that the Kullback-Leibler (KL) divergence, the most popular metric for comparing distributions, fails to detect certain changes due to its asymmetric property and its dependence on the variance of the data. We thus consider two metrics for detecting changes in univariate data streams: a symmetric KL-divergence and a divergence metric measuring the intersection area of two distributions. The experimental results show that these two metrics lead to more accurate results in change detection than baseline methods such as Change Finder and using conventional KL-divergence.

UR - http://www.scopus.com/inward/record.url?scp=84923163047&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84923163047&partnerID=8YFLogxK

U2 - 10.3233/978-1-61499-419-0-1081

DO - 10.3233/978-1-61499-419-0-1081

M3 - Conference contribution

AN - SCOPUS:84923163047

T3 - Frontiers in Artificial Intelligence and Applications

SP - 1081

EP - 1082

BT - ECAI 2014 - 21st European Conference on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems, PAIS 2014, Proceedings

PB - IOS Press

ER -