An integrated efficient solution for computing frequent and top-k elements in data streams

Ahmed Metwally, Divyakant Agrawal, Amr El Abbadi

Research output: Contribution to journalArticle

122 Citations (Scopus)

Abstract

We propose an approximate integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream coming from a large domain. Our solution is space efficient and reports both frequent and top-k elements with tight guarantees on errors. For general data distributions, our top-k algorithm returns k elements that have roughly the highest frequencies; and it uses limited space for calculating frequent elements. For realistic Zipfian data, the space requirement of the proposed algorithm for solving the exact frequent elements problem decreases dramatically with the parameter of the distribution; and for top-k queries, the analysis ensures that only the top-k elements, in the correct order, are reported. The experiments, using real and synthetic data sets, show space reductions with hardly any loss in accuracy. Having proved the effectiveness of the proposed approach through both analysis and experiments, we extend it to be able to answer continuous queries about frequent and top-k elements. Although the problems of incremental reporting of frequent and top-k elements are useful in many applications, to the best of our knowledge, no solution has been proposed.

Original languageEnglish
Pages (from-to)1095-1133
Number of pages39
JournalACM Transactions on Database Systems
Volume31
Issue number3
DOIs
Publication statusPublished - 25 Oct 2006
Externally publishedYes

Fingerprint

Experiments

Keywords

  • Advertising networks
  • Approximate queries
  • Continuous queries
  • Data streams
  • Exact queries
  • Frequent elements
  • Top-k elements
  • Zipfian data

ASJC Scopus subject areas

  • Information Systems
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

An integrated efficient solution for computing frequent and top-k elements in data streams. / Metwally, Ahmed; Agrawal, Divyakant; El Abbadi, Amr.

In: ACM Transactions on Database Systems, Vol. 31, No. 3, 25.10.2006, p. 1095-1133.

Research output: Contribution to journalArticle

Metwally, Ahmed ; Agrawal, Divyakant ; El Abbadi, Amr. / An integrated efficient solution for computing frequent and top-k elements in data streams. In: ACM Transactions on Database Systems. 2006 ; Vol. 31, No. 3. pp. 1095-1133.
@article{d6a64b64407b4aa1b91c5a7758c44953,
title = "An integrated efficient solution for computing frequent and top-k elements in data streams",
abstract = "We propose an approximate integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream coming from a large domain. Our solution is space efficient and reports both frequent and top-k elements with tight guarantees on errors. For general data distributions, our top-k algorithm returns k elements that have roughly the highest frequencies; and it uses limited space for calculating frequent elements. For realistic Zipfian data, the space requirement of the proposed algorithm for solving the exact frequent elements problem decreases dramatically with the parameter of the distribution; and for top-k queries, the analysis ensures that only the top-k elements, in the correct order, are reported. The experiments, using real and synthetic data sets, show space reductions with hardly any loss in accuracy. Having proved the effectiveness of the proposed approach through both analysis and experiments, we extend it to be able to answer continuous queries about frequent and top-k elements. Although the problems of incremental reporting of frequent and top-k elements are useful in many applications, to the best of our knowledge, no solution has been proposed.",
keywords = "Advertising networks, Approximate queries, Continuous queries, Data streams, Exact queries, Frequent elements, Top-k elements, Zipfian data",
author = "Ahmed Metwally and Divyakant Agrawal and {El Abbadi}, Amr",
year = "2006",
month = "10",
day = "25",
doi = "10.1145/1166074.1166084",
language = "English",
volume = "31",
pages = "1095--1133",
journal = "ACM Transactions on Database Systems",
issn = "0362-5915",
publisher = "Association for Computing Machinery (ACM)",
number = "3",

}

TY - JOUR

T1 - An integrated efficient solution for computing frequent and top-k elements in data streams

AU - Metwally, Ahmed

AU - Agrawal, Divyakant

AU - El Abbadi, Amr

PY - 2006/10/25

Y1 - 2006/10/25

N2 - We propose an approximate integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream coming from a large domain. Our solution is space efficient and reports both frequent and top-k elements with tight guarantees on errors. For general data distributions, our top-k algorithm returns k elements that have roughly the highest frequencies; and it uses limited space for calculating frequent elements. For realistic Zipfian data, the space requirement of the proposed algorithm for solving the exact frequent elements problem decreases dramatically with the parameter of the distribution; and for top-k queries, the analysis ensures that only the top-k elements, in the correct order, are reported. The experiments, using real and synthetic data sets, show space reductions with hardly any loss in accuracy. Having proved the effectiveness of the proposed approach through both analysis and experiments, we extend it to be able to answer continuous queries about frequent and top-k elements. Although the problems of incremental reporting of frequent and top-k elements are useful in many applications, to the best of our knowledge, no solution has been proposed.

AB - We propose an approximate integrated approach for solving both problems of finding the most popular k elements, and finding frequent elements in a data stream coming from a large domain. Our solution is space efficient and reports both frequent and top-k elements with tight guarantees on errors. For general data distributions, our top-k algorithm returns k elements that have roughly the highest frequencies; and it uses limited space for calculating frequent elements. For realistic Zipfian data, the space requirement of the proposed algorithm for solving the exact frequent elements problem decreases dramatically with the parameter of the distribution; and for top-k queries, the analysis ensures that only the top-k elements, in the correct order, are reported. The experiments, using real and synthetic data sets, show space reductions with hardly any loss in accuracy. Having proved the effectiveness of the proposed approach through both analysis and experiments, we extend it to be able to answer continuous queries about frequent and top-k elements. Although the problems of incremental reporting of frequent and top-k elements are useful in many applications, to the best of our knowledge, no solution has been proposed.

KW - Advertising networks

KW - Approximate queries

KW - Continuous queries

KW - Data streams

KW - Exact queries

KW - Frequent elements

KW - Top-k elements

KW - Zipfian data

UR - http://www.scopus.com/inward/record.url?scp=33750185779&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33750185779&partnerID=8YFLogxK

U2 - 10.1145/1166074.1166084

DO - 10.1145/1166074.1166084

M3 - Article

VL - 31

SP - 1095

EP - 1133

JO - ACM Transactions on Database Systems

JF - ACM Transactions on Database Systems

SN - 0362-5915

IS - 3

ER -