Mercury

A memory-constrained spatio-temporal real-time search on microblogs

Amr Magdy, Mohamed Mokbel, Sameh Elnikety, Suman Nath, Yuxiong He

Research output: Chapter in Book/Report/Conference proceedingConference contribution

30 Citations (Scopus)

Abstract

This paper presents Mercury; a system for real-time support of top-k spatio-temporal queries on microblogs, where users are able to browse recent microblogs near their locations. With high arrival rates of microblogs, Mercury ensures real-time query response within a tight memory-constrained environment. Mercury bounds its search space to include only those microblogs that have arrived within certain spatial and temporal boundaries, in which only the top-k microblogs, according to a spatio-temporal ranking function, are returned in the search results. Mercury employs: (a) a scalable dynamic in-memory index structure that is capable of digesting all incoming microblogs, (b) an efficient query processor that exploits the in-memory index through spatio-temporal pruning techniques that reduce the number of visited microblogs to return the final answer, (c) an index size tuning module that dynamically finds and adjusts the minimum index size to ensure that incoming queries will be answered accurately, and (d) a load shedding technique that trades slight decrease in query accuracy for significant storage savings. Extensive experimental results based on a real-time Twitter Firehose feed and actual locations of Bing search queries show that Mercury supports high arrival rates of up to 64K microblogs/second and average query latency of 4 msec.

Original languageEnglish
Title of host publication2014 IEEE 30th International Conference on Data Engineering, ICDE 2014
PublisherIEEE Computer Society
Pages172-183
Number of pages12
ISBN (Print)9781479925544
DOIs
Publication statusPublished - 1 Jan 2014
Externally publishedYes
Event30th IEEE International Conference on Data Engineering, ICDE 2014 - Chicago, IL, United States
Duration: 31 Mar 20144 Apr 2014

Other

Other30th IEEE International Conference on Data Engineering, ICDE 2014
CountryUnited States
CityChicago, IL
Period31/3/144/4/14

Fingerprint

Data storage equipment
Tuning

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Information Systems

Cite this

Magdy, A., Mokbel, M., Elnikety, S., Nath, S., & He, Y. (2014). Mercury: A memory-constrained spatio-temporal real-time search on microblogs. In 2014 IEEE 30th International Conference on Data Engineering, ICDE 2014 (pp. 172-183). [6816649] IEEE Computer Society. https://doi.org/10.1109/ICDE.2014.6816649

Mercury : A memory-constrained spatio-temporal real-time search on microblogs. / Magdy, Amr; Mokbel, Mohamed; Elnikety, Sameh; Nath, Suman; He, Yuxiong.

2014 IEEE 30th International Conference on Data Engineering, ICDE 2014. IEEE Computer Society, 2014. p. 172-183 6816649.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Magdy, A, Mokbel, M, Elnikety, S, Nath, S & He, Y 2014, Mercury: A memory-constrained spatio-temporal real-time search on microblogs. in 2014 IEEE 30th International Conference on Data Engineering, ICDE 2014., 6816649, IEEE Computer Society, pp. 172-183, 30th IEEE International Conference on Data Engineering, ICDE 2014, Chicago, IL, United States, 31/3/14. https://doi.org/10.1109/ICDE.2014.6816649
Magdy A, Mokbel M, Elnikety S, Nath S, He Y. Mercury: A memory-constrained spatio-temporal real-time search on microblogs. In 2014 IEEE 30th International Conference on Data Engineering, ICDE 2014. IEEE Computer Society. 2014. p. 172-183. 6816649 https://doi.org/10.1109/ICDE.2014.6816649
Magdy, Amr ; Mokbel, Mohamed ; Elnikety, Sameh ; Nath, Suman ; He, Yuxiong. / Mercury : A memory-constrained spatio-temporal real-time search on microblogs. 2014 IEEE 30th International Conference on Data Engineering, ICDE 2014. IEEE Computer Society, 2014. pp. 172-183
@inproceedings{e14cdc37979440c8aba91c616f562271,
title = "Mercury: A memory-constrained spatio-temporal real-time search on microblogs",
abstract = "This paper presents Mercury; a system for real-time support of top-k spatio-temporal queries on microblogs, where users are able to browse recent microblogs near their locations. With high arrival rates of microblogs, Mercury ensures real-time query response within a tight memory-constrained environment. Mercury bounds its search space to include only those microblogs that have arrived within certain spatial and temporal boundaries, in which only the top-k microblogs, according to a spatio-temporal ranking function, are returned in the search results. Mercury employs: (a) a scalable dynamic in-memory index structure that is capable of digesting all incoming microblogs, (b) an efficient query processor that exploits the in-memory index through spatio-temporal pruning techniques that reduce the number of visited microblogs to return the final answer, (c) an index size tuning module that dynamically finds and adjusts the minimum index size to ensure that incoming queries will be answered accurately, and (d) a load shedding technique that trades slight decrease in query accuracy for significant storage savings. Extensive experimental results based on a real-time Twitter Firehose feed and actual locations of Bing search queries show that Mercury supports high arrival rates of up to 64K microblogs/second and average query latency of 4 msec.",
author = "Amr Magdy and Mohamed Mokbel and Sameh Elnikety and Suman Nath and Yuxiong He",
year = "2014",
month = "1",
day = "1",
doi = "10.1109/ICDE.2014.6816649",
language = "English",
isbn = "9781479925544",
pages = "172--183",
booktitle = "2014 IEEE 30th International Conference on Data Engineering, ICDE 2014",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - Mercury

T2 - A memory-constrained spatio-temporal real-time search on microblogs

AU - Magdy, Amr

AU - Mokbel, Mohamed

AU - Elnikety, Sameh

AU - Nath, Suman

AU - He, Yuxiong

PY - 2014/1/1

Y1 - 2014/1/1

N2 - This paper presents Mercury; a system for real-time support of top-k spatio-temporal queries on microblogs, where users are able to browse recent microblogs near their locations. With high arrival rates of microblogs, Mercury ensures real-time query response within a tight memory-constrained environment. Mercury bounds its search space to include only those microblogs that have arrived within certain spatial and temporal boundaries, in which only the top-k microblogs, according to a spatio-temporal ranking function, are returned in the search results. Mercury employs: (a) a scalable dynamic in-memory index structure that is capable of digesting all incoming microblogs, (b) an efficient query processor that exploits the in-memory index through spatio-temporal pruning techniques that reduce the number of visited microblogs to return the final answer, (c) an index size tuning module that dynamically finds and adjusts the minimum index size to ensure that incoming queries will be answered accurately, and (d) a load shedding technique that trades slight decrease in query accuracy for significant storage savings. Extensive experimental results based on a real-time Twitter Firehose feed and actual locations of Bing search queries show that Mercury supports high arrival rates of up to 64K microblogs/second and average query latency of 4 msec.

AB - This paper presents Mercury; a system for real-time support of top-k spatio-temporal queries on microblogs, where users are able to browse recent microblogs near their locations. With high arrival rates of microblogs, Mercury ensures real-time query response within a tight memory-constrained environment. Mercury bounds its search space to include only those microblogs that have arrived within certain spatial and temporal boundaries, in which only the top-k microblogs, according to a spatio-temporal ranking function, are returned in the search results. Mercury employs: (a) a scalable dynamic in-memory index structure that is capable of digesting all incoming microblogs, (b) an efficient query processor that exploits the in-memory index through spatio-temporal pruning techniques that reduce the number of visited microblogs to return the final answer, (c) an index size tuning module that dynamically finds and adjusts the minimum index size to ensure that incoming queries will be answered accurately, and (d) a load shedding technique that trades slight decrease in query accuracy for significant storage savings. Extensive experimental results based on a real-time Twitter Firehose feed and actual locations of Bing search queries show that Mercury supports high arrival rates of up to 64K microblogs/second and average query latency of 4 msec.

UR - http://www.scopus.com/inward/record.url?scp=84901811009&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84901811009&partnerID=8YFLogxK

U2 - 10.1109/ICDE.2014.6816649

DO - 10.1109/ICDE.2014.6816649

M3 - Conference contribution

SN - 9781479925544

SP - 172

EP - 183

BT - 2014 IEEE 30th International Conference on Data Engineering, ICDE 2014

PB - IEEE Computer Society

ER -