Venus: Scalable Real-Time Spatial Queries on Microblogs with Adaptive Load Shedding

Amr Magdy, Mohamed Mokbel, Sameh Elnikety, Suman Nath, Yuxiong He

Research output: Contribution to journalArticle

12 Citations (Scopus)

Abstract

Microblogging services have become among the most popular services on the web in the last few years. This led to significant increase in data size, speed, and applications. This paper presents Venus; a system that supports real-time spatial queries on microblogs. Venus supports its queries on a spatial boundary R and a temporal boundary T , from which only the top- k microblogs are returned in the query answer based on a spatio-temporal ranking function. Supporting such queries requires Venus to digest hundreds of millions of real-time microblogs in main-memory with high rates, yet, it provides low query responses and efficient memory utilization. To this end, Venus employs: (1) an efficient in-memory spatio-temporal index that digests high rates of incoming microblogs in real time, (2) a scalable query processor that prune the search space, R and T, effectively to provide low query latency on millions of items in real time, and (3) a group of memory optimization techniques that provide system administrators with different options to save significant memory resources while keeping the query accuracy almost perfect. Venus memory optimization techniques make use of the local arrival rates of microblogs to smartly shed microblogs that are old enough not to contribute to any query answer. In addition, Venus can adaptively, in real time, adjust its load shedding based on both the spatial distribution and the parameters of incoming query loads. All Venus components can accommodate different spatial and temporal ranking functions that are able to capture the importance of each dimension differently depending on the applications requirements. Extensive experimental results based on real Twitter data and actual locations of Bing search queries show that Venus supports high arrival rates of up to 64 K microblogs/second and average query latency of 4 msec.

Original languageEnglish
Article number7303957
Pages (from-to)356-370
Number of pages15
JournalIEEE Transactions on Knowledge and Data Engineering
Volume28
Issue number2
DOIs
Publication statusPublished - 1 Feb 2016
Externally publishedYes

    Fingerprint

Keywords

  • Efficiency
  • Location
  • Memory Optimization
  • Microblogs
  • Performance
  • Scalability
  • Social
  • Spatial
  • Temporal

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics

Cite this