LocationSpark: A distributed in-memory data management system for big spatial data

Mingjie Tangy, Yongyang Yuy, Qutaibah M. Malluhiz, Mourad Ouzzani, Walid G. Arefy

Research output: Contribution to journalConference article

53 Citations (Scopus)

Abstract

We present LocationSpark, a spatial data processing system built on top of Apache Spark, a widely used distributed data processing system. LocationSpark offers a rich set of spatial query operators, e.g., range search, kNN, spatio-textual operation, spatial-join, and kNN-join. To achieve high performance, LocationSpark employs various spatial indexes for in-memory data, and guarantees that immutable spatial indexes have low overhead with fault tolerance. In addition, we build two new layers over Spark, namely a query scheduler and a query executor. The query scheduler is responsible for mitigating skew in spatial queries, while the query executor selects the best plan based on the indexes and the nature of the spatial queries. Furthermore, to avoid unnecessary network communication overhead when processing overlapped spatial data, We embed an efficient spatial Bloom filter into LocationSpark's indexes. Finally, LocationSpark tracks frequently accessed spatial data, and dynamically ushes less frequently accessed data into disk. We evaluate our system on real workloads and demonstrate that it achieves an order of magnitude performance gain over a baseline framework.

Original languageEnglish
Pages (from-to)1565-1568
Number of pages4
JournalProceedings of the VLDB Endowment
Volume9
Issue number13
Publication statusPublished - 1 Jan 2015

    Fingerprint

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Cite this