MD-HBase

A scalable multi-dimensional data infrastructure for location aware services

Shoji Nishimura, Sudipto Das, Divyakant Agrawal, Amr El Abbadi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

138 Citations (Scopus)

Abstract

The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, Key-value stores can effectively support large scale operation, but do not natively support multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present MD-HBase, a scalable data management system for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a Key-value store. The underlying Key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. We present the design of MD-HBase that builds two standard index structures - "the K-d tree and the Quad tree - "over a range partitioned Key-value store. Our prototype implementation using HBase, a standard open-source Key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multidimensional range queries and nearest neighbor queries in real-time with response times as low as hundreds of milliseconds.

Original languageEnglish
Title of host publicationProceedings - IEEE International Conference on Mobile Data Management
Pages7-16
Number of pages10
Volume1
DOIs
Publication statusPublished - 29 Nov 2011
Externally publishedYes
Event2011 12th IEEE International Conference on Mobile Data Management, MDM 2011 - Lulea, Sweden
Duration: 6 Jun 20119 Jun 2011

Other

Other2011 12th IEEE International Conference on Mobile Data Management, MDM 2011
CountrySweden
CityLulea
Period6/6/119/6/11

Fingerprint

Relational database systems
Location based services
Query processing
Fault tolerance
Information management
Throughput
Availability
Processing

Keywords

  • key value stores
  • location based services
  • multidimensional data
  • real time analysis

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Nishimura, S., Das, S., Agrawal, D., & El Abbadi, A. (2011). MD-HBase: A scalable multi-dimensional data infrastructure for location aware services. In Proceedings - IEEE International Conference on Mobile Data Management (Vol. 1, pp. 7-16). [6068416] https://doi.org/10.1109/MDM.2011.41

MD-HBase : A scalable multi-dimensional data infrastructure for location aware services. / Nishimura, Shoji; Das, Sudipto; Agrawal, Divyakant; El Abbadi, Amr.

Proceedings - IEEE International Conference on Mobile Data Management. Vol. 1 2011. p. 7-16 6068416.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Nishimura, S, Das, S, Agrawal, D & El Abbadi, A 2011, MD-HBase: A scalable multi-dimensional data infrastructure for location aware services. in Proceedings - IEEE International Conference on Mobile Data Management. vol. 1, 6068416, pp. 7-16, 2011 12th IEEE International Conference on Mobile Data Management, MDM 2011, Lulea, Sweden, 6/6/11. https://doi.org/10.1109/MDM.2011.41
Nishimura S, Das S, Agrawal D, El Abbadi A. MD-HBase: A scalable multi-dimensional data infrastructure for location aware services. In Proceedings - IEEE International Conference on Mobile Data Management. Vol. 1. 2011. p. 7-16. 6068416 https://doi.org/10.1109/MDM.2011.41
Nishimura, Shoji ; Das, Sudipto ; Agrawal, Divyakant ; El Abbadi, Amr. / MD-HBase : A scalable multi-dimensional data infrastructure for location aware services. Proceedings - IEEE International Conference on Mobile Data Management. Vol. 1 2011. pp. 7-16
@inproceedings{bf00dc14d31e4623bd795d328eb35c8d,
title = "MD-HBase: A scalable multi-dimensional data infrastructure for location aware services",
abstract = "The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, Key-value stores can effectively support large scale operation, but do not natively support multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present MD-HBase, a scalable data management system for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a Key-value store. The underlying Key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. We present the design of MD-HBase that builds two standard index structures - {"}the K-d tree and the Quad tree - {"}over a range partitioned Key-value store. Our prototype implementation using HBase, a standard open-source Key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multidimensional range queries and nearest neighbor queries in real-time with response times as low as hundreds of milliseconds.",
keywords = "key value stores, location based services, multidimensional data, real time analysis",
author = "Shoji Nishimura and Sudipto Das and Divyakant Agrawal and {El Abbadi}, Amr",
year = "2011",
month = "11",
day = "29",
doi = "10.1109/MDM.2011.41",
language = "English",
isbn = "9780769544366",
volume = "1",
pages = "7--16",
booktitle = "Proceedings - IEEE International Conference on Mobile Data Management",

}

TY - GEN

T1 - MD-HBase

T2 - A scalable multi-dimensional data infrastructure for location aware services

AU - Nishimura, Shoji

AU - Das, Sudipto

AU - Agrawal, Divyakant

AU - El Abbadi, Amr

PY - 2011/11/29

Y1 - 2011/11/29

N2 - The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, Key-value stores can effectively support large scale operation, but do not natively support multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present MD-HBase, a scalable data management system for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a Key-value store. The underlying Key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. We present the design of MD-HBase that builds two standard index structures - "the K-d tree and the Quad tree - "over a range partitioned Key-value store. Our prototype implementation using HBase, a standard open-source Key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multidimensional range queries and nearest neighbor queries in real-time with response times as low as hundreds of milliseconds.

AB - The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, Key-value stores can effectively support large scale operation, but do not natively support multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present MD-HBase, a scalable data management system for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a Key-value store. The underlying Key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. We present the design of MD-HBase that builds two standard index structures - "the K-d tree and the Quad tree - "over a range partitioned Key-value store. Our prototype implementation using HBase, a standard open-source Key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multidimensional range queries and nearest neighbor queries in real-time with response times as low as hundreds of milliseconds.

KW - key value stores

KW - location based services

KW - multidimensional data

KW - real time analysis

UR - http://www.scopus.com/inward/record.url?scp=82055165170&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=82055165170&partnerID=8YFLogxK

U2 - 10.1109/MDM.2011.41

DO - 10.1109/MDM.2011.41

M3 - Conference contribution

SN - 9780769544366

VL - 1

SP - 7

EP - 16

BT - Proceedings - IEEE International Conference on Mobile Data Management

ER -