Squall

Fine-grained live reconfiguration for partitioned main memory databases

Aaron J. Elmore, Vaibhav Arora, Rebecca Taft, Andrew Pavlo, Divyakant Agrawal, Amr El Abbadi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

For data-intensive applications with many concurrent users, modern distributed main memory database management systems (DBMS) provide the necessary scale-out support beyond what is possible with single-node systems. These DBMSs are optimized for the short-lived transactions that are common in on-line transaction processing (OLTP) workloads. One way that they achieve this is to partition the database into disjoint subsets and use a single-threaded transaction manager per partition that executes transactions one-atatime in serial order. This minimizes the overhead of concurrency control mechanisms, but requires careful partitioning to limit distributed transactions that span multiple partitions. Previous methods used off-line analysis to determine how to partition data, but the dynamic nature of these applications means that they are prone to hotspots. In these situations, the DBMS needs to reconfigure how data is partitioned in real-time to maintain performance objectives. Bringing the system off-line to reorganize the database is unacceptable for on-line applications. To overcome this problem, we introduce the Squall technique for supporting live reconfiguration in partitioned, main memory DBMSs. Squall supports fine-grained repartitioning of databases in the presence of distributed transactions, high throughput client workloads, and replicated data. An evaluation of our approach on a distributed DBMS shows that Squall can reconfigure a database with no downtime and minimal overhead on transaction latency.

Original languageEnglish
Title of host publicationProceedings of the ACM SIGMOD International Conference on Management of Data
PublisherAssociation for Computing Machinery
Pages299-313
Number of pages15
Volume2015-May
ISBN (Print)9781450327589
DOIs
Publication statusPublished - 27 May 2015
EventACM SIGMOD International Conference on Management of Data, SIGMOD 2015 - Melbourne, Australia
Duration: 31 May 20154 Jun 2015

Other

OtherACM SIGMOD International Conference on Management of Data, SIGMOD 2015
CountryAustralia
CityMelbourne
Period31/5/154/6/15

Fingerprint

Data storage equipment
Concurrency control
Managers
Throughput
Processing

ASJC Scopus subject areas

  • Software
  • Information Systems

Cite this

Elmore, A. J., Arora, V., Taft, R., Pavlo, A., Agrawal, D., & El Abbadi, A. (2015). Squall: Fine-grained live reconfiguration for partitioned main memory databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data (Vol. 2015-May, pp. 299-313). Association for Computing Machinery. https://doi.org/10.1145/2723372.2723726

Squall : Fine-grained live reconfiguration for partitioned main memory databases. / Elmore, Aaron J.; Arora, Vaibhav; Taft, Rebecca; Pavlo, Andrew; Agrawal, Divyakant; El Abbadi, Amr.

Proceedings of the ACM SIGMOD International Conference on Management of Data. Vol. 2015-May Association for Computing Machinery, 2015. p. 299-313.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Elmore, AJ, Arora, V, Taft, R, Pavlo, A, Agrawal, D & El Abbadi, A 2015, Squall: Fine-grained live reconfiguration for partitioned main memory databases. in Proceedings of the ACM SIGMOD International Conference on Management of Data. vol. 2015-May, Association for Computing Machinery, pp. 299-313, ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, Melbourne, Australia, 31/5/15. https://doi.org/10.1145/2723372.2723726
Elmore AJ, Arora V, Taft R, Pavlo A, Agrawal D, El Abbadi A. Squall: Fine-grained live reconfiguration for partitioned main memory databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data. Vol. 2015-May. Association for Computing Machinery. 2015. p. 299-313 https://doi.org/10.1145/2723372.2723726
Elmore, Aaron J. ; Arora, Vaibhav ; Taft, Rebecca ; Pavlo, Andrew ; Agrawal, Divyakant ; El Abbadi, Amr. / Squall : Fine-grained live reconfiguration for partitioned main memory databases. Proceedings of the ACM SIGMOD International Conference on Management of Data. Vol. 2015-May Association for Computing Machinery, 2015. pp. 299-313
@inproceedings{7c82394a39db4f1c9ad1cbdf99aafa29,
title = "Squall: Fine-grained live reconfiguration for partitioned main memory databases",
abstract = "For data-intensive applications with many concurrent users, modern distributed main memory database management systems (DBMS) provide the necessary scale-out support beyond what is possible with single-node systems. These DBMSs are optimized for the short-lived transactions that are common in on-line transaction processing (OLTP) workloads. One way that they achieve this is to partition the database into disjoint subsets and use a single-threaded transaction manager per partition that executes transactions one-atatime in serial order. This minimizes the overhead of concurrency control mechanisms, but requires careful partitioning to limit distributed transactions that span multiple partitions. Previous methods used off-line analysis to determine how to partition data, but the dynamic nature of these applications means that they are prone to hotspots. In these situations, the DBMS needs to reconfigure how data is partitioned in real-time to maintain performance objectives. Bringing the system off-line to reorganize the database is unacceptable for on-line applications. To overcome this problem, we introduce the Squall technique for supporting live reconfiguration in partitioned, main memory DBMSs. Squall supports fine-grained repartitioning of databases in the presence of distributed transactions, high throughput client workloads, and replicated data. An evaluation of our approach on a distributed DBMS shows that Squall can reconfigure a database with no downtime and minimal overhead on transaction latency.",
author = "Elmore, {Aaron J.} and Vaibhav Arora and Rebecca Taft and Andrew Pavlo and Divyakant Agrawal and {El Abbadi}, Amr",
year = "2015",
month = "5",
day = "27",
doi = "10.1145/2723372.2723726",
language = "English",
isbn = "9781450327589",
volume = "2015-May",
pages = "299--313",
booktitle = "Proceedings of the ACM SIGMOD International Conference on Management of Data",
publisher = "Association for Computing Machinery",

}

TY - GEN

T1 - Squall

T2 - Fine-grained live reconfiguration for partitioned main memory databases

AU - Elmore, Aaron J.

AU - Arora, Vaibhav

AU - Taft, Rebecca

AU - Pavlo, Andrew

AU - Agrawal, Divyakant

AU - El Abbadi, Amr

PY - 2015/5/27

Y1 - 2015/5/27

N2 - For data-intensive applications with many concurrent users, modern distributed main memory database management systems (DBMS) provide the necessary scale-out support beyond what is possible with single-node systems. These DBMSs are optimized for the short-lived transactions that are common in on-line transaction processing (OLTP) workloads. One way that they achieve this is to partition the database into disjoint subsets and use a single-threaded transaction manager per partition that executes transactions one-atatime in serial order. This minimizes the overhead of concurrency control mechanisms, but requires careful partitioning to limit distributed transactions that span multiple partitions. Previous methods used off-line analysis to determine how to partition data, but the dynamic nature of these applications means that they are prone to hotspots. In these situations, the DBMS needs to reconfigure how data is partitioned in real-time to maintain performance objectives. Bringing the system off-line to reorganize the database is unacceptable for on-line applications. To overcome this problem, we introduce the Squall technique for supporting live reconfiguration in partitioned, main memory DBMSs. Squall supports fine-grained repartitioning of databases in the presence of distributed transactions, high throughput client workloads, and replicated data. An evaluation of our approach on a distributed DBMS shows that Squall can reconfigure a database with no downtime and minimal overhead on transaction latency.

AB - For data-intensive applications with many concurrent users, modern distributed main memory database management systems (DBMS) provide the necessary scale-out support beyond what is possible with single-node systems. These DBMSs are optimized for the short-lived transactions that are common in on-line transaction processing (OLTP) workloads. One way that they achieve this is to partition the database into disjoint subsets and use a single-threaded transaction manager per partition that executes transactions one-atatime in serial order. This minimizes the overhead of concurrency control mechanisms, but requires careful partitioning to limit distributed transactions that span multiple partitions. Previous methods used off-line analysis to determine how to partition data, but the dynamic nature of these applications means that they are prone to hotspots. In these situations, the DBMS needs to reconfigure how data is partitioned in real-time to maintain performance objectives. Bringing the system off-line to reorganize the database is unacceptable for on-line applications. To overcome this problem, we introduce the Squall technique for supporting live reconfiguration in partitioned, main memory DBMSs. Squall supports fine-grained repartitioning of databases in the presence of distributed transactions, high throughput client workloads, and replicated data. An evaluation of our approach on a distributed DBMS shows that Squall can reconfigure a database with no downtime and minimal overhead on transaction latency.

UR - http://www.scopus.com/inward/record.url?scp=84957547819&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84957547819&partnerID=8YFLogxK

U2 - 10.1145/2723372.2723726

DO - 10.1145/2723372.2723726

M3 - Conference contribution

SN - 9781450327589

VL - 2015-May

SP - 299

EP - 313

BT - Proceedings of the ACM SIGMOD International Conference on Management of Data

PB - Association for Computing Machinery

ER -