Accelerating batch analytics with residual resources from interactive clouds

R. Benjamin Clay, Zhiming Shen, Xiaosong Ma

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented 'background' workloads using a hybrid of residual and dedicated resources. These hybrid clusters offer significant gains over normal dedicated clusters: 20-40% cost and 20-29% energy savings in our test bed. For a given background job, RHIC intelligently discovers and maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy with bursty and heterogeneous residual resources. We demonstrate the effectiveness and adaptivity of our RHIC prototype with two parallel data analytics frameworks, Hadoop and HBase. Our results show that RHIC finds near-ideal cluster sizes and compositions across a wide range of workload/goal combinations.

Original languageEnglish
Title of host publicationProceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS
Pages414-423
Number of pages10
DOIs
Publication statusPublished - 1 Dec 2013
Externally publishedYes
Event2013 IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication, MASCOTS 2013 - San Francisco, CA, United States
Duration: 14 Aug 201316 Aug 2013

Other

Other2013 IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication, MASCOTS 2013
CountryUnited States
CitySan Francisco, CA
Period14/8/1316/8/13

Fingerprint

Harvesters
Batch
Resources
Workload
Chemical analysis
Consolidation
Costs
Energy conservation
Cost Minimization
Throughput
Performance Modeling
Energy Minimization
Adaptivity
Deadline
Energy Saving
Black Box
Testbed
Latency
Prototype
Metric

Keywords

  • Adaptive systems
  • Distributed computing
  • Performance analysis

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Computer Networks and Communications
  • Software
  • Modelling and Simulation

Cite this

Clay, R. B., Shen, Z., & Ma, X. (2013). Accelerating batch analytics with residual resources from interactive clouds. In Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS (pp. 414-423). [6730798] https://doi.org/10.1109/MASCOTS.2013.63

Accelerating batch analytics with residual resources from interactive clouds. / Clay, R. Benjamin; Shen, Zhiming; Ma, Xiaosong.

Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS. 2013. p. 414-423 6730798.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Clay, RB, Shen, Z & Ma, X 2013, Accelerating batch analytics with residual resources from interactive clouds. in Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS., 6730798, pp. 414-423, 2013 IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication, MASCOTS 2013, San Francisco, CA, United States, 14/8/13. https://doi.org/10.1109/MASCOTS.2013.63
Clay RB, Shen Z, Ma X. Accelerating batch analytics with residual resources from interactive clouds. In Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS. 2013. p. 414-423. 6730798 https://doi.org/10.1109/MASCOTS.2013.63
Clay, R. Benjamin ; Shen, Zhiming ; Ma, Xiaosong. / Accelerating batch analytics with residual resources from interactive clouds. Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS. 2013. pp. 414-423
@inproceedings{19ce450437d34958b363c675e8959a0c,
title = "Accelerating batch analytics with residual resources from interactive clouds",
abstract = "The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented 'background' workloads using a hybrid of residual and dedicated resources. These hybrid clusters offer significant gains over normal dedicated clusters: 20-40{\%} cost and 20-29{\%} energy savings in our test bed. For a given background job, RHIC intelligently discovers and maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy with bursty and heterogeneous residual resources. We demonstrate the effectiveness and adaptivity of our RHIC prototype with two parallel data analytics frameworks, Hadoop and HBase. Our results show that RHIC finds near-ideal cluster sizes and compositions across a wide range of workload/goal combinations.",
keywords = "Adaptive systems, Distributed computing, Performance analysis",
author = "Clay, {R. Benjamin} and Zhiming Shen and Xiaosong Ma",
year = "2013",
month = "12",
day = "1",
doi = "10.1109/MASCOTS.2013.63",
language = "English",
isbn = "9780769551029",
pages = "414--423",
booktitle = "Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS",

}

TY - GEN

T1 - Accelerating batch analytics with residual resources from interactive clouds

AU - Clay, R. Benjamin

AU - Shen, Zhiming

AU - Ma, Xiaosong

PY - 2013/12/1

Y1 - 2013/12/1

N2 - The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented 'background' workloads using a hybrid of residual and dedicated resources. These hybrid clusters offer significant gains over normal dedicated clusters: 20-40% cost and 20-29% energy savings in our test bed. For a given background job, RHIC intelligently discovers and maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy with bursty and heterogeneous residual resources. We demonstrate the effectiveness and adaptivity of our RHIC prototype with two parallel data analytics frameworks, Hadoop and HBase. Our results show that RHIC finds near-ideal cluster sizes and compositions across a wide range of workload/goal combinations.

AB - The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented 'background' workloads using a hybrid of residual and dedicated resources. These hybrid clusters offer significant gains over normal dedicated clusters: 20-40% cost and 20-29% energy savings in our test bed. For a given background job, RHIC intelligently discovers and maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy with bursty and heterogeneous residual resources. We demonstrate the effectiveness and adaptivity of our RHIC prototype with two parallel data analytics frameworks, Hadoop and HBase. Our results show that RHIC finds near-ideal cluster sizes and compositions across a wide range of workload/goal combinations.

KW - Adaptive systems

KW - Distributed computing

KW - Performance analysis

UR - http://www.scopus.com/inward/record.url?scp=84894578522&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84894578522&partnerID=8YFLogxK

U2 - 10.1109/MASCOTS.2013.63

DO - 10.1109/MASCOTS.2013.63

M3 - Conference contribution

AN - SCOPUS:84894578522

SN - 9780769551029

SP - 414

EP - 423

BT - Proceedings - IEEE Computer Society's Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, MASCOTS

ER -