Building and scaling virtual clusters with residual resources from interactive clouds

R. Benjamin Clay, Zhiming Shen, Xiaosong Ma

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented "background" workloads using a hybrid of residual and dedicated resources. For a given background job, RHIC intelligently discovers/maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy under bursty and heterogeneous residual resources. Our results show that RHIC finds near-ideal cluster sizes/compositions across a wide range of workload/goal combinations, significantly outperforms alternative approaches, tolerates high instability in the harvested interactive cloud, works with heterogeneous hardware and imposes minimal overhead.

Original languageEnglish
Title of host publicationHPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing
Pages119-120
Number of pages2
DOIs
Publication statusPublished - 17 Jul 2013
Externally publishedYes
Event22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013 - New York, NY, United States
Duration: 17 Jun 201321 Jun 2013

Other

Other22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013
CountryUnited States
CityNew York, NY
Period17/6/1321/6/13

Fingerprint

Harvesters
Chemical analysis
Consolidation
Throughput
Hardware
Costs

Keywords

  • cloud computing
  • resource harvesting
  • volunteer computing

ASJC Scopus subject areas

  • Software

Cite this

Clay, R. B., Shen, Z., & Ma, X. (2013). Building and scaling virtual clusters with residual resources from interactive clouds. In HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing (pp. 119-120) https://doi.org/10.1145/2462902.2462927

Building and scaling virtual clusters with residual resources from interactive clouds. / Clay, R. Benjamin; Shen, Zhiming; Ma, Xiaosong.

HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing. 2013. p. 119-120.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Clay, RB, Shen, Z & Ma, X 2013, Building and scaling virtual clusters with residual resources from interactive clouds. in HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing. pp. 119-120, 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013, New York, NY, United States, 17/6/13. https://doi.org/10.1145/2462902.2462927
Clay RB, Shen Z, Ma X. Building and scaling virtual clusters with residual resources from interactive clouds. In HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing. 2013. p. 119-120 https://doi.org/10.1145/2462902.2462927
Clay, R. Benjamin ; Shen, Zhiming ; Ma, Xiaosong. / Building and scaling virtual clusters with residual resources from interactive clouds. HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing. 2013. pp. 119-120
@inproceedings{747718a69031442db02f0cf947c78237,
title = "Building and scaling virtual clusters with residual resources from interactive clouds",
abstract = "The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented {"}background{"} workloads using a hybrid of residual and dedicated resources. For a given background job, RHIC intelligently discovers/maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy under bursty and heterogeneous residual resources. Our results show that RHIC finds near-ideal cluster sizes/compositions across a wide range of workload/goal combinations, significantly outperforms alternative approaches, tolerates high instability in the harvested interactive cloud, works with heterogeneous hardware and imposes minimal overhead.",
keywords = "cloud computing, resource harvesting, volunteer computing",
author = "Clay, {R. Benjamin} and Zhiming Shen and Xiaosong Ma",
year = "2013",
month = "7",
day = "17",
doi = "10.1145/2462902.2462927",
language = "English",
pages = "119--120",
booktitle = "HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing",

}

TY - GEN

T1 - Building and scaling virtual clusters with residual resources from interactive clouds

AU - Clay, R. Benjamin

AU - Shen, Zhiming

AU - Ma, Xiaosong

PY - 2013/7/17

Y1 - 2013/7/17

N2 - The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented "background" workloads using a hybrid of residual and dedicated resources. For a given background job, RHIC intelligently discovers/maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy under bursty and heterogeneous residual resources. Our results show that RHIC finds near-ideal cluster sizes/compositions across a wide range of workload/goal combinations, significantly outperforms alternative approaches, tolerates high instability in the harvested interactive cloud, works with heterogeneous hardware and imposes minimal overhead.

AB - The popularity of cloud-based interactive computing services (e.g., virtual desktops) brings new management challenges. Each interactive user leaves abundant but fluctuating residual resources while being intolerant to latency, precluding the use of aggressive VM consolidation. In this paper, we present the Resource Harvester for Interactive Clouds (RHIC), an autonomous management framework that harnesses dynamic residual resources aggressively without slowing the harvested interactive services. RHIC builds ad-hoc clusters for running throughput-oriented "background" workloads using a hybrid of residual and dedicated resources. For a given background job, RHIC intelligently discovers/maintains the ideal cluster size and composition, to meet user-specified goals such as cost/energy minimization or deadlines. RHIC employs black-box workload performance modeling, requiring only system-level metrics and incorporating techniques to improve modeling accuracy under bursty and heterogeneous residual resources. Our results show that RHIC finds near-ideal cluster sizes/compositions across a wide range of workload/goal combinations, significantly outperforms alternative approaches, tolerates high instability in the harvested interactive cloud, works with heterogeneous hardware and imposes minimal overhead.

KW - cloud computing

KW - resource harvesting

KW - volunteer computing

UR - http://www.scopus.com/inward/record.url?scp=84880062790&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84880062790&partnerID=8YFLogxK

U2 - 10.1145/2462902.2462927

DO - 10.1145/2462902.2462927

M3 - Conference contribution

SP - 119

EP - 120

BT - HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing

ER -