Positioning dynamic storage caches for transient data

Sudharshan S. Vazhkudai, Douglas Thain, Xiaosong Ma, Vincent W. Freeh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Simulations, experiments and observatories are generating a deluge of scientific data. Even more staggering is the ever growing application demand to process and assimilate these datasets. Application users perform a range of data operations, collaborate and share data in many novel ways. The current storage landscape is struggling to keep up with these trends in scientific data processing. Application users pay the price due to over-crowded sharedfilesystems, or expensive storage area networks, or not enough local storage, or high-latency archival or wide-area transfers. In order to sustain and maximize I/O bandwidth relative to increasing CPU speeds, applications must take advantage of large amounts of intermediate commodity storage, However, intermediate storage presents new challenges above and beyond the traditional distributed filesystem paradigm: persistent scheduling, storage/CPU coallocation, namespace management, lifetime management, and novel application interfaces. In this paper, we describe applications that require intermediate storage management, suggest several open research problems, and illustrate two systems - Freeloader and Tactical Storage - that attack different aspects of these problems.

Original languageEnglish
Title of host publicationProceedings - IEEE International Conference on Cluster Computing, ICCC
DOIs
Publication statusPublished - 1 Dec 2006
Externally publishedYes
Event2006 IEEE International Conference on Cluster Computing, Cluster 2006 - Barcelona, Spain
Duration: 25 Sep 200628 Sep 2006

Other

Other2006 IEEE International Conference on Cluster Computing, Cluster 2006
CountrySpain
CityBarcelona
Period25/9/0628/9/06

Fingerprint

Dynamic positioning
Program processors
Storage management
Observatories
Scheduling
Bandwidth

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Vazhkudai, S. S., Thain, D., Ma, X., & Freeh, V. W. (2006). Positioning dynamic storage caches for transient data. In Proceedings - IEEE International Conference on Cluster Computing, ICCC [4100406] https://doi.org/10.1109/CLUSTR.2006.311900

Positioning dynamic storage caches for transient data. / Vazhkudai, Sudharshan S.; Thain, Douglas; Ma, Xiaosong; Freeh, Vincent W.

Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2006. 4100406.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Vazhkudai, SS, Thain, D, Ma, X & Freeh, VW 2006, Positioning dynamic storage caches for transient data. in Proceedings - IEEE International Conference on Cluster Computing, ICCC., 4100406, 2006 IEEE International Conference on Cluster Computing, Cluster 2006, Barcelona, Spain, 25/9/06. https://doi.org/10.1109/CLUSTR.2006.311900
Vazhkudai SS, Thain D, Ma X, Freeh VW. Positioning dynamic storage caches for transient data. In Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2006. 4100406 https://doi.org/10.1109/CLUSTR.2006.311900
Vazhkudai, Sudharshan S. ; Thain, Douglas ; Ma, Xiaosong ; Freeh, Vincent W. / Positioning dynamic storage caches for transient data. Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2006.
@inproceedings{fba516e60a43450aa669b0002e4eb95f,
title = "Positioning dynamic storage caches for transient data",
abstract = "Simulations, experiments and observatories are generating a deluge of scientific data. Even more staggering is the ever growing application demand to process and assimilate these datasets. Application users perform a range of data operations, collaborate and share data in many novel ways. The current storage landscape is struggling to keep up with these trends in scientific data processing. Application users pay the price due to over-crowded sharedfilesystems, or expensive storage area networks, or not enough local storage, or high-latency archival or wide-area transfers. In order to sustain and maximize I/O bandwidth relative to increasing CPU speeds, applications must take advantage of large amounts of intermediate commodity storage, However, intermediate storage presents new challenges above and beyond the traditional distributed filesystem paradigm: persistent scheduling, storage/CPU coallocation, namespace management, lifetime management, and novel application interfaces. In this paper, we describe applications that require intermediate storage management, suggest several open research problems, and illustrate two systems - Freeloader and Tactical Storage - that attack different aspects of these problems.",
author = "Vazhkudai, {Sudharshan S.} and Douglas Thain and Xiaosong Ma and Freeh, {Vincent W.}",
year = "2006",
month = "12",
day = "1",
doi = "10.1109/CLUSTR.2006.311900",
language = "English",
isbn = "1424403286",
booktitle = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

}

TY - GEN

T1 - Positioning dynamic storage caches for transient data

AU - Vazhkudai, Sudharshan S.

AU - Thain, Douglas

AU - Ma, Xiaosong

AU - Freeh, Vincent W.

PY - 2006/12/1

Y1 - 2006/12/1

N2 - Simulations, experiments and observatories are generating a deluge of scientific data. Even more staggering is the ever growing application demand to process and assimilate these datasets. Application users perform a range of data operations, collaborate and share data in many novel ways. The current storage landscape is struggling to keep up with these trends in scientific data processing. Application users pay the price due to over-crowded sharedfilesystems, or expensive storage area networks, or not enough local storage, or high-latency archival or wide-area transfers. In order to sustain and maximize I/O bandwidth relative to increasing CPU speeds, applications must take advantage of large amounts of intermediate commodity storage, However, intermediate storage presents new challenges above and beyond the traditional distributed filesystem paradigm: persistent scheduling, storage/CPU coallocation, namespace management, lifetime management, and novel application interfaces. In this paper, we describe applications that require intermediate storage management, suggest several open research problems, and illustrate two systems - Freeloader and Tactical Storage - that attack different aspects of these problems.

AB - Simulations, experiments and observatories are generating a deluge of scientific data. Even more staggering is the ever growing application demand to process and assimilate these datasets. Application users perform a range of data operations, collaborate and share data in many novel ways. The current storage landscape is struggling to keep up with these trends in scientific data processing. Application users pay the price due to over-crowded sharedfilesystems, or expensive storage area networks, or not enough local storage, or high-latency archival or wide-area transfers. In order to sustain and maximize I/O bandwidth relative to increasing CPU speeds, applications must take advantage of large amounts of intermediate commodity storage, However, intermediate storage presents new challenges above and beyond the traditional distributed filesystem paradigm: persistent scheduling, storage/CPU coallocation, namespace management, lifetime management, and novel application interfaces. In this paper, we describe applications that require intermediate storage management, suggest several open research problems, and illustrate two systems - Freeloader and Tactical Storage - that attack different aspects of these problems.

UR - http://www.scopus.com/inward/record.url?scp=46049116478&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=46049116478&partnerID=8YFLogxK

U2 - 10.1109/CLUSTR.2006.311900

DO - 10.1109/CLUSTR.2006.311900

M3 - Conference contribution

SN - 1424403286

SN - 9781424403288

BT - Proceedings - IEEE International Conference on Cluster Computing, ICCC

ER -