A scientific data management system for irregular applications

Jaechun No, R. Thakur, D. Kaushik, L. Freitag, A. Choudhary

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Many scientific applications are I/O intensive and generate large data sets, spanning hundreds or thousands of "files." Management, storage, efficient access, and analysis of this data present an extremely challenging task. We have developed a software system, called Scientific Data Manager (SDM), that uses a combination of parallel file I/O and database support for high-performance scientific data management. SDM provides a high-level API to the user and, internally, uses a parallel file system to store real data and a database to store application-related metadata. In this paper, we describe how we designed and implemented SDM to support irregular applications. SDM can efficiently handle the reading and writing of data in an irregular mesh, as well as the distribution of index values. We describe the SDM user interface and how we have implemented it to achieve high performance. SDM makes extensive use of MPI-IO's noncontiguous collective I/O functions. SDM also uses the concept of a history file to optimize the cost of the index distribution using the metadata stored in database. We present performance results with two irregular applications, a CFD code called FUN3D and a Rayleigh-Taylor instability code, on the SGI Origin2000 at Argonne National Laboratory.

Original languageEnglish
Title of host publicationProceedings - 15th International Parallel and Distributed Processing Symposium, IPDPS 2001
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1216-1223
Number of pages8
ISBN (Electronic)0769509908, 9780769509907
DOIs
Publication statusPublished - 1 Jan 2001
Event15th International Parallel and Distributed Processing Symposium, IPDPS 2001 - San Francisco, United States
Duration: 23 Apr 200127 Apr 2001

Publication series

NameProceedings - 15th International Parallel and Distributed Processing Symposium, IPDPS 2001

Other

Other15th International Parallel and Distributed Processing Symposium, IPDPS 2001
CountryUnited States
CitySan Francisco
Period23/4/0127/4/01

ASJC Scopus subject areas

  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'A scientific data management system for irregular applications'. Together they form a unique fingerprint.

  • Cite this

    No, J., Thakur, R., Kaushik, D., Freitag, L., & Choudhary, A. (2001). A scientific data management system for irregular applications. In Proceedings - 15th International Parallel and Distributed Processing Symposium, IPDPS 2001 (pp. 1216-1223). [925096] (Proceedings - 15th International Parallel and Distributed Processing Symposium, IPDPS 2001). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2001.925096