S-preconditioner for multi-fold data reduction with guaranteed user-controlled accuracy

Ye Jin, Sriram Lakshminarasimhan, Neil Shah, Zhenhuan Gong, C. S. Chang, Jackie Chen, Stephane Ethier, Hemanth Kolla, Seung Hoe Ku, Scott Klasky, Robert Latham, Robert Ross, Karen Schuchardt, Nagiza F. Samatova

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)


The growing gap between the massive amounts of data generated by petascale scientific simulation codes and the capability of system hardware and software to effectively analyze this data necessitates data reduction. Yet, the increasing data complexity challenges most, if not all, of the existing data compression methods. In fact, lossless compression techniques offer no more than 10% reduction on scientific data that we have experience with, which is widely regarded as effectively incompressible. To bridge this gap, in this paper, we advocate a transformative strategy that enables fast, accurate, and multi-fold reduction of double-precision floating-point scientific data. The intuition behind our method is inspired by an effective use of preconditioners for linear algebra solvers optimized for a particular class of computational "dwarfs" (e.g., dense or sparse matrices). Focusing on a commonly used multi-resolution wavelet compression technique as the underlying "solver" for data reduction we propose the S-preconditioner, which transforms scientific data into a form with high global regularity to ensure a significant decrease in the number of wavelet coefficients stored for a segment of data. Combined with the subsequent EQ-calibrator, our resultant method (called S-Preconditioned EQ-Calibrated Wavelets (SPEQC-WAVELETS)), robustly achieved a 4- to 5-fold data reduction-while guaranteeing user-defined accuracy of reconstructed data to be within 1% point-by-point relative error, lower than 0.01 Normalized RMSE, and higher than 0.99 Pearson Correlation. In this paper, we show the results we obtained by testing our method on six petascale simulation codes including fusion, combustion, climate, astrophysics, and subsurface groundwater in addition to 13 publicly available scientific datasets. We also demonstrate that application-driven data mining tasks performed on decompressed variables or their derived quantities produce results of comparable quality with the ones for the original data.

Original languageEnglish
Title of host publicationProceedings - 11th IEEE International Conference on Data Mining, ICDM 2011
Number of pages10
Publication statusPublished - 1 Dec 2011
Event11th IEEE International Conference on Data Mining, ICDM 2011 - Vancouver, BC, Canada
Duration: 11 Dec 201114 Dec 2011

Publication series

NameProceedings - IEEE International Conference on Data Mining, ICDM
ISSN (Print)1550-4786


Other11th IEEE International Conference on Data Mining, ICDM 2011
CityVancouver, BC



  • Data mining over decompressed data
  • Data reduction
  • Extreme-scale data analytics
  • In situ data analytics
  • Preconditioners for data mining

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Jin, Y., Lakshminarasimhan, S., Shah, N., Gong, Z., Chang, C. S., Chen, J., Ethier, S., Kolla, H., Ku, S. H., Klasky, S., Latham, R., Ross, R., Schuchardt, K., & Samatova, N. F. (2011). S-preconditioner for multi-fold data reduction with guaranteed user-controlled accuracy. In Proceedings - 11th IEEE International Conference on Data Mining, ICDM 2011 (pp. 290-299). [6137233] (Proceedings - IEEE International Conference on Data Mining, ICDM). https://doi.org/10.1109/ICDM.2011.138