Non-parametric message importance measure: Storage code design and transmission planning for big data

Shanyun Liu, Rui She, Pingyi Fan, Khaled Letaief

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

The storage and the transmission of messages in big data are discussed in this paper, where message importance is taken into account. To this end, we propose to use non-parametric message importance measure (NMIM) as a measure of message importance, which can characterize the uncertainty of random events like Shannon entropy and Rényi entropy. We prove that NMIM sufficiently describes the two key characters of big data, i.e., the rare events finding and the large diversities of events. Based on NMIM, we then propose an effective compressed encoding mode for data storage, and discuss the transmission of messages over some typical channel models with limited message importance loss. Our numerical results show that the proposed strategy occupies less storage space without losing too much important information, and the maximum received entropy rate increases with the increasing of message importance loss until it reaches saturation, which contributes to designing of better practical communication system.

Original languageEnglish
Article number8386858
Pages (from-to)5181-5196
Number of pages16
JournalIEEE Transactions on Communications
Volume66
Issue number11
DOIs
Publication statusPublished - 1 Nov 2018
Externally publishedYes

    Fingerprint

Keywords

  • big data
  • channel transmission
  • compressed storage
  • message importance measure
  • NMIM loss distortion
  • Non-parametric

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Cite this