Non-parametric message importance measure

Storage code design and transmission planning for big data

Shanyun Liu, Rui She, Pingyi Fan, Khaled Letaief

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

The storage and the transmission of messages in big data are discussed in this paper, where message importance is taken into account. To this end, we propose to use non-parametric message importance measure (NMIM) as a measure of message importance, which can characterize the uncertainty of random events like Shannon entropy and Rényi entropy. We prove that NMIM sufficiently describes the two key characters of big data, i.e., the rare events finding and the large diversities of events. Based on NMIM, we then propose an effective compressed encoding mode for data storage, and discuss the transmission of messages over some typical channel models with limited message importance loss. Our numerical results show that the proposed strategy occupies less storage space without losing too much important information, and the maximum received entropy rate increases with the increasing of message importance loss until it reaches saturation, which contributes to designing of better practical communication system.

Original languageEnglish
Article number8386858
Pages (from-to)5181-5196
Number of pages16
JournalIEEE Transactions on Communications
Volume66
Issue number11
DOIs
Publication statusPublished - 1 Nov 2018
Externally publishedYes

Fingerprint

Entropy
Planning
Communication systems
Data storage equipment
Big data
Uncertainty

Keywords

  • big data
  • channel transmission
  • compressed storage
  • message importance measure
  • NMIM loss distortion
  • Non-parametric

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Cite this

Non-parametric message importance measure : Storage code design and transmission planning for big data. / Liu, Shanyun; She, Rui; Fan, Pingyi; Letaief, Khaled.

In: IEEE Transactions on Communications, Vol. 66, No. 11, 8386858, 01.11.2018, p. 5181-5196.

Research output: Contribution to journalArticle

@article{e0557db50deb4d5dbce7b1ccc05b9ad7,
title = "Non-parametric message importance measure: Storage code design and transmission planning for big data",
abstract = "The storage and the transmission of messages in big data are discussed in this paper, where message importance is taken into account. To this end, we propose to use non-parametric message importance measure (NMIM) as a measure of message importance, which can characterize the uncertainty of random events like Shannon entropy and R{\'e}nyi entropy. We prove that NMIM sufficiently describes the two key characters of big data, i.e., the rare events finding and the large diversities of events. Based on NMIM, we then propose an effective compressed encoding mode for data storage, and discuss the transmission of messages over some typical channel models with limited message importance loss. Our numerical results show that the proposed strategy occupies less storage space without losing too much important information, and the maximum received entropy rate increases with the increasing of message importance loss until it reaches saturation, which contributes to designing of better practical communication system.",
keywords = "big data, channel transmission, compressed storage, message importance measure, NMIM loss distortion, Non-parametric",
author = "Shanyun Liu and Rui She and Pingyi Fan and Khaled Letaief",
year = "2018",
month = "11",
day = "1",
doi = "10.1109/TCOMM.2018.2847666",
language = "English",
volume = "66",
pages = "5181--5196",
journal = "IEEE Transactions on Communications",
issn = "0096-1965",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "11",

}

TY - JOUR

T1 - Non-parametric message importance measure

T2 - Storage code design and transmission planning for big data

AU - Liu, Shanyun

AU - She, Rui

AU - Fan, Pingyi

AU - Letaief, Khaled

PY - 2018/11/1

Y1 - 2018/11/1

N2 - The storage and the transmission of messages in big data are discussed in this paper, where message importance is taken into account. To this end, we propose to use non-parametric message importance measure (NMIM) as a measure of message importance, which can characterize the uncertainty of random events like Shannon entropy and Rényi entropy. We prove that NMIM sufficiently describes the two key characters of big data, i.e., the rare events finding and the large diversities of events. Based on NMIM, we then propose an effective compressed encoding mode for data storage, and discuss the transmission of messages over some typical channel models with limited message importance loss. Our numerical results show that the proposed strategy occupies less storage space without losing too much important information, and the maximum received entropy rate increases with the increasing of message importance loss until it reaches saturation, which contributes to designing of better practical communication system.

AB - The storage and the transmission of messages in big data are discussed in this paper, where message importance is taken into account. To this end, we propose to use non-parametric message importance measure (NMIM) as a measure of message importance, which can characterize the uncertainty of random events like Shannon entropy and Rényi entropy. We prove that NMIM sufficiently describes the two key characters of big data, i.e., the rare events finding and the large diversities of events. Based on NMIM, we then propose an effective compressed encoding mode for data storage, and discuss the transmission of messages over some typical channel models with limited message importance loss. Our numerical results show that the proposed strategy occupies less storage space without losing too much important information, and the maximum received entropy rate increases with the increasing of message importance loss until it reaches saturation, which contributes to designing of better practical communication system.

KW - big data

KW - channel transmission

KW - compressed storage

KW - message importance measure

KW - NMIM loss distortion

KW - Non-parametric

UR - http://www.scopus.com/inward/record.url?scp=85048686170&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048686170&partnerID=8YFLogxK

U2 - 10.1109/TCOMM.2018.2847666

DO - 10.1109/TCOMM.2018.2847666

M3 - Article

VL - 66

SP - 5181

EP - 5196

JO - IEEE Transactions on Communications

JF - IEEE Transactions on Communications

SN - 0096-1965

IS - 11

M1 - 8386858

ER -