Efficient exploration of telco big data with compression and decaying

Constantinos Costa, Georgios Chatzimilioudis, Demetrios Zeinalipour-Yazti, Mohamed Mokbel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Citations (Scopus)

Abstract

In the realm of smart cities, telecommunication companies (telcos) are expected to play a protagonistic role as these can capture a variety of natural phenomena on an ongoing basis, e.g., traffic in a city, mobility patterns for emergency response or city planning. The key challenges for telcos in this era is to ingest in the most compact manner huge amounts of network logs, perform big data exploration and analytics on the generated data within a tolerable elapsed time. This paper introduces SPATE, an innovative telco big data exploration framework whose objectives are two-fold: (i) minimizing the storage space needed to incrementally retain data over time; and (ii) minimizing the response time for spatiotemporal data exploration queries over recent data. The storage layer of our framework uses lossless data compression to ingest recent streams of telco big data in the most compact manner retaining full resolution for data exploration tasks. The indexing layer of our system then takes care of the progressive loss of detail in information, coined decaying, as data ages with time. The exploration layer provides visual means to explore the generated spatio-Temporal information space. We measure the efficiency of the proposed framework using a 5GB anonymized real telco network trace and a variety of telco-specific tasks, such as OLAP and OLTP querying, privacy-Aware data sharing, multivariate statistics, clustering and regression. We show that out framework can achieve comparable response times to the state-of-The-Art using an order of magnitude less storage space.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017
PublisherIEEE Computer Society
Pages1332-1343
Number of pages12
ISBN (Electronic)9781509065431
DOIs
Publication statusPublished - 16 May 2017
Externally publishedYes
Event33rd IEEE International Conference on Data Engineering, ICDE 2017 - San Diego, United States
Duration: 19 Apr 201722 Apr 2017

Other

Other33rd IEEE International Conference on Data Engineering, ICDE 2017
CountryUnited States
CitySan Diego
Period19/4/1722/4/17

Fingerprint

Telecommunication
Urban planning
Data compression
Industry
Statistics
Big data
Smart city

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Information Systems

Cite this

Costa, C., Chatzimilioudis, G., Zeinalipour-Yazti, D., & Mokbel, M. (2017). Efficient exploration of telco big data with compression and decaying. In Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017 (pp. 1332-1343). [7930071] IEEE Computer Society. https://doi.org/10.1109/ICDE.2017.175

Efficient exploration of telco big data with compression and decaying. / Costa, Constantinos; Chatzimilioudis, Georgios; Zeinalipour-Yazti, Demetrios; Mokbel, Mohamed.

Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017. IEEE Computer Society, 2017. p. 1332-1343 7930071.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Costa, C, Chatzimilioudis, G, Zeinalipour-Yazti, D & Mokbel, M 2017, Efficient exploration of telco big data with compression and decaying. in Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017., 7930071, IEEE Computer Society, pp. 1332-1343, 33rd IEEE International Conference on Data Engineering, ICDE 2017, San Diego, United States, 19/4/17. https://doi.org/10.1109/ICDE.2017.175
Costa C, Chatzimilioudis G, Zeinalipour-Yazti D, Mokbel M. Efficient exploration of telco big data with compression and decaying. In Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017. IEEE Computer Society. 2017. p. 1332-1343. 7930071 https://doi.org/10.1109/ICDE.2017.175
Costa, Constantinos ; Chatzimilioudis, Georgios ; Zeinalipour-Yazti, Demetrios ; Mokbel, Mohamed. / Efficient exploration of telco big data with compression and decaying. Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017. IEEE Computer Society, 2017. pp. 1332-1343
@inproceedings{5a5122885a474885a1ccb5f61e3da3ee,
title = "Efficient exploration of telco big data with compression and decaying",
abstract = "In the realm of smart cities, telecommunication companies (telcos) are expected to play a protagonistic role as these can capture a variety of natural phenomena on an ongoing basis, e.g., traffic in a city, mobility patterns for emergency response or city planning. The key challenges for telcos in this era is to ingest in the most compact manner huge amounts of network logs, perform big data exploration and analytics on the generated data within a tolerable elapsed time. This paper introduces SPATE, an innovative telco big data exploration framework whose objectives are two-fold: (i) minimizing the storage space needed to incrementally retain data over time; and (ii) minimizing the response time for spatiotemporal data exploration queries over recent data. The storage layer of our framework uses lossless data compression to ingest recent streams of telco big data in the most compact manner retaining full resolution for data exploration tasks. The indexing layer of our system then takes care of the progressive loss of detail in information, coined decaying, as data ages with time. The exploration layer provides visual means to explore the generated spatio-Temporal information space. We measure the efficiency of the proposed framework using a 5GB anonymized real telco network trace and a variety of telco-specific tasks, such as OLAP and OLTP querying, privacy-Aware data sharing, multivariate statistics, clustering and regression. We show that out framework can achieve comparable response times to the state-of-The-Art using an order of magnitude less storage space.",
author = "Constantinos Costa and Georgios Chatzimilioudis and Demetrios Zeinalipour-Yazti and Mohamed Mokbel",
year = "2017",
month = "5",
day = "16",
doi = "10.1109/ICDE.2017.175",
language = "English",
pages = "1332--1343",
booktitle = "Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - Efficient exploration of telco big data with compression and decaying

AU - Costa, Constantinos

AU - Chatzimilioudis, Georgios

AU - Zeinalipour-Yazti, Demetrios

AU - Mokbel, Mohamed

PY - 2017/5/16

Y1 - 2017/5/16

N2 - In the realm of smart cities, telecommunication companies (telcos) are expected to play a protagonistic role as these can capture a variety of natural phenomena on an ongoing basis, e.g., traffic in a city, mobility patterns for emergency response or city planning. The key challenges for telcos in this era is to ingest in the most compact manner huge amounts of network logs, perform big data exploration and analytics on the generated data within a tolerable elapsed time. This paper introduces SPATE, an innovative telco big data exploration framework whose objectives are two-fold: (i) minimizing the storage space needed to incrementally retain data over time; and (ii) minimizing the response time for spatiotemporal data exploration queries over recent data. The storage layer of our framework uses lossless data compression to ingest recent streams of telco big data in the most compact manner retaining full resolution for data exploration tasks. The indexing layer of our system then takes care of the progressive loss of detail in information, coined decaying, as data ages with time. The exploration layer provides visual means to explore the generated spatio-Temporal information space. We measure the efficiency of the proposed framework using a 5GB anonymized real telco network trace and a variety of telco-specific tasks, such as OLAP and OLTP querying, privacy-Aware data sharing, multivariate statistics, clustering and regression. We show that out framework can achieve comparable response times to the state-of-The-Art using an order of magnitude less storage space.

AB - In the realm of smart cities, telecommunication companies (telcos) are expected to play a protagonistic role as these can capture a variety of natural phenomena on an ongoing basis, e.g., traffic in a city, mobility patterns for emergency response or city planning. The key challenges for telcos in this era is to ingest in the most compact manner huge amounts of network logs, perform big data exploration and analytics on the generated data within a tolerable elapsed time. This paper introduces SPATE, an innovative telco big data exploration framework whose objectives are two-fold: (i) minimizing the storage space needed to incrementally retain data over time; and (ii) minimizing the response time for spatiotemporal data exploration queries over recent data. The storage layer of our framework uses lossless data compression to ingest recent streams of telco big data in the most compact manner retaining full resolution for data exploration tasks. The indexing layer of our system then takes care of the progressive loss of detail in information, coined decaying, as data ages with time. The exploration layer provides visual means to explore the generated spatio-Temporal information space. We measure the efficiency of the proposed framework using a 5GB anonymized real telco network trace and a variety of telco-specific tasks, such as OLAP and OLTP querying, privacy-Aware data sharing, multivariate statistics, clustering and regression. We show that out framework can achieve comparable response times to the state-of-The-Art using an order of magnitude less storage space.

UR - http://www.scopus.com/inward/record.url?scp=85021189248&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85021189248&partnerID=8YFLogxK

U2 - 10.1109/ICDE.2017.175

DO - 10.1109/ICDE.2017.175

M3 - Conference contribution

SP - 1332

EP - 1343

BT - Proceedings - 2017 IEEE 33rd International Conference on Data Engineering, ICDE 2017

PB - IEEE Computer Society

ER -