FAHES

A robust disguised missing values detector

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Missing values are common in real-world data and may seriously affect data analytics such as simple statistics and hypothesis testing. Generally speaking, there are two types of missing values: explicitly missing values (i.e., NULL values), and implicitly missing values (a.k.a. disguised missing values (DMVs)) such as “11111111" for a phone number and “Some college" for education. While detecting explicitly missing values is trivial, detecting DMVs is not; the essential challenge is the lack of standardization about how DMVs are generated. In this paper, we present FAHES, a robust system for detecting DMVs from two angles: DMVs as detectable outliers and as detectable inliers. For DMVs as outliers, we propose a syntactic outlier detection module for categorical data, and a density-based outlier detection module for numerical values. For DMVs as inliers, we propose a method that detects DMVs which follow either missing-completely-at-random or missing-at-random models. The robustness of FAHES is achieved through an ensemble technique that is inspired by outlier ensembles. Our extensive experiments using real-world data sets show that FAHES delivers better results than existing solutions.

Original languageEnglish
Title of host publicationKDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
PublisherAssociation for Computing Machinery
Pages2100-2109
Number of pages10
ISBN (Print)9781450355520
DOIs
Publication statusPublished - 19 Jul 2018
Event24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018 - London, United Kingdom
Duration: 19 Aug 201823 Aug 2018

Other

Other24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018
CountryUnited Kingdom
CityLondon
Period19/8/1823/8/18

Fingerprint

Syntactics
Standardization
Education
Statistics
Detectors
Testing
Experiments

Keywords

  • Disguised Missing Value
  • Numerical Outliers
  • Syntactic Outliers
  • Syntactic Patterns

ASJC Scopus subject areas

  • Software
  • Information Systems

Cite this

Qahtan, A., Elmagarmid, A., Fernandez, R. C., Ouzzani, M., & Tang, N. (2018). FAHES: A robust disguised missing values detector. In KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 2100-2109). Association for Computing Machinery. https://doi.org/10.1145/3219819.3220109

FAHES : A robust disguised missing values detector. / Qahtan, Abdulhakim; Elmagarmid, Ahmed; Fernandez, Raul Castro; Ouzzani, Mourad; Tang, Nan.

KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2018. p. 2100-2109.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Qahtan, A, Elmagarmid, A, Fernandez, RC, Ouzzani, M & Tang, N 2018, FAHES: A robust disguised missing values detector. in KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, pp. 2100-2109, 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018, London, United Kingdom, 19/8/18. https://doi.org/10.1145/3219819.3220109
Qahtan A, Elmagarmid A, Fernandez RC, Ouzzani M, Tang N. FAHES: A robust disguised missing values detector. In KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery. 2018. p. 2100-2109 https://doi.org/10.1145/3219819.3220109
Qahtan, Abdulhakim ; Elmagarmid, Ahmed ; Fernandez, Raul Castro ; Ouzzani, Mourad ; Tang, Nan. / FAHES : A robust disguised missing values detector. KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2018. pp. 2100-2109
@inproceedings{fb66f66a50b24edf816a308157f9e5d2,
title = "FAHES: A robust disguised missing values detector",
abstract = "Missing values are common in real-world data and may seriously affect data analytics such as simple statistics and hypothesis testing. Generally speaking, there are two types of missing values: explicitly missing values (i.e., NULL values), and implicitly missing values (a.k.a. disguised missing values (DMVs)) such as “11111111{"} for a phone number and “Some college{"} for education. While detecting explicitly missing values is trivial, detecting DMVs is not; the essential challenge is the lack of standardization about how DMVs are generated. In this paper, we present FAHES, a robust system for detecting DMVs from two angles: DMVs as detectable outliers and as detectable inliers. For DMVs as outliers, we propose a syntactic outlier detection module for categorical data, and a density-based outlier detection module for numerical values. For DMVs as inliers, we propose a method that detects DMVs which follow either missing-completely-at-random or missing-at-random models. The robustness of FAHES is achieved through an ensemble technique that is inspired by outlier ensembles. Our extensive experiments using real-world data sets show that FAHES delivers better results than existing solutions.",
keywords = "Disguised Missing Value, Numerical Outliers, Syntactic Outliers, Syntactic Patterns",
author = "Abdulhakim Qahtan and Ahmed Elmagarmid and Fernandez, {Raul Castro} and Mourad Ouzzani and Nan Tang",
year = "2018",
month = "7",
day = "19",
doi = "10.1145/3219819.3220109",
language = "English",
isbn = "9781450355520",
pages = "2100--2109",
booktitle = "KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",
publisher = "Association for Computing Machinery",

}

TY - GEN

T1 - FAHES

T2 - A robust disguised missing values detector

AU - Qahtan, Abdulhakim

AU - Elmagarmid, Ahmed

AU - Fernandez, Raul Castro

AU - Ouzzani, Mourad

AU - Tang, Nan

PY - 2018/7/19

Y1 - 2018/7/19

N2 - Missing values are common in real-world data and may seriously affect data analytics such as simple statistics and hypothesis testing. Generally speaking, there are two types of missing values: explicitly missing values (i.e., NULL values), and implicitly missing values (a.k.a. disguised missing values (DMVs)) such as “11111111" for a phone number and “Some college" for education. While detecting explicitly missing values is trivial, detecting DMVs is not; the essential challenge is the lack of standardization about how DMVs are generated. In this paper, we present FAHES, a robust system for detecting DMVs from two angles: DMVs as detectable outliers and as detectable inliers. For DMVs as outliers, we propose a syntactic outlier detection module for categorical data, and a density-based outlier detection module for numerical values. For DMVs as inliers, we propose a method that detects DMVs which follow either missing-completely-at-random or missing-at-random models. The robustness of FAHES is achieved through an ensemble technique that is inspired by outlier ensembles. Our extensive experiments using real-world data sets show that FAHES delivers better results than existing solutions.

AB - Missing values are common in real-world data and may seriously affect data analytics such as simple statistics and hypothesis testing. Generally speaking, there are two types of missing values: explicitly missing values (i.e., NULL values), and implicitly missing values (a.k.a. disguised missing values (DMVs)) such as “11111111" for a phone number and “Some college" for education. While detecting explicitly missing values is trivial, detecting DMVs is not; the essential challenge is the lack of standardization about how DMVs are generated. In this paper, we present FAHES, a robust system for detecting DMVs from two angles: DMVs as detectable outliers and as detectable inliers. For DMVs as outliers, we propose a syntactic outlier detection module for categorical data, and a density-based outlier detection module for numerical values. For DMVs as inliers, we propose a method that detects DMVs which follow either missing-completely-at-random or missing-at-random models. The robustness of FAHES is achieved through an ensemble technique that is inspired by outlier ensembles. Our extensive experiments using real-world data sets show that FAHES delivers better results than existing solutions.

KW - Disguised Missing Value

KW - Numerical Outliers

KW - Syntactic Outliers

KW - Syntactic Patterns

UR - http://www.scopus.com/inward/record.url?scp=85051569092&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051569092&partnerID=8YFLogxK

U2 - 10.1145/3219819.3220109

DO - 10.1145/3219819.3220109

M3 - Conference contribution

SN - 9781450355520

SP - 2100

EP - 2109

BT - KDD 2018 - Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

PB - Association for Computing Machinery

ER -