An axiomatically derived measure for the evaluation of classification algorithms

Fabrizio Sebastiani

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)

Abstract

We address the general problem of finding suitable evaluation measures for classification systems. To this end, we adopt an axiomatic approach, i.e., we discuss a number of properties ("axioms") that an evaluation measure for classification should arguably satisfy. We start our analysis by addressing binary classification. We show that F1, nowadays considered a standard measure for the evaluation of binary classification systems, does not comply with a number of them, and should thus be considered unsatisfactory. We go on to discuss an alternative, simple evaluation measure for binary classification, that we call K, and show that it instead satisfies all the previously proposed axioms. We thus argue that researchers and practitioners should replace F1 with K in their everyday binary classification practice. We carry on our analysis by showing that K can be smoothly extended to deal with single-label multi-class classification, cost-sensitive classification, and ordinal classification.

Original languageEnglish
Title of host publicationICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages11-20
Number of pages10
ISBN (Print)9781450338332
DOIs
Publication statusPublished - 27 Sep 2015
Externally publishedYes
Event5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015 - Northampton, United States
Duration: 27 Sep 201530 Sep 2015

Other

Other5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015
CountryUnited States
CityNorthampton
Period27/9/1530/9/15

Fingerprint

Labels
Costs

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Information Systems

Cite this

Sebastiani, F. (2015). An axiomatically derived measure for the evaluation of classification algorithms. In ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (pp. 11-20). Association for Computing Machinery, Inc. https://doi.org/10.1145/2808194.2809449

An axiomatically derived measure for the evaluation of classification algorithms. / Sebastiani, Fabrizio.

ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc, 2015. p. 11-20.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sebastiani, F 2015, An axiomatically derived measure for the evaluation of classification algorithms. in ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc, pp. 11-20, 5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015, Northampton, United States, 27/9/15. https://doi.org/10.1145/2808194.2809449
Sebastiani F. An axiomatically derived measure for the evaluation of classification algorithms. In ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc. 2015. p. 11-20 https://doi.org/10.1145/2808194.2809449
Sebastiani, Fabrizio. / An axiomatically derived measure for the evaluation of classification algorithms. ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc, 2015. pp. 11-20
@inproceedings{8be761d9fdfa41c69adc388667b14649,
title = "An axiomatically derived measure for the evaluation of classification algorithms",
abstract = "We address the general problem of finding suitable evaluation measures for classification systems. To this end, we adopt an axiomatic approach, i.e., we discuss a number of properties ({"}axioms{"}) that an evaluation measure for classification should arguably satisfy. We start our analysis by addressing binary classification. We show that F1, nowadays considered a standard measure for the evaluation of binary classification systems, does not comply with a number of them, and should thus be considered unsatisfactory. We go on to discuss an alternative, simple evaluation measure for binary classification, that we call K, and show that it instead satisfies all the previously proposed axioms. We thus argue that researchers and practitioners should replace F1 with K in their everyday binary classification practice. We carry on our analysis by showing that K can be smoothly extended to deal with single-label multi-class classification, cost-sensitive classification, and ordinal classification.",
author = "Fabrizio Sebastiani",
year = "2015",
month = "9",
day = "27",
doi = "10.1145/2808194.2809449",
language = "English",
isbn = "9781450338332",
pages = "11--20",
booktitle = "ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - An axiomatically derived measure for the evaluation of classification algorithms

AU - Sebastiani, Fabrizio

PY - 2015/9/27

Y1 - 2015/9/27

N2 - We address the general problem of finding suitable evaluation measures for classification systems. To this end, we adopt an axiomatic approach, i.e., we discuss a number of properties ("axioms") that an evaluation measure for classification should arguably satisfy. We start our analysis by addressing binary classification. We show that F1, nowadays considered a standard measure for the evaluation of binary classification systems, does not comply with a number of them, and should thus be considered unsatisfactory. We go on to discuss an alternative, simple evaluation measure for binary classification, that we call K, and show that it instead satisfies all the previously proposed axioms. We thus argue that researchers and practitioners should replace F1 with K in their everyday binary classification practice. We carry on our analysis by showing that K can be smoothly extended to deal with single-label multi-class classification, cost-sensitive classification, and ordinal classification.

AB - We address the general problem of finding suitable evaluation measures for classification systems. To this end, we adopt an axiomatic approach, i.e., we discuss a number of properties ("axioms") that an evaluation measure for classification should arguably satisfy. We start our analysis by addressing binary classification. We show that F1, nowadays considered a standard measure for the evaluation of binary classification systems, does not comply with a number of them, and should thus be considered unsatisfactory. We go on to discuss an alternative, simple evaluation measure for binary classification, that we call K, and show that it instead satisfies all the previously proposed axioms. We thus argue that researchers and practitioners should replace F1 with K in their everyday binary classification practice. We carry on our analysis by showing that K can be smoothly extended to deal with single-label multi-class classification, cost-sensitive classification, and ordinal classification.

UR - http://www.scopus.com/inward/record.url?scp=84964341378&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84964341378&partnerID=8YFLogxK

U2 - 10.1145/2808194.2809449

DO - 10.1145/2808194.2809449

M3 - Conference contribution

SN - 9781450338332

SP - 11

EP - 20

BT - ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval

PB - Association for Computing Machinery, Inc

ER -