Evaluation measures for ordinal regression

Stefano Baccianella, Andrea Esuli, Fabrizio Sebastiani

Research output: Chapter in Book/Report/Conference proceedingConference contribution

102 Citations (Scopus)

Abstract

Ordinal regression (OR - also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.

Original languageEnglish
Title of host publicationISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications
Pages283-287
Number of pages5
DOIs
Publication statusPublished - 2009
Externally publishedYes
Event9th International Conference on Intelligent Systems Design and Applications, ISDA 2009 - Pisa
Duration: 30 Nov 20092 Dec 2009

Other

Other9th International Conference on Intelligent Systems Design and Applications, ISDA 2009
CityPisa
Period30/11/092/12/09

Fingerprint

Testing

Keywords

  • Class imbalance
  • Evaluation measures
  • Ordinal classification
  • Ordinal regression
  • Product reviews

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computational Theory and Mathematics
  • Signal Processing
  • Software

Cite this

Baccianella, S., Esuli, A., & Sebastiani, F. (2009). Evaluation measures for ordinal regression. In ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications (pp. 283-287). [5364825] https://doi.org/10.1109/ISDA.2009.230

Evaluation measures for ordinal regression. / Baccianella, Stefano; Esuli, Andrea; Sebastiani, Fabrizio.

ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications. 2009. p. 283-287 5364825.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Baccianella, S, Esuli, A & Sebastiani, F 2009, Evaluation measures for ordinal regression. in ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications., 5364825, pp. 283-287, 9th International Conference on Intelligent Systems Design and Applications, ISDA 2009, Pisa, 30/11/09. https://doi.org/10.1109/ISDA.2009.230
Baccianella S, Esuli A, Sebastiani F. Evaluation measures for ordinal regression. In ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications. 2009. p. 283-287. 5364825 https://doi.org/10.1109/ISDA.2009.230
Baccianella, Stefano ; Esuli, Andrea ; Sebastiani, Fabrizio. / Evaluation measures for ordinal regression. ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications. 2009. pp. 283-287
@inproceedings{eaeb96e81930431dab6631b74836e867,
title = "Evaluation measures for ordinal regression",
abstract = "Ordinal regression (OR - also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.",
keywords = "Class imbalance, Evaluation measures, Ordinal classification, Ordinal regression, Product reviews",
author = "Stefano Baccianella and Andrea Esuli and Fabrizio Sebastiani",
year = "2009",
doi = "10.1109/ISDA.2009.230",
language = "English",
isbn = "9780769538723",
pages = "283--287",
booktitle = "ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications",

}

TY - GEN

T1 - Evaluation measures for ordinal regression

AU - Baccianella, Stefano

AU - Esuli, Andrea

AU - Sebastiani, Fabrizio

PY - 2009

Y1 - 2009

N2 - Ordinal regression (OR - also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.

AB - Ordinal regression (OR - also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.

KW - Class imbalance

KW - Evaluation measures

KW - Ordinal classification

KW - Ordinal regression

KW - Product reviews

UR - http://www.scopus.com/inward/record.url?scp=77949512941&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77949512941&partnerID=8YFLogxK

U2 - 10.1109/ISDA.2009.230

DO - 10.1109/ISDA.2009.230

M3 - Conference contribution

SN - 9780769538723

SP - 283

EP - 287

BT - ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications

ER -