Evaluation measures for ordinal regression

Stefano Baccianella, Andrea Esuli, Fabrizio Sebastiani

Research output: Chapter in Book/Report/Conference proceedingConference contribution

110 Citations (Scopus)

Abstract

Ordinal regression (OR - also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.

Original languageEnglish
Title of host publicationISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications
Pages283-287
Number of pages5
DOIs
Publication statusPublished - 1 Dec 2009
Event9th International Conference on Intelligent Systems Design and Applications, ISDA 2009 - Pisa, Italy
Duration: 30 Nov 20092 Dec 2009

Publication series

NameISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications

Conference

Conference9th International Conference on Intelligent Systems Design and Applications, ISDA 2009
CountryItaly
CityPisa
Period30/11/092/12/09

    Fingerprint

Keywords

  • Class imbalance
  • Evaluation measures
  • Ordinal classification
  • Ordinal regression
  • Product reviews

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computational Theory and Mathematics
  • Signal Processing
  • Software

Cite this

Baccianella, S., Esuli, A., & Sebastiani, F. (2009). Evaluation measures for ordinal regression. In ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications (pp. 283-287). [5364825] (ISDA 2009 - 9th International Conference on Intelligent Systems Design and Applications). https://doi.org/10.1109/ISDA.2009.230