Online hate ratings vary by extremes: A statistical analysis

Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon Gyo Jung, Bernard Jansen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.

Original languageEnglish
Title of host publicationCHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages213-217
Number of pages5
ISBN (Electronic)9781450360258
DOIs
Publication statusPublished - 8 Mar 2019
Event4th ACM SIGIR Conference on Information Interaction and Retrieval, CHIIR 2019 - Glasgow, United Kingdom
Duration: 10 Mar 201914 Mar 2019

Publication series

NameCHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval

Conference

Conference4th ACM SIGIR Conference on Information Interaction and Retrieval, CHIIR 2019
CountryUnited Kingdom
CityGlasgow
Period10/3/1914/3/19

    Fingerprint

Keywords

  • Crowdsourcing
  • Interpretation
  • Online hate
  • Ratings
  • Toxicity

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Information Systems

Cite this

Salminen, J., Almerekhi, H., Kamel, A. M., Jung, S. G., & Jansen, B. (2019). Online hate ratings vary by extremes: A statistical analysis. In CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (pp. 213-217). (CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval). Association for Computing Machinery, Inc. https://doi.org/10.1145/3295750.3298954