Online hate ratings vary by extremes

A statistical analysis

Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon Gyo Jung, Bernard Jansen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.

Original languageEnglish
Title of host publicationCHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages213-217
Number of pages5
ISBN (Electronic)9781450360258
DOIs
Publication statusPublished - 8 Mar 2019
Event4th ACM SIGIR Conference on Information Interaction and Retrieval, CHIIR 2019 - Glasgow, United Kingdom
Duration: 10 Mar 201914 Mar 2019

Publication series

NameCHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval

Conference

Conference4th ACM SIGIR Conference on Information Interaction and Retrieval, CHIIR 2019
CountryUnited Kingdom
CityGlasgow
Period10/3/1914/3/19

Fingerprint

Statistical methods
Learning systems
Processing

Keywords

  • Crowdsourcing
  • Interpretation
  • Online hate
  • Ratings
  • Toxicity

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Information Systems

Cite this

Salminen, J., Almerekhi, H., Kamel, A. M., Jung, S. G., & Jansen, B. (2019). Online hate ratings vary by extremes: A statistical analysis. In CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (pp. 213-217). (CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval). Association for Computing Machinery, Inc. https://doi.org/10.1145/3295750.3298954

Online hate ratings vary by extremes : A statistical analysis. / Salminen, Joni; Almerekhi, Hind; Kamel, Ahmed Mohamed; Jung, Soon Gyo; Jansen, Bernard.

CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. Association for Computing Machinery, Inc, 2019. p. 213-217 (CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Salminen, J, Almerekhi, H, Kamel, AM, Jung, SG & Jansen, B 2019, Online hate ratings vary by extremes: A statistical analysis. in CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, Association for Computing Machinery, Inc, pp. 213-217, 4th ACM SIGIR Conference on Information Interaction and Retrieval, CHIIR 2019, Glasgow, United Kingdom, 10/3/19. https://doi.org/10.1145/3295750.3298954
Salminen J, Almerekhi H, Kamel AM, Jung SG, Jansen B. Online hate ratings vary by extremes: A statistical analysis. In CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. Association for Computing Machinery, Inc. 2019. p. 213-217. (CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval). https://doi.org/10.1145/3295750.3298954
Salminen, Joni ; Almerekhi, Hind ; Kamel, Ahmed Mohamed ; Jung, Soon Gyo ; Jansen, Bernard. / Online hate ratings vary by extremes : A statistical analysis. CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. Association for Computing Machinery, Inc, 2019. pp. 213-217 (CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval).
@inproceedings{3757de4811974256929ab23b2b214448,
title = "Online hate ratings vary by extremes: A statistical analysis",
abstract = "Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.",
keywords = "Crowdsourcing, Interpretation, Online hate, Ratings, Toxicity",
author = "Joni Salminen and Hind Almerekhi and Kamel, {Ahmed Mohamed} and Jung, {Soon Gyo} and Bernard Jansen",
year = "2019",
month = "3",
day = "8",
doi = "10.1145/3295750.3298954",
language = "English",
series = "CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval",
publisher = "Association for Computing Machinery, Inc",
pages = "213--217",
booktitle = "CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval",

}

TY - GEN

T1 - Online hate ratings vary by extremes

T2 - A statistical analysis

AU - Salminen, Joni

AU - Almerekhi, Hind

AU - Kamel, Ahmed Mohamed

AU - Jung, Soon Gyo

AU - Jansen, Bernard

PY - 2019/3/8

Y1 - 2019/3/8

N2 - Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.

AB - Analyzing 5,665 crowd ratings on 1,133 social media comments, we find that individuals tend to agree on the extremes of a hate rating scale more than in the middle when evaluating the hatefulness of online comments. The agreement is higher for less hateful comments and lowest on moderately hateful comments. The results have implications for researchers developing machine learning models for online hate processing, as the extreme classes are likely to require fewer annotations for reaching statistical stability. Our findings suggest that the models developed in this domain should consider the distributions of hate ratings rather than average hate scores.

KW - Crowdsourcing

KW - Interpretation

KW - Online hate

KW - Ratings

KW - Toxicity

UR - http://www.scopus.com/inward/record.url?scp=85063134329&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063134329&partnerID=8YFLogxK

U2 - 10.1145/3295750.3298954

DO - 10.1145/3295750.3298954

M3 - Conference contribution

T3 - CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval

SP - 213

EP - 217

BT - CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval

PB - Association for Computing Machinery, Inc

ER -