STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games

Jeremy Blackburn, Haewoon Kwak

Research output: Chapter in Book/Report/Conference proceedingConference contribution

40 Citations (Scopus)

Abstract

One problem facing players of competitive games is negative, or toxic, behavior. League of Legends, the largest eSport game, uses a crowdsourcing platform called the Tribunal to judge whether a reported toxic player should be punished or not. The Tribunal is a two stage system requiring reports from those players that directly observe toxic behavior, and human experts that review aggregated reports. While this system has successfully dealt with the vague nature of toxic behavior by majority rules based on many votes, it naturally requires tremendous cost, time, and human efforts. In this paper, we propose a supervised learning approach for predicting crowdsourced decisions on toxic behavior with largescale labeled data collections; over 10 million user reports involved in 1.46 million toxic players and corresponding crowdsourced decisions. Our result shows good performance in detecting overwhelmingly majority cases and predicting crowdsourced decisions on them. We demonstrate good portability of our classifier across regions. Finally, we estimate the practical implications of our approach, potential cost savings and victim protection. Copyright is held by the International World Wide Web Conference Committee (IW3C2).

Original languageEnglish
Title of host publicationWWW 2014 - Proceedings of the 23rd International Conference on World Wide Web
PublisherAssociation for Computing Machinery, Inc
Pages877-887
Number of pages11
ISBN (Print)9781450327442
DOIs
Publication statusPublished - 1 Jan 2014
Externally publishedYes
Event23rd International Conference on World Wide Web, WWW 2014 - Seoul, Korea, Republic of
Duration: 7 Apr 201411 Apr 2014

Other

Other23rd International Conference on World Wide Web, WWW 2014
CountryKorea, Republic of
CitySeoul
Period7/4/1411/4/14

Fingerprint

Supervised learning
World Wide Web
Costs
Classifiers

Keywords

  • Crowdsourcing
  • League of legends
  • Machine learning
  • Online video games
  • Toxic behavior

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Cite this

Blackburn, J., & Kwak, H. (2014). STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games. In WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web (pp. 877-887). Association for Computing Machinery, Inc. https://doi.org/10.1145/2566486.2567987

STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games. / Blackburn, Jeremy; Kwak, Haewoon.

WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web. Association for Computing Machinery, Inc, 2014. p. 877-887.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Blackburn, J & Kwak, H 2014, STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games. in WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web. Association for Computing Machinery, Inc, pp. 877-887, 23rd International Conference on World Wide Web, WWW 2014, Seoul, Korea, Republic of, 7/4/14. https://doi.org/10.1145/2566486.2567987
Blackburn J, Kwak H. STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games. In WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web. Association for Computing Machinery, Inc. 2014. p. 877-887 https://doi.org/10.1145/2566486.2567987
Blackburn, Jeremy ; Kwak, Haewoon. / STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games. WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web. Association for Computing Machinery, Inc, 2014. pp. 877-887
@inproceedings{4bbe02fcc9d445cdbe75b9cf9954f04d,
title = "STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games",
abstract = "One problem facing players of competitive games is negative, or toxic, behavior. League of Legends, the largest eSport game, uses a crowdsourcing platform called the Tribunal to judge whether a reported toxic player should be punished or not. The Tribunal is a two stage system requiring reports from those players that directly observe toxic behavior, and human experts that review aggregated reports. While this system has successfully dealt with the vague nature of toxic behavior by majority rules based on many votes, it naturally requires tremendous cost, time, and human efforts. In this paper, we propose a supervised learning approach for predicting crowdsourced decisions on toxic behavior with largescale labeled data collections; over 10 million user reports involved in 1.46 million toxic players and corresponding crowdsourced decisions. Our result shows good performance in detecting overwhelmingly majority cases and predicting crowdsourced decisions on them. We demonstrate good portability of our classifier across regions. Finally, we estimate the practical implications of our approach, potential cost savings and victim protection. Copyright is held by the International World Wide Web Conference Committee (IW3C2).",
keywords = "Crowdsourcing, League of legends, Machine learning, Online video games, Toxic behavior",
author = "Jeremy Blackburn and Haewoon Kwak",
year = "2014",
month = "1",
day = "1",
doi = "10.1145/2566486.2567987",
language = "English",
isbn = "9781450327442",
pages = "877--887",
booktitle = "WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - STFU NOOB! Predicting crowdsourced decisions on toxic behavior in online games

AU - Blackburn, Jeremy

AU - Kwak, Haewoon

PY - 2014/1/1

Y1 - 2014/1/1

N2 - One problem facing players of competitive games is negative, or toxic, behavior. League of Legends, the largest eSport game, uses a crowdsourcing platform called the Tribunal to judge whether a reported toxic player should be punished or not. The Tribunal is a two stage system requiring reports from those players that directly observe toxic behavior, and human experts that review aggregated reports. While this system has successfully dealt with the vague nature of toxic behavior by majority rules based on many votes, it naturally requires tremendous cost, time, and human efforts. In this paper, we propose a supervised learning approach for predicting crowdsourced decisions on toxic behavior with largescale labeled data collections; over 10 million user reports involved in 1.46 million toxic players and corresponding crowdsourced decisions. Our result shows good performance in detecting overwhelmingly majority cases and predicting crowdsourced decisions on them. We demonstrate good portability of our classifier across regions. Finally, we estimate the practical implications of our approach, potential cost savings and victim protection. Copyright is held by the International World Wide Web Conference Committee (IW3C2).

AB - One problem facing players of competitive games is negative, or toxic, behavior. League of Legends, the largest eSport game, uses a crowdsourcing platform called the Tribunal to judge whether a reported toxic player should be punished or not. The Tribunal is a two stage system requiring reports from those players that directly observe toxic behavior, and human experts that review aggregated reports. While this system has successfully dealt with the vague nature of toxic behavior by majority rules based on many votes, it naturally requires tremendous cost, time, and human efforts. In this paper, we propose a supervised learning approach for predicting crowdsourced decisions on toxic behavior with largescale labeled data collections; over 10 million user reports involved in 1.46 million toxic players and corresponding crowdsourced decisions. Our result shows good performance in detecting overwhelmingly majority cases and predicting crowdsourced decisions on them. We demonstrate good portability of our classifier across regions. Finally, we estimate the practical implications of our approach, potential cost savings and victim protection. Copyright is held by the International World Wide Web Conference Committee (IW3C2).

KW - Crowdsourcing

KW - League of legends

KW - Machine learning

KW - Online video games

KW - Toxic behavior

UR - http://www.scopus.com/inward/record.url?scp=84909592433&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84909592433&partnerID=8YFLogxK

U2 - 10.1145/2566486.2567987

DO - 10.1145/2566486.2567987

M3 - Conference contribution

AN - SCOPUS:84909592433

SN - 9781450327442

SP - 877

EP - 887

BT - WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web

PB - Association for Computing Machinery, Inc

ER -