Rethinking the ESP game

Stephen Robertson, Milan Vojnovic, Ingmar Weber

Research output: Chapter in Book/Report/Conference proceedingConference contribution

31 Citations (Scopus)

Abstract

The ESP Game [7] was designed to harvest human intelligence to assign labels to images - a task which is still difficult for even the most advanced systems in image processing [3]. However, the ESP Game as it is currently implemented encourages players to assign "obvious" labels, which can be easily predicted given previously assigned labels. We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one "off-limits" term assigned to them. We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.

Original languageEnglish
Title of host publicationConference on Human Factors in Computing Systems - Proceedings
Pages3937-3942
Number of pages6
DOIs
Publication statusPublished - 22 Sep 2009
Externally publishedYes
Event27th International Conference Extended Abstracts on Human Factors in Computing Systems, CHI 2009 - Boston, MA, United States
Duration: 4 Apr 20099 Apr 2009

Other

Other27th International Conference Extended Abstracts on Human Factors in Computing Systems, CHI 2009
CountryUnited States
CityBoston, MA
Period4/4/099/4/09

Fingerprint

Labels
Image processing

Keywords

  • ESP game
  • Image labeler
  • Tagging

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

Robertson, S., Vojnovic, M., & Weber, I. (2009). Rethinking the ESP game. In Conference on Human Factors in Computing Systems - Proceedings (pp. 3937-3942) https://doi.org/10.1145/1520340.1520597

Rethinking the ESP game. / Robertson, Stephen; Vojnovic, Milan; Weber, Ingmar.

Conference on Human Factors in Computing Systems - Proceedings. 2009. p. 3937-3942.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Robertson, S, Vojnovic, M & Weber, I 2009, Rethinking the ESP game. in Conference on Human Factors in Computing Systems - Proceedings. pp. 3937-3942, 27th International Conference Extended Abstracts on Human Factors in Computing Systems, CHI 2009, Boston, MA, United States, 4/4/09. https://doi.org/10.1145/1520340.1520597
Robertson S, Vojnovic M, Weber I. Rethinking the ESP game. In Conference on Human Factors in Computing Systems - Proceedings. 2009. p. 3937-3942 https://doi.org/10.1145/1520340.1520597
Robertson, Stephen ; Vojnovic, Milan ; Weber, Ingmar. / Rethinking the ESP game. Conference on Human Factors in Computing Systems - Proceedings. 2009. pp. 3937-3942
@inproceedings{149cd21115c1411fb8628cec4afdc2d2,
title = "Rethinking the ESP game",
abstract = "The ESP Game [7] was designed to harvest human intelligence to assign labels to images - a task which is still difficult for even the most advanced systems in image processing [3]. However, the ESP Game as it is currently implemented encourages players to assign {"}obvious{"} labels, which can be easily predicted given previously assigned labels. We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69{\%} of all images, and for 81{\%} of images which have at least one {"}off-limits{"} term assigned to them. We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.",
keywords = "ESP game, Image labeler, Tagging",
author = "Stephen Robertson and Milan Vojnovic and Ingmar Weber",
year = "2009",
month = "9",
day = "22",
doi = "10.1145/1520340.1520597",
language = "English",
isbn = "9781605582474",
pages = "3937--3942",
booktitle = "Conference on Human Factors in Computing Systems - Proceedings",

}

TY - GEN

T1 - Rethinking the ESP game

AU - Robertson, Stephen

AU - Vojnovic, Milan

AU - Weber, Ingmar

PY - 2009/9/22

Y1 - 2009/9/22

N2 - The ESP Game [7] was designed to harvest human intelligence to assign labels to images - a task which is still difficult for even the most advanced systems in image processing [3]. However, the ESP Game as it is currently implemented encourages players to assign "obvious" labels, which can be easily predicted given previously assigned labels. We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one "off-limits" term assigned to them. We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.

AB - The ESP Game [7] was designed to harvest human intelligence to assign labels to images - a task which is still difficult for even the most advanced systems in image processing [3]. However, the ESP Game as it is currently implemented encourages players to assign "obvious" labels, which can be easily predicted given previously assigned labels. We present a language model which can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any use of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one "off-limits" term assigned to them. We discuss how the scoring system and the design of the ESP game can be improved to encourage users to add less predictable labels, thereby improving the quality of the collected information.

KW - ESP game

KW - Image labeler

KW - Tagging

UR - http://www.scopus.com/inward/record.url?scp=70349189175&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70349189175&partnerID=8YFLogxK

U2 - 10.1145/1520340.1520597

DO - 10.1145/1520340.1520597

M3 - Conference contribution

AN - SCOPUS:70349189175

SN - 9781605582474

SP - 3937

EP - 3942

BT - Conference on Human Factors in Computing Systems - Proceedings

ER -