Coordinating human and machine intelligence to classify microblog communications in crises

Muhammad Imran, Carlos Castillo, Ji Lucas, Patrick Meier, Jakob Rogstadius

Research output: Chapter in Book/Report/Conference proceedingConference contribution

42 Citations (Scopus)

Abstract

An emerging paradigm for the processing of data streams involves human and machine computation working together, allowing human intelligence to process large-scale data. We apply this approach to the classification of crisis-related messages in microblog streams. We begin by describing the platform AIDR (Artificial Intelligence for Disaster Response), which collects human annotations over time to create and maintain automatic supervised classifiers for social media messages. Next, we study two significant challenges in its design: (1) identifying which elements must be labeled by humans, and (2) determining when to ask for such annotations to be done. The first challenge is selecting the items to be labeled by crowd sourcing workers to maximize the productivity of their work. The second challenge is to schedule the work in order to reliably maintain high classification accuracy over time. We provide and validate answers to these challenges by extensive experimentation on real world datasets.

Original languageEnglish
Title of host publicationISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management
PublisherThe Pennsylvania State University
Pages712-721
Number of pages10
ISBN (Print)9780692211946
Publication statusPublished - 1 Jan 2014
Event11th International Conference on Information Systems for Crisis Response and Management, ISCRAM 2014 - University Park, PA, United States
Duration: 1 May 20141 May 2014

Other

Other11th International Conference on Information Systems for Crisis Response and Management, ISCRAM 2014
CountryUnited States
CityUniversity Park, PA
Period1/5/141/5/14

Fingerprint

Communication
Disasters
Artificial intelligence
Classifiers
Productivity
Processing

ASJC Scopus subject areas

  • Information Systems

Cite this

Imran, M., Castillo, C., Lucas, J., Meier, P., & Rogstadius, J. (2014). Coordinating human and machine intelligence to classify microblog communications in crises. In ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management (pp. 712-721). The Pennsylvania State University.

Coordinating human and machine intelligence to classify microblog communications in crises. / Imran, Muhammad; Castillo, Carlos; Lucas, Ji; Meier, Patrick; Rogstadius, Jakob.

ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management. The Pennsylvania State University, 2014. p. 712-721.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Imran, M, Castillo, C, Lucas, J, Meier, P & Rogstadius, J 2014, Coordinating human and machine intelligence to classify microblog communications in crises. in ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management. The Pennsylvania State University, pp. 712-721, 11th International Conference on Information Systems for Crisis Response and Management, ISCRAM 2014, University Park, PA, United States, 1/5/14.
Imran M, Castillo C, Lucas J, Meier P, Rogstadius J. Coordinating human and machine intelligence to classify microblog communications in crises. In ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management. The Pennsylvania State University. 2014. p. 712-721
Imran, Muhammad ; Castillo, Carlos ; Lucas, Ji ; Meier, Patrick ; Rogstadius, Jakob. / Coordinating human and machine intelligence to classify microblog communications in crises. ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management. The Pennsylvania State University, 2014. pp. 712-721
@inproceedings{5388f8d75ccf48e6bf3fdd92e963d67a,
title = "Coordinating human and machine intelligence to classify microblog communications in crises",
abstract = "An emerging paradigm for the processing of data streams involves human and machine computation working together, allowing human intelligence to process large-scale data. We apply this approach to the classification of crisis-related messages in microblog streams. We begin by describing the platform AIDR (Artificial Intelligence for Disaster Response), which collects human annotations over time to create and maintain automatic supervised classifiers for social media messages. Next, we study two significant challenges in its design: (1) identifying which elements must be labeled by humans, and (2) determining when to ask for such annotations to be done. The first challenge is selecting the items to be labeled by crowd sourcing workers to maximize the productivity of their work. The second challenge is to schedule the work in order to reliably maintain high classification accuracy over time. We provide and validate answers to these challenges by extensive experimentation on real world datasets.",
author = "Muhammad Imran and Carlos Castillo and Ji Lucas and Patrick Meier and Jakob Rogstadius",
year = "2014",
month = "1",
day = "1",
language = "English",
isbn = "9780692211946",
pages = "712--721",
booktitle = "ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management",
publisher = "The Pennsylvania State University",

}

TY - GEN

T1 - Coordinating human and machine intelligence to classify microblog communications in crises

AU - Imran, Muhammad

AU - Castillo, Carlos

AU - Lucas, Ji

AU - Meier, Patrick

AU - Rogstadius, Jakob

PY - 2014/1/1

Y1 - 2014/1/1

N2 - An emerging paradigm for the processing of data streams involves human and machine computation working together, allowing human intelligence to process large-scale data. We apply this approach to the classification of crisis-related messages in microblog streams. We begin by describing the platform AIDR (Artificial Intelligence for Disaster Response), which collects human annotations over time to create and maintain automatic supervised classifiers for social media messages. Next, we study two significant challenges in its design: (1) identifying which elements must be labeled by humans, and (2) determining when to ask for such annotations to be done. The first challenge is selecting the items to be labeled by crowd sourcing workers to maximize the productivity of their work. The second challenge is to schedule the work in order to reliably maintain high classification accuracy over time. We provide and validate answers to these challenges by extensive experimentation on real world datasets.

AB - An emerging paradigm for the processing of data streams involves human and machine computation working together, allowing human intelligence to process large-scale data. We apply this approach to the classification of crisis-related messages in microblog streams. We begin by describing the platform AIDR (Artificial Intelligence for Disaster Response), which collects human annotations over time to create and maintain automatic supervised classifiers for social media messages. Next, we study two significant challenges in its design: (1) identifying which elements must be labeled by humans, and (2) determining when to ask for such annotations to be done. The first challenge is selecting the items to be labeled by crowd sourcing workers to maximize the productivity of their work. The second challenge is to schedule the work in order to reliably maintain high classification accuracy over time. We provide and validate answers to these challenges by extensive experimentation on real world datasets.

UR - http://www.scopus.com/inward/record.url?scp=84905845537&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84905845537&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9780692211946

SP - 712

EP - 721

BT - ISCRAM 2014 Conference Proceedings - 11th International Conference on Information Systems for Crisis Response and Management

PB - The Pennsylvania State University

ER -