Processing Social Media Images by Combining Human and Machine Computing during Crises

Firoj Alam, Ferda Ofli, Muhammad Imran

Research output: Contribution to journalArticle

12 Citations (Scopus)

Abstract

The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness as disaster unfolds. In addition to textual content, people post overwhelming amounts of imagery content on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in computer vision research, making sense of the imagery content in real-time during disasters remains a challenging task. One of the important challenges is that a large proportion of images shared on social media is redundant or irrelevant, which requires robust filtering mechanisms. Another important challenge is that images acquired after major disasters do not share the same characteristics as those in large-scale image collections with clean annotations of well-defined object categories such as house, car, airplane, cat, dog, etc., used traditionally in computer vision research. To tackle these challenges, we present a social media image processing pipeline that combines human and machine intelligence to perform two important tasks: (i) capturing and filtering of social media imagery content (i.e., real-time image streaming, de-duplication, and relevancy filtering); and (ii) actionable information extraction (i.e., damage severity assessment) as a core situational awareness task during an on-going crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.

Original languageEnglish
Pages (from-to)1-17
Number of pages17
JournalInternational Journal of Human-Computer Interaction
DOIs
Publication statusAccepted/In press - 28 Jan 2018

Fingerprint

social media
Disasters
disaster
Processing
Computer vision
Pipelines
aircraft
intelligence
social network
Image processing
damages
Railroad cars
utilization
Aircraft
event
experiment
resources
Experiments
time

ASJC Scopus subject areas

  • Human Factors and Ergonomics
  • Human-Computer Interaction
  • Computer Science Applications

Cite this

Processing Social Media Images by Combining Human and Machine Computing during Crises. / Alam, Firoj; Ofli, Ferda; Imran, Muhammad.

In: International Journal of Human-Computer Interaction, 28.01.2018, p. 1-17.

Research output: Contribution to journalArticle

@article{e642de207cc34084a868f800d129673f,
title = "Processing Social Media Images by Combining Human and Machine Computing during Crises",
abstract = "The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness as disaster unfolds. In addition to textual content, people post overwhelming amounts of imagery content on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in computer vision research, making sense of the imagery content in real-time during disasters remains a challenging task. One of the important challenges is that a large proportion of images shared on social media is redundant or irrelevant, which requires robust filtering mechanisms. Another important challenge is that images acquired after major disasters do not share the same characteristics as those in large-scale image collections with clean annotations of well-defined object categories such as house, car, airplane, cat, dog, etc., used traditionally in computer vision research. To tackle these challenges, we present a social media image processing pipeline that combines human and machine intelligence to perform two important tasks: (i) capturing and filtering of social media imagery content (i.e., real-time image streaming, de-duplication, and relevancy filtering); and (ii) actionable information extraction (i.e., damage severity assessment) as a core situational awareness task during an on-going crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.",
author = "Firoj Alam and Ferda Ofli and Muhammad Imran",
year = "2018",
month = "1",
day = "28",
doi = "10.1080/10447318.2018.1427831",
language = "English",
pages = "1--17",
journal = "International Journal of Human-Computer Interaction",
issn = "1044-7318",
publisher = "Taylor and Francis Ltd.",

}

TY - JOUR

T1 - Processing Social Media Images by Combining Human and Machine Computing during Crises

AU - Alam, Firoj

AU - Ofli, Ferda

AU - Imran, Muhammad

PY - 2018/1/28

Y1 - 2018/1/28

N2 - The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness as disaster unfolds. In addition to textual content, people post overwhelming amounts of imagery content on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in computer vision research, making sense of the imagery content in real-time during disasters remains a challenging task. One of the important challenges is that a large proportion of images shared on social media is redundant or irrelevant, which requires robust filtering mechanisms. Another important challenge is that images acquired after major disasters do not share the same characteristics as those in large-scale image collections with clean annotations of well-defined object categories such as house, car, airplane, cat, dog, etc., used traditionally in computer vision research. To tackle these challenges, we present a social media image processing pipeline that combines human and machine intelligence to perform two important tasks: (i) capturing and filtering of social media imagery content (i.e., real-time image streaming, de-duplication, and relevancy filtering); and (ii) actionable information extraction (i.e., damage severity assessment) as a core situational awareness task during an on-going crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.

AB - The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness as disaster unfolds. In addition to textual content, people post overwhelming amounts of imagery content on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in computer vision research, making sense of the imagery content in real-time during disasters remains a challenging task. One of the important challenges is that a large proportion of images shared on social media is redundant or irrelevant, which requires robust filtering mechanisms. Another important challenge is that images acquired after major disasters do not share the same characteristics as those in large-scale image collections with clean annotations of well-defined object categories such as house, car, airplane, cat, dog, etc., used traditionally in computer vision research. To tackle these challenges, we present a social media image processing pipeline that combines human and machine intelligence to perform two important tasks: (i) capturing and filtering of social media imagery content (i.e., real-time image streaming, de-duplication, and relevancy filtering); and (ii) actionable information extraction (i.e., damage severity assessment) as a core situational awareness task during an on-going crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.

UR - http://www.scopus.com/inward/record.url?scp=85041121203&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041121203&partnerID=8YFLogxK

U2 - 10.1080/10447318.2018.1427831

DO - 10.1080/10447318.2018.1427831

M3 - Article

AN - SCOPUS:85041121203

SP - 1

EP - 17

JO - International Journal of Human-Computer Interaction

JF - International Journal of Human-Computer Interaction

SN - 1044-7318

ER -