A dynamic microtask scheduling approach for SLO based human-augmented computing

Koushik Sinha, Pratham Majumder, Geetha Manjunath

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Current machine algorithms for qualitative analysis of unstructured data (in the form of social media posts, audio and video, among others) do not perform well and show low accuracies due to the need for human-like intelligence. However, machines are easily scalable, fast and the quality of their output is predictable. On the other hand, though humans are much better than machine algorithms at image and video analysis, natural language text and speech processing, they are unfortunately unpredictable, slower and can be erroneous or even malicious as computing agents. Therefore, a task execution engine which can enable human-augmented cloud computing by intelligently orchestrating machine and human computing resources would be able to provide richer and superior analytics on unstructured data than either of the two types of computing agents in isolation. We believe that a key aspect of enabling such analytics would be to provide guaranteed service level objectives, in terms of accuracy, time and budget In this paper, we present a microtask scheduler with integrated service level objectives (SLO) management. With this goal, we have introduced two new decision parameters: H-M ratio and microtask completion rate. An early prototype has been built and validated through simulation with actual performance data collected from anonymous crowd workers on Amazon Mechanical Turk. Machine computation was done using Hewlett Packard's Autonomy IDOL while ground truth was established through the use of known, expert workers. To the best of our knowledge, ours is the first work that attempts to simultaneously attempt to address the three SLO parameters of accuracy, budget and deadline for data-parallel microtasks.

Original languageEnglish
Title of host publication2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages191-196
Number of pages6
ISBN (Electronic)9781479985791
DOIs
Publication statusPublished - 8 Jul 2016
Event3rd International Conference on Recent Advances in Information Technology, RAIT 2016 - Dhanbad, India
Duration: 3 Mar 20165 Mar 2016

Other

Other3rd International Conference on Recent Advances in Information Technology, RAIT 2016
CountryIndia
CityDhanbad
Period3/3/165/3/16

Fingerprint

Scheduling
Text processing
Speech processing
Budgets
Cloud computing
Engines
Social Media
Intelligence
Language

Keywords

  • Crowdsourcing
  • data analytics
  • human augmented computing
  • microtask
  • quality control
  • service level objectives
  • task scheduling

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Health Informatics
  • Information Systems

Cite this

Sinha, K., Majumder, P., & Manjunath, G. (2016). A dynamic microtask scheduling approach for SLO based human-augmented computing. In 2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016 (pp. 191-196). [7507900] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/RAIT.2016.7507900

A dynamic microtask scheduling approach for SLO based human-augmented computing. / Sinha, Koushik; Majumder, Pratham; Manjunath, Geetha.

2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 191-196 7507900.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sinha, K, Majumder, P & Manjunath, G 2016, A dynamic microtask scheduling approach for SLO based human-augmented computing. in 2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016., 7507900, Institute of Electrical and Electronics Engineers Inc., pp. 191-196, 3rd International Conference on Recent Advances in Information Technology, RAIT 2016, Dhanbad, India, 3/3/16. https://doi.org/10.1109/RAIT.2016.7507900
Sinha K, Majumder P, Manjunath G. A dynamic microtask scheduling approach for SLO based human-augmented computing. In 2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 191-196. 7507900 https://doi.org/10.1109/RAIT.2016.7507900
Sinha, Koushik ; Majumder, Pratham ; Manjunath, Geetha. / A dynamic microtask scheduling approach for SLO based human-augmented computing. 2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 191-196
@inproceedings{106f1e9e38884c85bd3f34cf973b47c2,
title = "A dynamic microtask scheduling approach for SLO based human-augmented computing",
abstract = "Current machine algorithms for qualitative analysis of unstructured data (in the form of social media posts, audio and video, among others) do not perform well and show low accuracies due to the need for human-like intelligence. However, machines are easily scalable, fast and the quality of their output is predictable. On the other hand, though humans are much better than machine algorithms at image and video analysis, natural language text and speech processing, they are unfortunately unpredictable, slower and can be erroneous or even malicious as computing agents. Therefore, a task execution engine which can enable human-augmented cloud computing by intelligently orchestrating machine and human computing resources would be able to provide richer and superior analytics on unstructured data than either of the two types of computing agents in isolation. We believe that a key aspect of enabling such analytics would be to provide guaranteed service level objectives, in terms of accuracy, time and budget In this paper, we present a microtask scheduler with integrated service level objectives (SLO) management. With this goal, we have introduced two new decision parameters: H-M ratio and microtask completion rate. An early prototype has been built and validated through simulation with actual performance data collected from anonymous crowd workers on Amazon Mechanical Turk. Machine computation was done using Hewlett Packard's Autonomy IDOL while ground truth was established through the use of known, expert workers. To the best of our knowledge, ours is the first work that attempts to simultaneously attempt to address the three SLO parameters of accuracy, budget and deadline for data-parallel microtasks.",
keywords = "Crowdsourcing, data analytics, human augmented computing, microtask, quality control, service level objectives, task scheduling",
author = "Koushik Sinha and Pratham Majumder and Geetha Manjunath",
year = "2016",
month = "7",
day = "8",
doi = "10.1109/RAIT.2016.7507900",
language = "English",
pages = "191--196",
booktitle = "2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - A dynamic microtask scheduling approach for SLO based human-augmented computing

AU - Sinha, Koushik

AU - Majumder, Pratham

AU - Manjunath, Geetha

PY - 2016/7/8

Y1 - 2016/7/8

N2 - Current machine algorithms for qualitative analysis of unstructured data (in the form of social media posts, audio and video, among others) do not perform well and show low accuracies due to the need for human-like intelligence. However, machines are easily scalable, fast and the quality of their output is predictable. On the other hand, though humans are much better than machine algorithms at image and video analysis, natural language text and speech processing, they are unfortunately unpredictable, slower and can be erroneous or even malicious as computing agents. Therefore, a task execution engine which can enable human-augmented cloud computing by intelligently orchestrating machine and human computing resources would be able to provide richer and superior analytics on unstructured data than either of the two types of computing agents in isolation. We believe that a key aspect of enabling such analytics would be to provide guaranteed service level objectives, in terms of accuracy, time and budget In this paper, we present a microtask scheduler with integrated service level objectives (SLO) management. With this goal, we have introduced two new decision parameters: H-M ratio and microtask completion rate. An early prototype has been built and validated through simulation with actual performance data collected from anonymous crowd workers on Amazon Mechanical Turk. Machine computation was done using Hewlett Packard's Autonomy IDOL while ground truth was established through the use of known, expert workers. To the best of our knowledge, ours is the first work that attempts to simultaneously attempt to address the three SLO parameters of accuracy, budget and deadline for data-parallel microtasks.

AB - Current machine algorithms for qualitative analysis of unstructured data (in the form of social media posts, audio and video, among others) do not perform well and show low accuracies due to the need for human-like intelligence. However, machines are easily scalable, fast and the quality of their output is predictable. On the other hand, though humans are much better than machine algorithms at image and video analysis, natural language text and speech processing, they are unfortunately unpredictable, slower and can be erroneous or even malicious as computing agents. Therefore, a task execution engine which can enable human-augmented cloud computing by intelligently orchestrating machine and human computing resources would be able to provide richer and superior analytics on unstructured data than either of the two types of computing agents in isolation. We believe that a key aspect of enabling such analytics would be to provide guaranteed service level objectives, in terms of accuracy, time and budget In this paper, we present a microtask scheduler with integrated service level objectives (SLO) management. With this goal, we have introduced two new decision parameters: H-M ratio and microtask completion rate. An early prototype has been built and validated through simulation with actual performance data collected from anonymous crowd workers on Amazon Mechanical Turk. Machine computation was done using Hewlett Packard's Autonomy IDOL while ground truth was established through the use of known, expert workers. To the best of our knowledge, ours is the first work that attempts to simultaneously attempt to address the three SLO parameters of accuracy, budget and deadline for data-parallel microtasks.

KW - Crowdsourcing

KW - data analytics

KW - human augmented computing

KW - microtask

KW - quality control

KW - service level objectives

KW - task scheduling

UR - http://www.scopus.com/inward/record.url?scp=84999029581&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84999029581&partnerID=8YFLogxK

U2 - 10.1109/RAIT.2016.7507900

DO - 10.1109/RAIT.2016.7507900

M3 - Conference contribution

SP - 191

EP - 196

BT - 2016 3rd International Conference on Recent Advances in Information Technology, RAIT 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -