Towards building a scholarly big data platform: Challenges, lessons and opportunities

Zhaohui Wu, Jian Wu, Madian Khabsa, Kyle Williams, Hung Hsuan Chen, Wenyi Huang, Suppawong Tuarob, Sagnik Ray Choudhury, Alexander Ororbia, Prasenjit Mitra, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

26 Citations (Scopus)

Abstract

We introduce a big data platform that provides various services for harvesting scholarly information and enabling efficient scholarly applications. The core architecture of the platform is built on a secured private cloud, crawls data using a scholarly focused crawler that leverages a dynamic scheduler, processes by utilizing a map reduce based crawl-extraction-ingestion (CEI) workflow, and is stored in distributed repositories and databases. Services such as scholarly data harvesting, information extraction, and user information and log data analytics are integrated into the platform and provided by an OAI and RESTful API. We also introduce a set of scholarly applications built on top of this platform including citation recommendation and collaborator discovery.

Original languageEnglish
Title of host publicationProceedings of the ACM/IEEE Joint Conference on Digital Libraries
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages117-126
Number of pages10
ISBN (Print)9781479955695
DOIs
Publication statusPublished - 1 Dec 2014
Externally publishedYes
Event2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014 - London
Duration: 8 Sep 201412 Sep 2014

Other

Other2014 14th IEEE/ACM Joint Conference on Digital Libraries, JCDL 2014
CityLondon
Period8/9/1412/9/14

    Fingerprint

Keywords

  • Big Data
  • Information Extraction
  • Scholarly Big Data

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Wu, Z., Wu, J., Khabsa, M., Williams, K., Chen, H. H., Huang, W., Tuarob, S., Choudhury, S. R., Ororbia, A., Mitra, P., & Giles, C. L. (2014). Towards building a scholarly big data platform: Challenges, lessons and opportunities. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (pp. 117-126). [6970157] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/JCDL.2014.6970157