LandQv2

A MapReduce-based system for processing arable land quality big data

Xiaochuang Yao, Mohamed Mokbel, Sijing Ye, Guoqing Li, Louai Alarabi, Ahmed Eldawy, Zuliang Zhao, Long Zhao, Dehai Zhu

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Arable land quality (ALQ) data are a foundational resource for national food security. With the rapid development of spatial information technologies, the annual acquisition and update of ALQ data covering the country have become more accurate and faster. ALQ data are mainly vector-based spatial big data in the ESRI (Environmental Systems Research Institute) shapefile format. Although the shapefile is the most common GIS vector data format, unfortunately, the usage of ALQ data is very constrained due to its massive size and the limited capabilities of traditional applications. To tackle the above issues, this paper introduces LandQv2, which is a MapReduce-based parallel processing system for ALQ big data. The core content of LandQv2 is composed of four key technologies including data preprocessing, the distributed R-tree index, the spatial range query, and the map tile pyramid model-based visualization. According to the functions in LandQv2, firstly, ALQ big data are transformed by a MapReduce-based parallel algorithm from the ESRI Shapefile format to the GeoCSV file format in HDFS (Hadoop Distributed File System), and then, the spatial coding-based partition and R-tree index are executed for the spatial range query operation. In addition, the visualization of ALQ big data with a GIS (Geographic Information System) web API (Application Programming Interface) uses the MapReduce program to generate a single image or pyramid tiles for big data display. Finally, a set of experiments running on a live system deployed on a cluster of machines shows the efficiency and scalability of the proposed system. All of these functions supported by LandQv2 are integrated into SpatialHadoop, and it is also able to efficiently support any other distributed spatial big data systems.

Original languageEnglish
Article number271
JournalISPRS International Journal of Geo-Information
Volume7
Issue number7
DOIs
Publication statusPublished - 1 Jul 2018
Externally publishedYes

Fingerprint

data quality
arable land
Processing
Tile
systems research
Geographic information systems
research facility
Visualization
visualization
information system
Parallel processing systems
key technology
Application programming interfaces (API)
Parallel algorithms
Information technology
Big data
Scalability
information technology
food security
Display devices

Keywords

  • Arable land quality (ALQ)
  • GIS
  • MapReduce
  • Parallel processing
  • Spatial big data

ASJC Scopus subject areas

  • Geography, Planning and Development
  • Computers in Earth Sciences
  • Earth and Planetary Sciences (miscellaneous)

Cite this

LandQv2 : A MapReduce-based system for processing arable land quality big data. / Yao, Xiaochuang; Mokbel, Mohamed; Ye, Sijing; Li, Guoqing; Alarabi, Louai; Eldawy, Ahmed; Zhao, Zuliang; Zhao, Long; Zhu, Dehai.

In: ISPRS International Journal of Geo-Information, Vol. 7, No. 7, 271, 01.07.2018.

Research output: Contribution to journalArticle

Yao, Xiaochuang ; Mokbel, Mohamed ; Ye, Sijing ; Li, Guoqing ; Alarabi, Louai ; Eldawy, Ahmed ; Zhao, Zuliang ; Zhao, Long ; Zhu, Dehai. / LandQv2 : A MapReduce-based system for processing arable land quality big data. In: ISPRS International Journal of Geo-Information. 2018 ; Vol. 7, No. 7.
@article{9359fc3111a644e588e1d003cdd9fe06,
title = "LandQv2: A MapReduce-based system for processing arable land quality big data",
abstract = "Arable land quality (ALQ) data are a foundational resource for national food security. With the rapid development of spatial information technologies, the annual acquisition and update of ALQ data covering the country have become more accurate and faster. ALQ data are mainly vector-based spatial big data in the ESRI (Environmental Systems Research Institute) shapefile format. Although the shapefile is the most common GIS vector data format, unfortunately, the usage of ALQ data is very constrained due to its massive size and the limited capabilities of traditional applications. To tackle the above issues, this paper introduces LandQv2, which is a MapReduce-based parallel processing system for ALQ big data. The core content of LandQv2 is composed of four key technologies including data preprocessing, the distributed R-tree index, the spatial range query, and the map tile pyramid model-based visualization. According to the functions in LandQv2, firstly, ALQ big data are transformed by a MapReduce-based parallel algorithm from the ESRI Shapefile format to the GeoCSV file format in HDFS (Hadoop Distributed File System), and then, the spatial coding-based partition and R-tree index are executed for the spatial range query operation. In addition, the visualization of ALQ big data with a GIS (Geographic Information System) web API (Application Programming Interface) uses the MapReduce program to generate a single image or pyramid tiles for big data display. Finally, a set of experiments running on a live system deployed on a cluster of machines shows the efficiency and scalability of the proposed system. All of these functions supported by LandQv2 are integrated into SpatialHadoop, and it is also able to efficiently support any other distributed spatial big data systems.",
keywords = "Arable land quality (ALQ), GIS, MapReduce, Parallel processing, Spatial big data",
author = "Xiaochuang Yao and Mohamed Mokbel and Sijing Ye and Guoqing Li and Louai Alarabi and Ahmed Eldawy and Zuliang Zhao and Long Zhao and Dehai Zhu",
year = "2018",
month = "7",
day = "1",
doi = "10.3390/ijgi7070271",
language = "English",
volume = "7",
journal = "ISPRS International Journal of Geo-Information",
issn = "2220-9964",
publisher = "MDPI AG",
number = "7",

}

TY - JOUR

T1 - LandQv2

T2 - A MapReduce-based system for processing arable land quality big data

AU - Yao, Xiaochuang

AU - Mokbel, Mohamed

AU - Ye, Sijing

AU - Li, Guoqing

AU - Alarabi, Louai

AU - Eldawy, Ahmed

AU - Zhao, Zuliang

AU - Zhao, Long

AU - Zhu, Dehai

PY - 2018/7/1

Y1 - 2018/7/1

N2 - Arable land quality (ALQ) data are a foundational resource for national food security. With the rapid development of spatial information technologies, the annual acquisition and update of ALQ data covering the country have become more accurate and faster. ALQ data are mainly vector-based spatial big data in the ESRI (Environmental Systems Research Institute) shapefile format. Although the shapefile is the most common GIS vector data format, unfortunately, the usage of ALQ data is very constrained due to its massive size and the limited capabilities of traditional applications. To tackle the above issues, this paper introduces LandQv2, which is a MapReduce-based parallel processing system for ALQ big data. The core content of LandQv2 is composed of four key technologies including data preprocessing, the distributed R-tree index, the spatial range query, and the map tile pyramid model-based visualization. According to the functions in LandQv2, firstly, ALQ big data are transformed by a MapReduce-based parallel algorithm from the ESRI Shapefile format to the GeoCSV file format in HDFS (Hadoop Distributed File System), and then, the spatial coding-based partition and R-tree index are executed for the spatial range query operation. In addition, the visualization of ALQ big data with a GIS (Geographic Information System) web API (Application Programming Interface) uses the MapReduce program to generate a single image or pyramid tiles for big data display. Finally, a set of experiments running on a live system deployed on a cluster of machines shows the efficiency and scalability of the proposed system. All of these functions supported by LandQv2 are integrated into SpatialHadoop, and it is also able to efficiently support any other distributed spatial big data systems.

AB - Arable land quality (ALQ) data are a foundational resource for national food security. With the rapid development of spatial information technologies, the annual acquisition and update of ALQ data covering the country have become more accurate and faster. ALQ data are mainly vector-based spatial big data in the ESRI (Environmental Systems Research Institute) shapefile format. Although the shapefile is the most common GIS vector data format, unfortunately, the usage of ALQ data is very constrained due to its massive size and the limited capabilities of traditional applications. To tackle the above issues, this paper introduces LandQv2, which is a MapReduce-based parallel processing system for ALQ big data. The core content of LandQv2 is composed of four key technologies including data preprocessing, the distributed R-tree index, the spatial range query, and the map tile pyramid model-based visualization. According to the functions in LandQv2, firstly, ALQ big data are transformed by a MapReduce-based parallel algorithm from the ESRI Shapefile format to the GeoCSV file format in HDFS (Hadoop Distributed File System), and then, the spatial coding-based partition and R-tree index are executed for the spatial range query operation. In addition, the visualization of ALQ big data with a GIS (Geographic Information System) web API (Application Programming Interface) uses the MapReduce program to generate a single image or pyramid tiles for big data display. Finally, a set of experiments running on a live system deployed on a cluster of machines shows the efficiency and scalability of the proposed system. All of these functions supported by LandQv2 are integrated into SpatialHadoop, and it is also able to efficiently support any other distributed spatial big data systems.

KW - Arable land quality (ALQ)

KW - GIS

KW - MapReduce

KW - Parallel processing

KW - Spatial big data

UR - http://www.scopus.com/inward/record.url?scp=85053636655&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85053636655&partnerID=8YFLogxK

U2 - 10.3390/ijgi7070271

DO - 10.3390/ijgi7070271

M3 - Article

VL - 7

JO - ISPRS International Journal of Geo-Information

JF - ISPRS International Journal of Geo-Information

SN - 2220-9964

IS - 7

M1 - 271

ER -