Xiaochuang Yao

Spatial coding-based approach for partitioning big spatial data in Hadoop

Abstract Spatial data partitioning (SDP) plays a powerful role in distributed storage and parallel computing for spatial data. However, due to skew distribution of spatial data and varying volume of spatial vector objects, it leads to a significant challenge to ensure both optimal performance of spatial operation and data balance in the cluster. To tackle this problem, we proposed a spatial coding-based approach for partitioning big spatial data in Hadoop. This approach, firstly, compressed the whole big spatial data based on spatial coding matrix to create a sensing information set (SIS), including spatial code, size, count and other information. SIS was then employed to build spatial partitioning matrix, which was used to spilt all spatial objects into different partitions in the cluster finally. Based on our approach, the neighbouring spatial objects can be partitioned into the same block. At the same time, it also can minimize the data skew in Hadoop distributed file system (HDFS). The presented approach with a case study in this paper is compared against random sampling based partitioning, with three measurement standards, namely, the spatial index quality, data skew in HDFS, and range query performance. The experimental results show that our method based on spatial coding technique can improve the query performance of big spatial data, as well as the data balance in HDFS. We implemented and deployed this approach in Hadoop, and it is also able to support efficiently any other distributed big spatial data systems.

Development of a Highly Flexible Mobile GIS-Based System for Collecting Arable Land Quality Data

In recent years, well-designed terminal-based methods for collecting index data have gradually replaced traditional pen-and-paper methods and have been extensively used in numerous studies. These new approaches offer users increased accuracy, efficiency, consumption, and data compatibility compared to traditional methods. In general, we find that spatial data content and quality index systems vary widely across different arable land regions. Thus, a system for the investigation of arable land quality indices that has the flexibility to utilize various types of spatial data and quality indices without requiring program modification is needed. This paper presents the framework, the module partition, and the structure of the data exchange interface for a highly flexible mobile GIS-based system, which we call the “arable land quality index data collection system” (ALQIDCS). This system incorporates a series of self-adaptive methods, a data table-driven model and two types of formulas for flexible data collection and processing. We tested our prototype system by investigating arable land quality in the Da Xing District, Beijing and in the Te Da La Qi District, Inner Mongolia, China. The results indicate that the ALQIDCS can effectively adapt to variations in spatial data and quality index systems and meet different objectives. The limitations of the ALQIDCS and suggestions for future work are also presented.

A WebGIS-based decision support system for locust prevention and control in China

Abstract Locust swarms are destructive agricultural and biological disasters in China. The green prevention and control (GPC, such as ecological regulation and physical control) of locusts is a comprehensive and complex process, especially in information technology. In this study, a web-based decision support system (DSS) integrated with geographic information system (GIS) is developed to prevent and control locusts efficiently, accurately, and rapidly. The locust prevention and control DSS (LPCDSS) is developed to assist farmers and local government agencies in Chinese provinces with high incidence of locust by providing spatial decision-making information. LPCDSS offers online access to county, city, provincial, and national level data queries and is capable of storing, spatial analyzing, and displaying geographically referenced information of locust data. The system can also provide the real-time tracking of global positioning system (GPS) location, as well as goods scheduling of locust plagues prevention. Six types of web service, real-time data synchronization model, and locust population estimation model are developed and implemented to improve the decision-making usability and feasibility of LPCDSS by adopting a three-layer system architecture. The system is developed by using several programming languages, libraries, and software components. As a result, this system has been running successfully for several years and has improved efficiency of the locust prevention and control in China with high efficiency and great accuracy. The approaches and methodologies presented in this paper can serve as a reference for those who are interested in developing integrated pest control system applications.

An Automatic Counting Method of Maize Ear Grain Based on Image Processing

Corn variety testing is a process to pick and cultivate a high yield, disease resistant and outstandingly adaptive variety from thousands of corn hybrid varieties. In this process, we have to do a large number of comparative tests, observation and measurement. The workload of this measurement is very huge, for the large number of varieties under test. The grain numbers of maize ear is an important parameter to the corn variety testing. At present, the grain counting is mostly done by manpower. In this way, both the deviation and workload is unacceptable. In this paper, an automatic counting method of maize ear grain is established basing on image processing. Image segmentation is the basis and classic difficult part of image processing. This paper presents an image pre-processing method, which is based on the characteristics of maize ear image. This method includes median filter to eliminate random noise, wallis filter to sharpen the image boundary and histogram enhancement. It also mainly introduces an in-depth study of Otsu algorithms. To overcome the problems of Otsu algorithm that background information being erroneously divided when object size is small. A new method based on traditional Otsu method is proposed, which combines the multi-threshold segmentation and RBGM gradient descent. The implementation of RBGM gradient descent leads to a remarkable improvement on the efficiency of multi-threshold segmentation which is generally an extremely time-consuming task. Our experimental evaluations on 25 sets of maize ear image datasets show that the proposed method can produce more competitive results on effectiveness and speed in comparison to the manpower. The grain counting accuracy of ear volume can reach to 96.8%.

LandQv2: A MapReduce-Based System for Processing Arable Land Quality Big Data

Arable land quality (ALQ) data are a foundational resource for national food security. With the rapid development of spatial information technologies, the annual acquisition and update of ALQ data covering the country have become more accurate and faster. ALQ data are mainly vector-based spatial big data in the ESRI (Environmental Systems Research Institute) shapefile format. Although the shapefile is the most common GIS vector data format, unfortunately, the usage of ALQ data is very constrained due to its massive size and the limited capabilities of traditional applications. To tackle the above issues, this paper introduces LandQv2, which is a MapReduce-based parallel processing system for ALQ big data. The core content of LandQv2 is composed of four key technologies including data preprocessing, the distributed R-tree index, the spatial range query, and the map tile pyramid model-based visualization. According to the functions in LandQv2, firstly, ALQ big data are transformed by a MapReduce-based parallel algorithm from the ESRI Shapefile format to the GeoCSV file format in HDFS (Hadoop Distributed File System), and then, the spatial coding-based partition and R-tree index are executed for the spatial range query operation. In addition, the visualization of ALQ big data with a GIS (Geographic Information System) web API (Application Programming Interface) uses the MapReduce program to generate a single image or pyramid tiles for big data display. Finally, a set of experiments running on a live system deployed on a cluster of machines shows the efficiency and scalability of the proposed system. All of these functions supported by LandQv2 are integrated into SpatialHadoop, and it is also able to efficiently support any other distributed spatial big data systems.

Design and implementation of geographic information systems, remote sensing, and global positioning system–based information platform for locust control

Abstract To monitor and control locusts efficiently, an information platform for locust control based on the global positioning system (GPS), remote sensing (RS), and geographic information systems (GIS) was developed. The platform can provide accurate information about locust occurrence and control strategies for a specific geographic place. The platform consists of three systems based on modern pest control: field ecology (locust occurrence) and GIS in a mobile GPS pad, a processing system for locust information based on GIS and RS, and a WebGIS-based real-time monitoring and controlling system. This platform was run at different geographical locations for three years and facilitated locust control in China with high efficiency and great accuracy.

RDCRMG: A Raster Dataset Clean & Reconstitution Multi-Grid Architecture for Remote Sensing Monitoring of Vegetation Dryness

In recent years, remote sensing (RS) research on crop growth status monitoring has gradually turned from static spectrum information retrieval in large-scale to meso-scale or micro-scale, timely multi-source data cooperative analysis; this change has presented higher requirements for RS data acquisition and analysis efficiency. How to implement rapid and stable massive RS data extraction and analysis becomes a serious problem. This paper reports on a Raster Dataset Clean & Reconstitution Multi-Grid (RDCRMG) architecture for remote sensing monitoring of vegetation dryness in which different types of raster datasets have been partitioned, organized and systematically applied. First, raster images have been subdivided into several independent blocks and distributed for storage in different data nodes by using the multi-grid as a consistent partition unit. Second, the “no metadata model” ideology has been referenced so that targets raster data can be speedily extracted by directly calculating the data storage path without retrieving metadata records; third, grids that cover the query range can be easily assessed. This assessment allows the query task to be easily split into several sub-tasks and executed in parallel by grouping these grids. Our RDCRMG-based change detection of the spectral reflectance information test and the data extraction efficiency comparative test shows that the RDCRMG is reliable for vegetation dryness monitoring with a slight reflectance information distortion and consistent percentage histograms. Furthermore, the RDCGMG-based data extraction in parallel circumstances has the advantages of high efficiency and excellent stability compared to that of the RDCGMG-based data extraction in serial circumstances and traditional data extraction. At last, an RDCRMG-based vegetation dryness monitoring platform (VDMP) has been constructed to apply RS data inversion in vegetation dryness monitoring. Through actual applications, the RDCRMG architecture is proven to be appropriate for timely vegetation dryness RS automatic monitoring with better performance, more reliability and higher extensibility. Our future works will focus on integrating more kinds of continuously Remote Sens. 2018, 10, 1376; doi:10.3390/rs10091376 www.mdpi.com/journal/remotesensing Remote Sens. 2018, 10, 1376 2 of 24 updated RS data into the RDCRMG-based VDMP and integrating more multi-source datasets based collaborative analysis models for agricultural monitoring.