Manoj Karkee

Sensors and systems for fruit detection and localization

We reviewed successes and challenges of sensing systems for fruit detection.We also reviewed the capabilities of sensors used for fruit localization.Challenges included occlusion, clustering and variable lighting condition.We presented future direction including amendment of orchard environment.Sensor fusion and human machine collaboration can also be areas for research. This paper reviews the research and development of machine vision systems for fruit detection and localization for robotic harvesting and/or crop-load estimation of specialty tree crops including apples, pears, and citrus. Variable lighting condition, occlusions, and clustering are some of the important issues needed to be addressed for accurate detection and localization of fruit in orchard environment. To address these issues, various techniques have been investigated using different types of sensors and their combinations as well as with different image processing techniques. This paper summarizes various techniques and their advantages and disadvantages in detecting fruit in plant or tree canopies. The paper also summarizes the sensors and systems developed and used by researchers to localize fruit as well as the potential and limitations of those systems. Finally, major challenges for the successful application of machine vision system for robotic fruit harvesting and crop-load estimation, and potential future directions for research and development are discussed.

Modeling and real-time simulation architectures for virtual prototyping of off-road vehicles

Virtual Reality-based simulation technology has evolved as a useful design and analysis tool at an early stage in the design for evaluating performance of human-operated agricultural and construction machinery. Detecting anomalies in the design prior to building physical prototypes and expensive testing leads to significant cost savings. The efficacy of such simulation technology depends on how realistically the simulation mimics the real-life operation of the machinery. It is therefore necessary to achieve ‘real-time’ dynamic simulation of such machines with operator-in-the-loop functionality. Such simulation often leads to intensive computational burdens. A distributed architecture was developed for off-road vehicle dynamic models and 3D graphics visualization to distribute the overall computational load of the system across multiple computational platforms. Multi-rate model simulation was also used to simulate various system dynamics with different integration time steps, so that the computational power can be distributed more intelligently. This architecture consisted of three major components: a dynamic model simulator, a virtual reality simulator for 3D graphics, and an interface to the controller and input hardware devices. Several off-road vehicle dynamics models were developed with varying degrees of fidelity, as well as automatic guidance controller models and a controller area network interface to embedded controllers and user input devices. The simulation architecture reduced the computational load to an individual machine and increased the real-time simulation capability with complex off-road vehicle system models and controllers. This architecture provides an environment to test virtual prototypes of the vehicle systems in real-time and the opportunity to test the functionality of newly developed controller software and hardware.

Apple crop-load estimation with over-the-row machine vision system

Innovative concept proposed for apple counting in tall spindle apple trees orchard.Over-the-row system with tunnel structure provide uniform illumination for imaging.Image processing for apple identification on tree canopy images.Repetitive counting of apples from two sides was avoided using 3D information. Accurate crop-load estimation is important for efficient management of pre- and post-harvest operations. This information is crucial for the planning of labor and equipment requirement for harvesting and transporting fruit from the orchard to packing house. Current machine vision-based techniques for crop-load estimation have achieved only limited success mostly due to: (i) occlusion of apples by branches, leaves and/or other apples, and (ii) variable outdoor lighting conditions. In order to minimize the effect of these factors, a new sensor system was developed with an over-the-row platform integrated with a tunnel structure which acquired images from opposite sides of apple trees. The tunnel structure minimized illumination of apples with direct sunlight and reduced the variability in lighting condition. Images captured in a tall spindle orchard were processed for identifying apples, which achieved an identification accuracy of 79.8%. The location of apples in three-dimensional (3D) space was used to eliminate duplicate counting of apples that were visible to cameras from both sides of the tree canopy. The error on identifying duplicate apples was found to be 21.1%. Overall, the method achieved an accuracy of 82% on estimating crop-load on trees with dual side imaging compared to 58% with single side imaging. Over-the-row machine vision system showed promise for accurate and reliable apple crop-load estimation in the apple orchards.

Real-time simulation of dynamic vehicle models using a high-performance reconfigurable platform

With the increase in the complexity of models and lack of flexibility offered by the analog computers, coupled with the advancements in digital hardware, the simulation industry has subsequently moved to digital computers and increased usage of programming languages such as C, C++, and MATLAB. However, the reduced time-step required to simulate complex and fast systems imposes a tighter constraint on the time within which the computations have to be performed. The sequential execution of these computations fails to cope with the real-time constraints which further restrict the usefulness of Real-Time Simulation (RTS) in a Virtual Reality (VR) environment. In this paper, we present a methodology for the design and implementation of RTS algorithms, based on the use of Field-Programmable Gate Array (FPGA) technology. We apply our methodology to an 8th order steering valve subsystem of a vehicle with relatively low response time requirements and use the FPGA technology to improve the response time of this model. Our methodology utilizes traditional hardware/software co-design approaches to generate a heterogeneous architecture for an FPGA-based simulator by porting the computationally complex regions to hardware. The hardware design was optimized such that it efficiently utilizes the parallel nature of FPGAs and pipelines the independent operations. Further enhancement was made by building a hardware component library of custom accelerators for common non-linear functions. The library also stores the information about resource utilization, cycle count, and the relative error with different bit-width combinations for these components, which is further used to evaluate different partitioning approaches. In this paper, we illustrate the partitioning of a hardware-based simulator design across dual FPGAs, initiate RTS using a system input from a Hardware-in-the-Loop (HIL) framework, and use these simulation results from our FPGA-based platform to perform response analysis. The total simulation time, which includes the time required to receive the system input over a socket (without HIL), software initialization, hardware computation, and transfer of simulation results back over a socket, shows a speedup of 2 × as compared to a similar setup with no hardware acceleration. The correctness of the simulation output from the hardware has also been validated with the simulated results from the software-only design.

3D Reconstruction of Apple Trees for Mechanical Pruning

Pruning is a labor intensive operation in fruit production, which constitutes a significant component of total production cost. Automation and mechanization can reduce labor demand from such labor intensive tasks. This work focused on development of a three dimensional (3D) machine vision system to map apple trees for automatic pruning. A sensor platform consisting of a time-of-flight-of-light-based 3D camera and a color vision camera was developed to move the sensors along a row of apple trees. A set of images were collected in a young commercial orchard nearby Washington State University Irrigated Agriculture Research and Extension Center (IAREC), Prosser, WA. The trees were trained in the central leader-based fruiting wall architecture. These 3D images were processed to remove noise and plants that did not belong to the row adjacent to the sensors. A 3D medial axis thinning-based skeletonization algorithm was used to obtain the 3D structures of these trees. An algorithm to identify trunk and branches was introduced. Pruning points were located in the reconstructed tree using a simplified two step pruning rule; remove branches longer than certain value and maintain certain spacing between branches. The threshold values for the length and the spacing were provided as input parameters to the system. Based on the analysis of a reconstructed model of an apple tree, the algorithm achieved about 90% accuracy in identifying trunks, branches and pruning points. Further experiments are needed for physical calibration of 3D skeleton and to develop statistical performance measures of the 3D reconstruction and pruning point identification methods.

Utilizing repeated gps surveys from field operations for development of agricultural field dems.

Topographic data collected using RTK-DGPS-equipped farm vehicles during field operations could addadditional benefits to the original capital investment in the equipment through the development of high-accuracy field DEMs. Repeated surveys of elevation data from field operations may improve DEM accuracy over time. However, minimizing the amount of data to be processed and stored is also an important goal for practical implementation. A method was developed to utilize repeated GPS surveys acquired during field operations for generating field-level DEMs. Elevation measurement error was corrected through a continuity analysis. Fuzzy logic (FL) and weighted averaging (WA) methods were used to combine new surveys with past elevation estimates without requiring storage and reprocessing of past survey data. After 20 surveys were included, the DEM of the study area generated with FL and WA methods had an average root mean squared error (RMSE) of 0.08 m, which was substantially lower than the RMSE of 0.16 m associated with the DEM developed by averaging all data points in each grid. With minimum control of errors in elevation measurements, the effect of these errors can be reduced with appropriate data processing, including continuity analysis, fuzzy logic, and weighted averaging. Two years of GPS surveys of elevation data from field operations could reduce elevation error by 50% in field DEMs.

Assessing the Effects of DEM Uncertainty on Erosion Rate Estimation in an Agricultural Field

The slope length and steepness (LS) factor is one of the factors in the Revised Universal Soil Loss Equation (RUSLE) needed to estimate average annual erosion rate. The LS factor is often derived from digital elevation models (DEM). DEM errors and uncertainty could affect LS factor estimation and consequently erosion rate estimation. However, DEM uncertainties are not always accounted for, and the effects are not always evaluated in erosion rate estimation. This study compared the erosion rate estimation of a 62.81 ha agricultural crop area using a 7.5 min USGS DEM and DEMs developed using real-time kinematic differential GPS (RTK-DGPS) and dual-frequency DGPS (DF-DGPS) field surveys. Spatial estimation and uncertainty analysis was carried out using sequential Gaussian simulation (SGS). A total of 50 equiprobable DEM realizations were produced using SGS to assess DEM uncertainty and quantify its effect on erosion rate estimation. DEM uncertainty substantially affected the resulting erosion rate estimation. The uncertainty of the average annual erosion rate estimates across the study field was represented using 95% confidence intervals (CI). For the DF-DGPS and USGS DEMs, the percentages of the field area that have erosion rate CIs greater than 11.21 Mg ha-1 year-1 (5 tons acre-1 year-1) were 81% and 85%, respectively, which were substantially larger than that of the RTK DEM (0.41%). The average annual erosion rate map produced using a USGS DEM contained artifacts and underestimated the erosion rate estimation in many areas of the field. The results suggested that higher-accuracy DEMs generated using RTK-DGPS measurements are more appropriate for erosion rate estimation in an agricultural field. Knowledge of DEM uncertainty and its effect on the erosion rate estimation was useful to better judge the reliability of erosion rate estimates.

Open and Closed Loop System Characteristics of a Tractor and an Implement Dynamic Model

Accurate guidance of towed implements is important for performing agricultural field operations and for gaining the ultimate benefit from an agricultural automatic guidance system. The study of open and closed loop system responses can be helpful in the design of practical guidance controllers. A dynamic model of a tractor and a towed implement system was developed. Open loop analysis of the kinematic and dynamic models revealed that the dynamic model was essential for capturing the higher order dynamics of the tractor and implement system at higher operating velocities. In addition, a higher fidelity dynamic model was also developed by incorporating steering dynamics and tire relaxation length dynamics. Closed loop system characteristics were studied by using a linear quadratic regulator (LQR) controller. The tractor position and heading and implement heading states along with respective rate states were fed back to close the loop. The higher fidelity closed loop system used a practical range of steering angles and rates to keep the response within nominal off-road vehicle guidance controller design specifications in the forward velocity range of 0.5 m/s to 10 m/s (1.8 km/h to 36 km/h). These simulation studies provided understanding about the characteristics of the tractor and towed implement system and showed promise in assisting in the development of automatic guidance controllers.

Effect of catching surface and tilt angle on bruise damage of sweet cherry due to mechanical impact

Mechanical impact force of fruits is determined by drop height and catching surface.Sufficient sickness of cushion material is required to reduce bruise damage.Fruit bruise damage area is affected heavily by drop height and catching surface.Tilt angle of catching surfaces reduces fruit bruise damage substantially. Fruit bruise damage induced by mechanical impact is the most critical obstacle for the application of mechanical harvesting on fresh-market sweet cherries. One of main sources of fruit bruise is the mechanical impact by fruit catching surfaces occurring in fruit collection during mechanical harvesting. The goal of this research was to investigate the effect of cushion material, fruit drop height, and tilt angle of catching surface on fruit bruise damage. Three catching surfaces with five tilt angles from 0? to 60? were used to catch fruits freely dropped from heights of 0.3-2.1m. The impact force and deformation of cushion materials was measured by a force sensing unit and a high speed camera, respectively. Results showed that maximum impact force increased linearly with drop height and was reduced by cushion materials with sufficient thickness. The fruit damage percentages of cushion material 1 and 2 were 25.0-89.0% and 72.0-100.0% less than that of non-cushion material at drop height of 0.3-2.1m at 0? tilt angle, respectively. Results also shown catching surfaces with tilt angle reduced bruise damage substantially. Damage percentage of catching surfaces at 60? tilt angle was around 75.0% less than that at 0? non-cushion and cushion material when fruit were dropped from 1.5m. The results show that catching surfaces with cushion materials at a tilt angle of 60? might be a promise for mechanical harvesting of fruits with low fruit bruise damage.

Characterizing apple picking patterns for robotic harvesting

Aims at gaining an understanding on the apple fruit detachment process during harvest.Focused on characterizing the detachment process using a few key parameters.Investigated four picking patterns by measuring and analyzing the minimal grasping pressure.Robotic picking requires higher grasping pressure in general than manual picking, which potentially induces higher bruising rate on picked fruit.There exists noticeable difference on required grasping pressure to pick a fruit between different robotic picking patterns. Fruit detachment is one of the essential tasks in apple harvest. The resistance of detaching an apple from the tree is largely influenced by picking patterns. This research aimed at gaining an understanding of fruit detachment process under different picking patterns, focused on characterizing those processes using a few key detaching parameters. It also aimed at identifying an effective robotic picking pattern using a three-finger gripper. To accomplish this goal, one manual and three robotic apple checking patterns were studied, by measuring and analyzing the minimal grasping pressure required to remove a fruit from the tree. The corresponding damage level on removed fruit was also analyzed. The results revealed that manual picking could create a bending moment which helped to reduce the required grasping pressure for fruit detachment, and resulted in no picking-induced fruit bruising on all collected samples. Results obtained from all three robotic picking patterns indicated that the use of a three-finger gripper required higher grasping pressure to detach apples, which resulted in higher percentages of picking-induced fruit bruising. It was found that one of the studied robotic patterns could offer a more manual-like performance than the other two robotic picking patterns. Further investigation assessing potentials and limitations of this identified robotic picking pattern on a more comprehensive scale to gain a deeper understanding of how this pattern works is recommended before it can be used as the base pattern for developing effective and efficient apple picking robots.