Huanyu Jiang

Measurement of 3-D Locations of Ripe Tomato by Binocular Stereo Vision for Tomato Harvesting

A measurement of 3-D locations of ripe tomato by binocular stereo vision was developed for tomato harvesting in greenhouse. In this method, a pair of stereo images was obtained by stereo cameras, and transformed to grey-scale images. According to the grey correlation, corresponding points of the stereo images were searched, and a depth image was obtained by calculating distances between tomatoes and stereo cameras based on triangulation principle. The center of tomato was extracted by distinguishing the tomato from background with image processing. The 3-D locations of ripe tomato were obtained by comparing coordinate values of the center of tomato with the depth image. The error of depth was ranged within ±20 mm when distance was less than 1000 mm.

A recognition algorithm for occluded tomatoes based on circle regression

To realize the automation of harvesting work, automatic recognition of fruits and vegetables should be realized firstly. In the researches on the recognition of fruits and vegetables, the recognition of occluded fruits and vegetables is a difficult point. This paper presents a recognition algorithm for occluded tomatoes based on circle regression. It mainly bases on the principle that there is a big difference in curvatures between edge points on the edge produced by occlusion and edge points on the edge without occlusion. First, the closed edges of occluded tomatoes were extracted after image segmentation. Second, the curvatures of edge points were computed. Third, edge points with abnormal curvatures were removed. Finally, occluded tomatoes were recognized using circle regression method for edge points with normal curvatures. Moreover, in order to reduce false recognition, edge recognition and circle regression rules were also applied in this study. Test results showed that the correct rate of recognition was larger than 90% for tomatoes with little occlusion, but it was not ideal for tomatoes with moderate and serious occlusion.

Tests of a recognition algorithm for clustered tomatoes based on mathematical morphology

Recognition of clustered fruits and vegetables is a most challenging subject in researches on the vision system of harvesting robot. A recognition algorithm for clustered tomatoes based on mathematical morphology was tested. This algorithm mainly included four steps. First, tomato image segmentation was realized based on a normalized color difference. Second, clustered region could be recognized according to the length of the longest edge of the minimum enclosing rectangle of the tomato region. Third, clustered regions in binary image were processed by an iterative erosion course to separate every tomato in this clustered region. Finally, every seed region in the clustered region acquired by the iterative erosion was restored using a circulatory dilation operation. As a result, every tomato in the clustered region was recognized. 99 clustered regions which were classified into two types based on the clustered degree, adhering tomatoes and overlapping tomatoes, were tested using this algorithm. Test results show that the average correct recognition rate for adhering tomatoes at the shooting distance of 500 mm was 87.5%, but that for two kinds of clustered tomatoes at the shooting distance from 300 to 700 mm was only 58.4%.

Three-dimensional location of tomato based on binocular stereo vision for tomato harvesting robot

Accurate harvesting depends on the order of the accuracy of 3D location for harvesting robot. The precision of location is lower when the distance between fruit and camera is larger than 0.8 m for the method based on binocular stereo vision. This is a big problem. In order to improve the precision of depth measurement for ripe tomato, two stereo matching methods were analyzed comparatively which were centroid-based matching and area-based matching. Their performances in depth measurement were also compared. Experiments showed that the relationship between distance and measurement was linear. Then, models of unitary linear regression (ULR) were used to improve the results of depth measurement. After correction by these models, the depth errors were in a range of -28 mm to 25 mm for centroid-based matching method and -8 mm to 15 mm for area-based matching method at a distance of 0.6 m to 1.15 m. It can be concluded that costs of computation can be decreased with the promise of good precision when the parallax of centroid which is acquired through centroid-based matching method is used to set the range of parallax for area-based matching method.

Tests of localization errors of tomatoes based on binocular stereo vision caused by occlusion

The accuracy of the three dimensional localization of the vision systems of harvesting robots is likely to be effected by the occlusion of leaves and branches. To quantitatively analyze the influence to the localization for tomatoes based on binocular stereo vision caused by the occlusion, we analyzed the three dimensional localization errors of occluded tomatoes in three kinds of occlusion degree based on three stereo matching methods. The occlusion degrees were classified to slight occlusion, moderate occlusion and serious occlusion based on the rate of the occluded area of a tomato to the total area of this tomato. Tomato image segmentation was realized using the fixed threshold image segmentation method based on the normalized color difference between the red and green component. Stereo matching methods included centroid-based matching, area-based matching and combination matching. After stereo matching, three dimensional coordinates of tomatoes were computed using the triangle ranging principle. Tests based on 240 stereo images of five tomatoes in three different occlusion degrees captured at distances from 300 to 1000 mm showed that there were little difference for the widths of the error intervals of x coordinates acquired based on three stereo matching methods, but the widths of the error intervals of y coordinates and depth values acquired based on area-based matching and combination matching were smaller. Three dimensional coordinates were all fluctuated with different occluded part and different occlusion degrees of occluded tomatoes. The measurement values of x and y coordinates were more likely to be affected by the occlusion than those of depth.

Research on Recognition Method for Overlapping Tomatoes Based on Depth Map

Overlapping harvesting objects’ recognition is a difficulty of researches in vision system of fruits and vegetables harvesting robots. In order to improve the recognition accuracy rate of overlapping tomatoes, a recognition method of front and back overlapping region is presented, which is based on the automatic threshold image segmentation of depth map. Depth map is obtained through combination stereo matching method based on binocular stereo vision. Denoising processing is applied to depth map before image segmentation of depth map. Segmentation of front and back overlapping region can be realized through iterative OTSU threshold segmentation method because there are obvious difference between front region and back region for depth information. 48 pairs of stereo images have been tested, and the recognition accuracy rate is 96.3%. It shows that this method is effective to recognition of front and back overlapping region. There are also some problems. This method doesn’t adapt to adhesive regions. It should be resolved combining other methods in future researches. This method doesn’t adapt to the images which are grabbed at the distance larger than 800mm either.