Wai Ho Li

Segmentation and modeling of visually symmetric objects by robot actions

Robots usually carry out object segmentation and modeling passively. Sensors such as cameras are actuated by a robot without disturbing objects in the scene. In this paper, we present an intelligent robotic system that physically moves objects in an active manner to perform segmentation and modeling using vision. By visually detecting bilateral symmetry, our robot is able to segment and model objects through controlled physical interactions. Extensive experiments show that our robot is able to accurately segment new objects autonomously. We also show that our robot is able to leverage segmentation results to autonomously learn visual models of new objects by physically grasping and rotating them. Object recognition experiments confirm that the robot-learned models allow robust recognition. Videos of the robotic experiments are also made available.

Real Time Object Tracking using Reflectional Symmetry and Motion

Many objects found in domestic environments are reflectionally symmetric. In this paper, we present a system that can visually track moving objects by their reflectional symmetry in real time. Apart from the assumption of symmetry, the tracking system does not require any prior object models of the target, such as its colour and shape. The system is robust to shadows and specular reflections. It can also deal with transparent objects. Block motion detection is used in conjunction with symmetry for object tracking. A Kalman filter is used to estimate the object state. Predictions from the Kalman filter is used to improve the efficiency of the symmetry detector. The tracker provides a real time segmentation of an object by searching for motion that is symmetric about the objects mirror line. The tracking system also generates a rotated bounding box, aligned with the objects symmetry line, which can be used as a window for other image processing operations. The final system can track single objects in 640times480 videos at over 40 frames per second using a standard notebook PC

Robust egomotion estimation using ICP in inverse depth coordinates

This paper presents a 6 degrees of freedom egomotion estimation method using Iterative Closest Point (ICP) for low cost and low accuracy range cameras such as the Microsoft Kinect. Instead of Euclidean coordinates, the method uses inverse depth coordinates which better conforms to the error characteristics of raw sensor data. Novel inverse depth formulations of point-to-point and point-to-plane error metrics are derived as part of our implementation. The implemented system runs in real time at an average of 28 frames per second (fps) on a standard computer. Extensive experiments were performed to evaluate different combinations of error metrics and parameters. Results show that our system is accurate and robust across a variety of motion trajectories. The point-to-plane error metric was found to be the best at coping with large inter-frame motion while remaining accurate and maintaining real time performance.

Interactive learning of visually symmetric objects

This paper describes a robotic system that learns visual models of symmetric objects autonomously. Our robot learns by physically interacting with an object using its end effector. This departs from eye-in-hand systems that move the camera while keeping the scene static. Our robot leverages a simple nudge action to obtain the motion segmentation of an object in stereo. The robot uses the segmentation results to pick up the object. The robot collects training images by rotating the grasped object in front of a camera. Robotic experiments show that this interactive object learning approach can deal with top-heavy and fragile objects. Trials confirm that the robot-learned object models allow robust object recognition.

Distributed visual processing for augmented reality

Recent advances have made augmented reality on smartphones possible but these applications are still constrained by the limited computational power available. This paper presents a system which combines smartphones with networked infrastructure and fixed sensors and shows how these elements can be combined to deliver real-time augmented reality. A key feature of this framework is the asymmetric nature of the distributed computing environment. Smartphones have high bandwidth video cameras but limited computational ability. Our system connects multiple smartphones through relatively low bandwidth network links to a server with large computational resources connected to fixed sensors that observe the environment. By contrast to other systems that use preprocessed static models or markers, our system has the ability to rapidly build dynamic models of the environment on the fly at frame rate. We achieve this by processing data from a Microsoft Kinect, to build a trackable point cloud model of each frame. The smartphones process their video camera data on-board to extract their own set of compact and efficient feature descriptors which are sent via WiFi to a server. The server runs computationally intensive algorithms including feature matching, pose estimation and occlusion testing for each smartphone. Our system demonstrates real-time performance for two smartphones.

Real Time Detection and Segmentation of Reflectionally Symmetric Objects in Digital Images

Symmetry is a salient visual feature of many man-made objects. This paper describes research into the detection and segmentation of reflectionally symmetric objects in digital images, without the use of a priori object models. The detection method does not assume uniform object colour or texture, and does not rely on prebuilt models such as 3D geometric primitives. A novel detection algorithm has been developed to find lines of reflectional symmetry in images. This detection algorithm can operate at 10 frames per second on 640 by 480 pixel images. Using the detected symmetry, objects are segmented with a dynamic programming approach. Both algorithms have been extended to accommodate skew symmetry

Transformative Reality: Improving bionic vision with robotic sensing

Implanted visual prostheses provide bionic vision with very low spatial and intensity resolution when compared against healthy human vision. Vision processing converts camera video to low resolution imagery for bionic vision with the aim of preserving salient features such as edges. Transformative Reality extends and improves upon traditional vision processing in three ways. Firstly, a combination of visual and non-visual sensors are used to provide multi-modal data of a persons surroundings. This enables the sensing of features that are difficult to sense with only a camera. Secondly, robotic sensing algorithms construct models of the world in real time. This enables the detection of complex features such as navigable empty ground or people. Thirdly, models are visually rendered so that visually complex entities such as people can be effectively represented in low resolution. Preliminary simulated prosthetic vision trials, where a head mounted display is used to constrain a subjects vision to 25×25 binary phosphenes, suggest that Transformative Reality provides functional bionic vision for tasks such as indoor navigation, object manipulation and people detection in scenes where traditional processing is unusable.

Autonomous segmentation of Near-Symmetric objects through vision and robotic nudging

This paper details a robust and accurate segmentation method for near-symmetric objects placed on a table of known geometry. Here we define visual segmentation as the problem of isolating all portions of an image that belongs to a physically coherent object. The term near-symmetric is used as our method can segment objects with some non-symmetric parts, such as a coffee mug and its handle. Using bilateral symmetry this problem is solved autonomously and robustly through the aid of physical action provided by a robot manipulator. Our proposed approach does not require prior models of target objects and assumes no previously collected background statistics. Instead, our approach relies on a precise robotic nudge to generate the necessary object motion to perform segmentation. Experiments performed on ten objects show that our model-free approach can autonomously and accurately segment a variety of objects. These experiments also indicate that our segmentation approach is not adversely affected when operating in cluttered scenes and can segment multi-coloured and transparent objects in a robust manner.

Going beyond vision to improve bionic vision

Currently, most implanted visual prosthetic systems generate vision by translating sensor data from a headworn camera into electrical stimulation of the human vision system. Unfortunately, the resulting bionic vision has low spatial resolution and limited dynamic range. This dramatically reduces the usefulness of bionic vision in many real world scenarios. Historically, this problem is treated as immutable pathology. Recently, image processing has been proposed as a potential remedy to improve the useability of bionic vision. We explore another alternative: Combining multiple sensing modalities and robotic sensing algorithms. This paper gives a top level summary of ongoing research exploring this alternative.

Bilateral Symmetry Detection for Real-time Robotics Applications

Bilateral symmetry is a salient visual feature of many man-made objects. In this paper, we present research that uses bilateral symmetry to identify, segment and track objects in real time using vision. Apart from the assumption of symmetry, the algorithms presented do not require any object models, such as color, shape or three-dimensional primitives. In order to counter the high computational cost of traditional symmetry detection methods, a novel computationally efficient algorithm is proposed. To investigate symmetry as an object feature, our fast detection scheme is applied to the tasks of object detection, segmentation and tracking. We find that objects with a line of symmetry can be segmented without relying on color or shape models by using a dynamic programming approach. Object tracking is achieved by estimating symmetry line parameters using a Kalman filter. The tracker operates at 40 frames per second on 640 x 480 video while running on a standard laptop PC. We use 10 difficult real-world tracking sequences to test our approach. We also quantitatively analyze symmetry as a tracking feature by comparing detected symmetry lines against ground truth. Color tracking is also performed to provide a qualitative comparison.