Lilong Shi

Quaternion color texture segmentation

The quaternion representation of color is shown here to be effective in the context of segmenting color images into regions of similar color texture. The advantage of using quaternion arithmetic is that a color can be represented and analyzed as a single entity. A low-dimensional basis for the color textures found in a given image is derived via quaternion principal component analysis (QPCA) of a training set of color texture samples. A color texture sample is then projected onto this basis to obtain a concise (single quaternion) description of the texture. To handle the large amount of training data, QPCA is extended to incremental QPCA. The power of the proposed quaternion color texture representation is demonstrated by its use in an unsupervised segmentation algorithm that successfully divides an image into regions on basis of texture.

Illumination estimation via thin-plate spline interpolation.

Thin-plate spline interpolation is used to interpolate the chromaticity of the color of the incident scene illumination across a training set of images. Given the image of a scene under unknown illumination, the chromaticity of the scene illumination can be found from the interpolated function. The resulting illumination-estimation method can be used to provide color constancy under changing illumination conditions and automatic white balancing for digital cameras. A thin-plate spline interpolates over a nonuniformly sampled input space, which in this case is a training set of image thumbnails and associated illumination chromaticities. To reduce the size of the training set, incremental k medians are applied. Tests on real images demonstrate that the thin-plate spline method can estimate the color of the incident illumination quite accurately, and the proposed training set pruning significantly decreases the computation.

Demo: a multi-modal training environment for surgeons

This demonstration presents the current state of an on-going team project at Simon Fraser University in developing a virtual environment for helping to train surgeons in performing laparoscopic surgery. In collaboration with surgeons, an initial set of training procedures has been developed. Our goal has been to develop procedures in each of several general categories, such as basic hand-eye coordination, single-handed and bi-manual approaches and dexterous manipulation. The environment is based on an effective data structure that offers fast graphics and physically based modeling of both rigid and deformable objects. In addition, the environment supports both 3D and 5D input devices and devices generating haptic feedback. The demonstration allows users to interact with a scene using a haptic device.

The effect of exposure on MaxRGB color constancy

The performance of the MaxRGB illumination-estimation method for color constancy and automatic white balancing has been reported in the literature as being mediocre at best; however, MaxRGB has usually been tested on images of only 8-bits per channel. The question arises as to whether the method itself is inadequate, or rather whether it has simply been tested on data of inadequate dynamic range. To address this question, a database of sets of exposure-bracketed images was created. The image sets include exposures ranging from very underexposed to slightly overexposed. The color of the scene illumination was determined by taking an extra image of the scene containing 4 Gretag Macbeth mini Colorcheckers placed at an angle to one another. MaxRGB was then run on the images of increasing exposure. The results clearly show that its performance drops dramatically when the 14-bit exposure range of the Nikon D700 camera is exceeded, thereby resulting in clipping of high values. For those images exposed such that no clipping occurs, the median error in MaxRGBs estimate of the color of the scene illumination is found to be relatively small.

Illumination estimation via nonnegative matrix factorization

Abstract. The problem of illumination estimation for color constancy and automatic white balancing of digital color imagery can be viewed as the separation of the image into illumination and reflectance components. We propose using nonnegative matrix factorization with sparseness constraints to separate these components. Since illumination and reflectance are combined multiplicatively, the first step is to move to the logarithm domain so that the components are additive. The image data is then organized as a matrix to be factored into nonnegative components. Sparseness constraints imposed on the resulting factors help distinguish illumination from reflectance. The proposed approach provides a pixel-wise estimate of the illumination chromaticity throughout the entire image. This approach and its variations can also be used to provide an estimate of the overall scene illumination chromaticity.

Reducing weight precision of convolutional neural networks towards large-scale on-chip image recognition

In this paper, we develop a server-client quantization scheme to reduce bit resolution of deep learning architecture, i.e., Convolutional Neural Networks, for image recognition tasks. Low bit resolution is an important factor in bringing the deep learning neural network into hardware implementation, which directly determines the cost and power consumption. We aim to reduce the bit resolution of the network without sacrificing its performance. To this end, we design a new quantization algorithm called supervised iterative quantization to reduce the bit resolution of learned network weights. In the training stage, the supervised iterative quantization is conducted via two steps on server – apply k-means based adaptive quantization on learned network weights and retrain the network based on quantized weights. These two steps are alternated until the convergence criterion is met. In this testing stage, the network configuration and low-bit weights are loaded to the client hardware device to recognize coming input in real time, where optimized but expensive quantization becomes infeasible. Considering this, we adopt a uniform quantization for the inputs and internal network responses (called feature maps) to maintain low on-chip expenses. The Convolutional Neural Network with reduced weight and input/response precision is demonstrated in recognizing two types of images: one is hand-written digit images and the other is real-life images in office scenarios. Both results show that the new network is able to achieve the performance of the neural network with full bit resolution, even though in the new network the bit resolution of both weight and input are significantly reduced, e.g., from 64 bits to 4-5 bits.

Multi-frame super resolution robust to local and global motion

Super resolution (SR) is to produce a higher resolution image from one or a sequence of low resolution images of a scene. It is essential in medical image analysis as a zooming of a specific area of interest is often required. This paper presents a new multi-frame super resolution (SR) method that is robust to both global and local motion. One of major challenges in multi-frame SR is concurrent global and local motion emergent in the sequence of low resolution images. It poses difficulties in aligning the low resolution images, resulting in artifacts or blurred pixels in the computed high resolution image. We solve the problem via a series of new methods. We first align the upscaled images from bicubic interpolation, and analyze the pixel distribution for the presence of local motion. If local motion is identified, we conduct the local image registration using dense SIFT features. Based on the local registration of images, we analyze pixel locations whose cross-frame variation is high and adaptively select subset of frame pixels in those locations. The adaptive selection of frame pixels is based on a clustering analysis of luminance values of pixels aligned at the same position, such that noise and motion biases are excluded. At the end, a median filter is applied for the selected pixels at each pixel location for super resolution image. We conduct experiments for multi-frame SR, where the proposed method delivers favorable results, especially better than state-of-the-art in dealing with concurrent local and global motions across frames.

Demosaicing for RGBZ sensor

In this paper, we proposed a new technique for demosaicing a unique RGBZ color-depth imaging sensor, which captures color and depth images simultaneously, with a specially designed color-filter-array (CFA) where two out of six RGB color rows are replaced by “Z” pixels that capture depth information but no color information. Therefore, in an RGBZ image, the red, green and blue colors are more sparsely sampled than in a standard Bayer image. Due to the missing rows in the data image, commonly used demosaicing algorithms for the standard Bayer CFA cannot be applied directly. To this end, our method first fills-in the missing rows to reconstruct a full Bayer CFA, followed by a color-selective adaptive demosaicing algorithm that interpolates missing color components. In the first step, unlike common bilinear interpolation approaches that tend to blur edges, our edge-based directional interpolation approach, derived from de-interlacing techniques, emphasizes reconstructing more straight and sharp edges with fewer artifacts and thereby preserves the vertical resolution in the reconstructed the image. In the second step, to avoid using the newly estimated pixels for demosaicing, the bilateral-filter-based approach interpolates the missing color samples based on weighted average of adaptively selected known pixels from the local neighborhoods. Tests show that the proposed method reconstructs full color images while preserving edges details, avoiding artifacts, and removing noise with high efficiency.

Pseudo-random modulation for multiple 3D time-of-flight camera operation

3D time-of-flight depth cameras utilize modulated light sources to detect the distance to objects as phase information. A serious limitation may exist in cases when multiple depth time-of-flight cameras are imaging the same scene simultaneously. The interference caused by the multiple modulated light sources can severely distort captured depth images. To prevent this problem and enable concurrent 3D multi-camera imaging, we propose modulating the camera light source and demodulating the received signal using sequences of pulses, where the phase of each sequence is varied in a pseudo-random fashion. The proposed algorithm is mathematically derived and proved by experiment.

A New Type of ART2 Architecture and Application to Color Image Segmentation

A new neural network architecture based on adaptive resonance theory (ART) is proposed and applied to color image segmentation. A new mechanism of similarity measurement between patterns has been introduced to make sure that spatial information in feature space, including both magnitude and phase of input vector, has been taken into consideration. By these improvements, the new ART2 architecture is characterized by the advantages: (i) keeping the traits of classical ART2 network such as self-organizing learning, categorizing without need of the number of clusters, etc.; (ii) developing better performance in grouping clustering patterns; (iii) improving pattern-shifting problem of classical ART2. The new architecture is believed to achieve effective unsupervised segmentation of color image and it has been experimentally found to perform well in a modified Li¾?ui¾?vi¾?color space in which the perceptual color difference can be measured properly by spatial information.