Hayaru Shouno

Analysis of function of rectified linear unit used in deep learning

Deep Learning is attracting much attention in object recognition and speech processing. A benefit of using the deep learning is that it provides automatic pre-training. Several proposed methods that include auto-encoder are being successfully used in various applications. Moreover, deep learning uses a multilayer network that consists of many layers, a huge number of units, and huge amount of data. Thus, executing deep learning requires heavy computation, so deep learning is usually utilized with parallel computation with many cores or many machines. Deep learning employs the gradient algorithm, however this traps the learning into the saddle point or local minima. To avoid this difficulty, a rectified linear unit (ReLU) is proposed to speed up the learning convergence. However, the reasons the convergence is speeded up are not well understood. In this paper, we analyze the ReLU by a using simpler network called the soft-committee machine and clarify the reason for the speedup. We also train the network in an on-line manner. The soft-committee machine provides a good test bed to analyze deep learning. The results provide some reasons for the speedup of the convergence of the deep learning.

Distribution estimation of hyperparameters in Markov random field models

We developed a method of distribution estimation of hyperparameters in Markov random field (MRF) models. This study was motivated by the growing quantity of image data in natural sciences owing to recent advances in measurement techniques. MRF models are used to restore images in information science, and the hyperparameters of these models can be adjusted to improve restoration performance. The parameters appearing in data analysis represent physical quantities such as diffusion coefficients. Indeed, many frameworks of hyperparameter estimation have been proposed, but most are point estimation that is susceptible to stochastic fluctuations. Distribution estimation can be used to evaluate the confidence one has in point estimates of hyperparameters, in a similar way to physicists using error bars when they evaluate important physical quantities. We use a solvable MRF model to investigate the performance of distribution estimation in simulations.

Dark channel prior based blurred image restoration method using total variation and morphology

The blurred image restoration method can dramatically highlight the image details and enhance the global contrast, which is of benefit to improvement of the visual effect during practical applications. This paper is based on the dark channel prior principle and aims at the prior information absent blurred image degradation situation. A lot of improvements have been made to estimate the transmission map of blurred images. Since the dark channel prior principle can effectively restore the blurred image at the cost of a large amount of computation, the total variation (TV) and image morphology transform (specifically top-hat transform and bottom-hat transform) have been introduced into the improved method. Compared with original transmission map estimation methods, the proposed method features both simplicity and accuracy. The estimated transmission map together with the element can restore the image. Simulation results show that this method could inhibit the ill-posed problem during image restoration, meanwhile it can greatly improve the image quality and definition.

Dictionary-Based Image Denoising by Fused-Lasso Atom Selection

We proposed an efficient image denoising scheme by fused lasso with dictionary learning. The scheme has two important contributions. The first one is that we learned the patch-based adaptive dictionary by principal component analysis (PCA) with clustering the image into many subsets, which can better preserve the local geometric structure. The second one is that we coded the patches in each subset by fused lasso with the clustering learned dictionary and proposed an iterative Split Bregman to solve it rapidly. We present the capabilities with several experiments. The results show that the proposed scheme is competitive to some excellent denoising algorithms.

Correlated topographic analysis: estimating an ordering of correlated components

This paper describes a novel method, which we call correlated topographic analysis (CTA), to estimate non-Gaussian components and their ordering (topography). The method is inspired by a central motivation of recent variants of independent component analysis (ICA), namely, to make use of the residual statistical dependency which ICA cannot remove. We assume that components nearby on the topographic arrangement have both linear and energy correlations, while far-away components are statistically independent. We use these dependencies to fix the ordering of the components. We start by proposing the generative model for the components. Then, we derive an approximation of the likelihood based on the model. Furthermore, since gradient methods tend to get stuck in local optima, we propose a three-step optimization method which dramatically improves topographic estimation. Using simulated data, we show that CTA estimates an ordering of the components and generalizes a previous method in terms of topography estimation. Finally, to demonstrate that CTA is widely applicable, we learn topographic representations for three kinds of real data: natural images, outputs of simulated complex cells and text data.

A Transfer Learning Method with Deep Convolutional Neural Network for Diffuse Lung Disease Classification

We introduce a deep convolutional neural network (DCNN) as feature extraction method in a computer aided diagnosis (CAD) system in order to support diagnosis of diffuse lung diseases (DLD) on high-resolution computed tomography (HRCT) images. DCNN is a kind of multi layer neural network which can automatically extract features expression from the input data, however, it requires large amount of training data. In the field of medical image analysis, the number of acquired data is sometimes insufficient to train the learning system. Overcoming the problem, we apply a kind of transfer learning method into the training of the DCNN. At first, we apply massive natural images, which we can easily collect, for the pre-training. After that, small number of the DLD HRCT image as the labeled data is applied for fine-tuning. We compare DCNNs with training of (i) DLD HRCT images only, (ii) natural images only, and (iii) DLD HRCT images + natural images, and show the result of the case (iii) would be better DCNN feature rather than those of others.

Deterministic Algorithm for Nonlinear Markov Random Field Model

We propose a deterministic algorithm for image restoration using a nonlinear Markov random field (MRF) model. Recent advances in measurement techniques allow us to obtain a large quantity of imaging data in various natural science fields. These data are often exposed to observation noise. For the removal of noise from imaging data, we use an MRF model, in which the Bayesian inference framework enables us to estimate hyperparameters through free-energy minimization. When a nonlinear function represents an observation process, a Markov chain Monte Carlo (MCMC) method is often used for image restoration. An MCMC method retains nonlinearity, but it is a probabilistic algorithm, which increases computational cost. The proposed deterministic algorithm linearizes the observation process to achieve more efficient hyperparameter estimation and image restoration. We also applied the proposed algorithm to artificial images to show its efficiency.

Bayesian Image Restoration for Medical Images Using Radon Transform

We propose an image reconstruction algorithm using Bayesian inference for Radon transformed observation data, which often appears in the field of medical image reconstruction known as computed tomography (CT). In order to apply our Bayesian reconstruction method, we introduced several hyper-parameters that control the ratio between prior information and the fidelity of the observation process. Since the quality of the reconstructed image is influenced by the estimation accuracy of these hyper-parameters, we propose an inference method for them based on the marginal likelihood maximization principle as well as the image reconstruction method. We are able to demonstrate a reconstruction result superior to that obtained using the conventional filtered back projection method.

A Next Generation Modeling Environment PLATO: Platform for Collaborative Brain System Modeling

To understand the details of brain function, a large scale system model that reflects anatomical and neurophysiological characteristics needs to be implemented. Though numerous computational models of different brain areas have been proposed, these integration for the development of a large scale model have not yet been accomplished because these models were described by different programming languages, and mostly because they used different data formats. This paper introduces a platform for a collaborative brain system modeling (PLATO) where one can construct computational models using several programming languages and connect them at the I/O level with a common data format. As an example, a whole visual system model including eye movement, eye optics, retinal network and visual cortex is being developed. Preliminary results demonstrate that the integrated model successfully simulates the signal processing flow at the different stages of visual system.

Classification of patterns for diffuse lung diseases in thoracic CT images by AdaBoost algorithm

CT images are considered as effective for differential diagnosis of diffuse lung diseases. However, the diagnosis of diffuse lung diseases is a difficult problem for the radiologists, because they show a variety of patterns on CT images. So, our purpose is to construct a computer-aided diagnosis (CAD) system for classification of patterns for diffuse lung diseases in thoracic CT images, which gives both quantitative and objective information as a second opinion, to decrease the burdens of radiologists. In this article, we propose a CAD system based on the conventional pattern recognition framework, which consists of two sub-systems; one is feature extraction part and the other is classification part. In the feature extraction part, we adopted a Gabor filter, which can extract patterns such like local edges and segments from input textures, as a feature extraction of CT images. In the recognition part, we used a boosting method. Boosting is a kind of voting method by several classifiers to improve decision precision. We applied AdaBoost algorithm for boosting method. At first, we evaluated each boosting component classifier, and we confirmed they had not enough performances in classification of patterns for diffuse lung diseases. Next, we evaluated the performance of boosting method. As a result, by use of our system, we could improve the classification rate of patterns for diffuse lung diseases.