Son N. Tran

Improving search personalisation with dynamic group formation

Recent research has shown that the performance of search engines can be improved by enriching a users personal profile with information about other users with shared interests. In the existing approaches, groups of similar users are often statically determined, e.g., based on the common documents that users clicked. However, these static grouping methods are query-independent and neglect the fact that users in a group may have different interests with respect to different topics. In this paper, we argue that common interest groups should be dynamically constructed in response to the users input query. We propose a personalisation framework in which a user profile is enriched using information from other users dynamically grouped with respect to an input query. The experimental results on query logs from a major commercial web search engine demonstrate that our framework improves the performance of the web search engine and also achieves better performance than the static grouping method.

Temporal latent topic user profiles for search personalisation

The performance of search personalisation largely depends on how to build user profiles effectively. Many approaches have been developed to build user profiles using topics discussed in relevant documents, where the topics are usually obtained from human-generated online ontology such as Open Directory Project. The limitation of these approaches is that many documents may not contain the topics covered in the ontology. Moreover, the human-generated topics require expensive manual effort to determine the correct categories for each document. This paper addresses these problems by using Latent Dirichlet Allocation for unsupervised extraction of the topics from documents. With the learned topics, we observe that the search intent and user interests are dynamic, i.e., they change from time to time. In order to evaluate the effectiveness of temporal aspects in personalisation, we apply three typical time scales for building a long-term profile, a daily profile and a session profile. In the experiments, we utilise the profiles to re-rank search results returned by a commercial web search engine. Our experimental results demonstrate that our temporal profiles can significantly improve the ranking quality. The results further show a promising effect of temporal features in correlation with click entropy and query position in a search session.

Deep Logic Networks: Inserting and Extracting Knowledge From Deep Belief Networks

Developments in deep learning have seen the use of layerwise unsupervised learning combined with supervised learning for fine-tuning. With this layerwise approach, a deep network can be seen as a more modular system that lends itself well to learning representations. In this paper, we investigate whether such modularity can be useful to the insertion of background knowledge into deep networks, whether it can improve learning performance when it is available, and to the extraction of knowledge from trained deep networks, and whether it can offer a better understanding of the representations learned by such networks. To this end, we use a simple symbolic language—a set of logical rules that we call confidence rules—and show that it is suitable for the representation of quantitative reasoning in deep networks. We show by knowledge extraction that confidence rules can offer a low-cost representation for layerwise networks (or restricted Boltzmann machines). We also show that layerwise extraction can produce an improvement in the accuracy of deep belief networks. Furthermore, the proposed symbolic characterization of deep networks provides a novel method for the insertion of prior knowledge and training of deep networks. With the use of this method, a deep neural–symbolic system is proposed and evaluated, with the experimental results indicating that modularity through the use of confidence rules and knowledge insertion can be beneficial to network performance.

Speaker recognition with hybrid features from a deep belief network

Learning representation from audio data has shown advantages over the handcrafted features such as mel-frequency cepstral coefficients (MFCCs) in many audio applications. In most of the representation learning approaches, the connectionist systems have been used to learn and extract latent features from the fixed length data. In this paper, we propose an approach to combine the learned features and the MFCC features for speaker recognition task, which can be applied to audio scripts of different lengths. In particular, we study the use of features from different levels of deep belief network for quantizing the audio data into vectors of audio word counts. These vectors represent the audio scripts of different lengths that make them easier to train a classifier. We show in the experiment that the audio word count vectors generated from mixture of DBN features at different layers give better performance than the MFCC features. We also can achieve further improvement by combining the audio word count vector and the MFCC features.

Discriminative learning and inference in the Recurrent Temporal RBM for melody modelling

We are interested in modelling musical pitch sequences in melodies in the symbolic form. The task here is to learn a model to predict the probability distribution over the various possible values of pitch of the next note in a melody, given those leading up to it. For this task, we propose the Recurrent Temporal Discriminative Restricted Boltzmann Machine (RTDRBM). It is obtained by carrying out discriminative learning and inference as put forward in the Discriminative RBM (DRBM), in a temporal setting by incorporating the recurrent structure of the Recurrent Temporal RBM (RTRBM). The model is evaluated on the cross entropy of its predictions using a corpus containing 8 datasets of folk and chorale melodies, and compared with n-grams and other standard connectionist models. Results show that the RTDRBM has a better predictive performance than the rest of the models, and that the improvement is statistically significant.

Learning motion-difference features using Gaussian restricted Boltzmann machines for efficient human action recognition

Learning visual words from video frames is challenging because deciding which word to assign to each subset of frames is a difficult task. For example, two similar frames may have different meanings in describing human actions such as starting to run and starting to walk. In order to associate richer information to vector-quantization and generate visual words, several approaches have been proposed recently that use complex algorithms to extract or learn spatio-temporal features from 3-D volumes of video frames. In this paper, we propose an efficient method to use Gaussian RBMs for learning motion-difference features from actions in videos. The difference between two video frames is defined by a subtraction function of one frame by another that preserves positive and negative changes, thus creating a simple spatio-temporal saliency map for an action. This subtraction function removes, by construction, the common shapes and background images that should not be relevant for action learning and recognition, and highlights the movement patterns in space, making it easier to learn the actions from such saliency maps using shallow feature learning models such as RBMs. In the experiments reported in this paper, we used a Gaussian restricted Boltzmann machine to learn the actions from saliency maps of different motion images. Despite its simplicity, the motion-difference method achieved very good performance in benchmark datasets, specifically the Weizmann dataset (98.81%) and the KTH dataset (88.89%). A comparative analysis with hand-crafted and learned features using similar classifiers indicates that motion-difference can be competitive and very efficient.

Generalising the Discriminative Restricted Boltzmann Machines

We present a novel theoretical result that generalises the Discriminative Restricted Boltzmann Machine (DRBM). While originally the DRBM was defined assuming the \(\{0, 1\}\)-Bernoulli distribution in each of its hidden units, this result makes it possible to derive cost functions for variants of the DRBM that utilise other distributions, including some that are often encountered in the literature. This paper shows that this function can be extended to the Binomial and \(\{-1,+1\}\)-Bernoulli hidden units.

Adaptive Transferred-profile Likelihood Learning.

The recent success of representation learning is built upon the learning of relevant features, in particular from unlabelled data available in different domains. This raises the question of how to transfer and reuse such knowledge effectively so that the learning of a new task can be made easier or be improved. This poses a difficult challenge for the area of transfer learning where there is no label in the source data, and no source data is ever transferred to the target domain. In previous work, the most capable approach has been self-taught learning which, however, relies heavily upon the compatibility across the domains. In this paper, we propose a novel transfer learning framework called Adaptive Transferred-profile Likelihood Learning (aTPL), which performs adaptation on the representations to be transferred, so that they become more compatible with the target domain. At the same time, it learns supplementary knowledge about the target domain. Experiments on five images datasets and a sentiment dataset demonstrate the effectiveness of the approach in comparison with self-taught learning and other common feature extraction methods. The results also indicate that the new transfer method is less sensitive to negative transfer.

Efficient representation ranking for transfer learning

Representation learning has emerged recently as a useful tool in the extraction of features from data. In a range of applications, features learned from data have been shown superior to their hand-crafted counterpart. Many deep learning approaches have taken advantage of such feature extraction. However, further research is needed on how such features can be evaluated for re-use in related applications, hopefully then improving performance on such applications. In this paper, we present a new method for ranking the representations learned by a Restricted Boltzmann Machine, which has been used regularly as a feature learner by deep networks. We show that high-ranking features, according to our method, should capture more information than low-ranking ones. We then apply representation ranking for pruning the network, and propose a new transfer learning algorithm, which uses such features extracted from a trained network to improve learning performance in another network trained on an analogous domain. We show that by transferring a small number of highest scored representations from source domain our method encourages the learning of new knowledge in target domain while preserving most of the information of the source domain during the transfer. This transfer learning is similar to self-taught learning in that it does not use the source domain data during the transfer process.

Low-Cost Representation for Restricted Boltzmann Machines

This paper presents a method for extracting a low-cost representation from restricted Boltzmann machines. The new representation can be considered as a compression of the network, requiring much less storage capacity while reasonably preserving the network’s performance at feature learning. We show that the compression can be done by converting the weight matrix of real numbers into a matrix of three values { − 1, 0, 1} associated with a score vector of real numbers. This set of values is similar enough to Boolean values which help us further translate the representation into logical rules. In the experiments reported in this paper, we evaluate the performance of our compression method on image datasets, obtaining promising results. Experiments on the MNIST handwritten digit classification dataset, for example, have shown that a 95% saving in memory can be achieved with no significant drop in accuracy.