Kewei Tu

Joint Video and Text Parsing for Understanding Events and Answering Queries

This article proposes a multimedia analysis framework to process video and text jointly for understanding events and answering user queries. The framework produces a parse graph that represents the compositional structures of spatial information (objects and scenes), temporal information (actions and events), and causal information (causalities between events and fluents) in the video and text. The knowledge representation of the framework is based on a spatial-temporal-causal AND-OR graph (S/T/C-AOG), which jointly models possible hierarchical compositions of objects, scenes, and events as well as their interactions and mutual contexts, and specifies the prior probabilistic distribution of the parse graphs. The authors present a probabilistic generative model for joint parsing that captures the relations between the input video/text, their corresponding parse graphs, and the joint parse graph. Based on the probabilistic model, the authors propose a joint parsing system consisting of three modules: video parsing, text parsing, and joint inference. Video parsing and text parsing produce two parse graphs from the input video and text, respectively. The joint inference module produces a joint parse graph by performing matching, deduction, and revision on the video and text parse graphs. The proposed framework has the following objectives: to provide deep semantic parsing of video and text that goes beyond the traditional bag-of-words approaches; to perform parsing and reasoning across the spatial, temporal, and causal dimensions based on the joint S/T/C-AOG representation; and to show that deep joint parsing facilitates subsequent applications such as generating narrative text descriptions and answering queries in the forms of who, what, when, where, and why. The authors empirically evaluated the system based on comparison against ground-truth as well as accuracy of query answering and obtained satisfactory results.

CMC: combining multiple schema-matching strategies based on credibility prediction

Schema matching is a key operation in data engineering. Combining multiple matching strategies is a very promising technique for schema matching. To overcome the limitations of existing combination systems and to achieve better performances, in this paper the CMC system is proposed, which combines multiple matchers based on credibility prediction. We first predict the accuracy of each matcher on the current matching task, and accordingly calculate each matchers credibility. These credibilities are then used as weights in aggregating the matching results of different matchers into a combined one. Our experiments on real world schemas validate the merits of our system.

Unsupervised Learning of Probabilistic Context-Free Grammar using Iterative Biclustering

This paper presents PCFG-BCL, an unsupervised algorithm that learns a probabilistic context-free grammar (PCFG) from positive samples. The algorithm acquires rules of an unknown PCFG through iterative biclustering of bigrams in the training corpus. Our analysis shows that this procedure uses a greedy approach to adding rules such that each set of rules that is added to the grammar results in the largest increase in the posterior of the grammar given the training corpus. Results of our experiments on several benchmark datasets show that PCFG-BCL is competitive with existing methods for unsupervised CFG learning.

ORIENT: integrate ontology engineering into industry tooling environment

Orient is a project to develop an ontology engineering tool that integrates into existing industry tooling environments - the Eclipse platform and the WebSphere Studio developing tools family. This paper describes how two important issues are addressed during the project, namely tool integration and scalability. We show how Orient morphs into the Eclipse platform and achieves UI and data level integration with the Eclipse platform and other modelling tools. We also describe how we implemented a scalable RDF(S) storage, query, manipulation and inference mechanism on top of a relational database. In particular, we report the empirical performance of our RDF(S) closure inference algorithm on a DB2 database.

An approach to RDF(S) query, manipulation and inference on databases

In order to lay a solid foundation for the emerging semantic web, effective and efficient management of large RDF(S) data is in high demand. In this paper we propose an approach to the storage, query, manipulation and inference of large RDF(S) data on top of relational databases. Specifically, RDF(S) inference is done on the database in advance instead of on the fly, so that the query efficiency is maximized. To reduce the cost of inference, two inference modes, the batch mode and the incremental mode, are provided for different scenarios. In both modes, optimized strategies are applied for efficiency purpose. In order to support efficient query and inference on the database, the storage schema is also specially designed. In addition, a powerful RDF(S) query and manipulation language RQML is provided for easy and uniform data access in a declarative way. Finally, we evaluate and report the performance on both query and inference of our approach. Experiments show that our approach achieves encouraging performance in million-scale real data.

Context-Dependent Sense Embedding.

Word embedding has been widely studied and proven helpful in solving many natural language processing tasks. However, the ambiguity of natural language is always a problem on learning high quality word embeddings. A possible solution is sense embedding which trains embedding for each sense of words instead of each word. Some recent work on sense embedding uses context clustering methods to determine the senses of words, which is heuristic in nature. Other work creates a probabilistic model and performs word sense disambiguation and sense embedding iteratively. However, most of the previous work has the problems of learning sense embeddings based on imperfect word embeddings as well as ignoring the dependency between sense choices of neighboring words. In this paper, we propose a novel probabilistic model for sense embedding that is not based on problematic word embedding of polysemous words and takes into account the dependency between sense choices. Based on our model, we derive a dynamic programming inference algorithm and an Expectation-Maximization style unsupervised learning algorithm. The empirical studies show that our model outperforms the state-of-the-art model on a word sense induction task by a 13% relative gain.

Mapping the Energy Landscape of Non-convex Optimization Problems

An energy landscape map (ELM) characterizes and visualizes an energy function with a tree structure, in which each leaf node represents a local minimum and each non-leaf node represents the barrier between adjacent energy basins. We demonstrate the utility of ELMs in analyzing non-convex energy minimization problems with two case studies: clustering with Gaussian mixture models and learning mixtures of Bernoulli templates from images. By plotting the ELMs, we are able to visualize the impact of different problem settings on the energy landscape as well as to examine and compare the behaviors of different learning algorithms on the ELMs.

Semi-supervised Structured Prediction with Neural CRF Autoencoder

In this paper we propose an end-toend neural CRF autoencoder (NCRF-AE) model for semi-supervised learning of sequential structured prediction problems. Our NCRF-AE consists of two parts: an encoder which is a CRF model enhanced by deep neural networks, and a decoder which is a generative model trying to reconstruct the input. Our model has a unified structure with different loss functions for labeled and unlabeled data with shared parameters. We developed a variation of the EM algorithm for optimizing both the encoder and the decoder simultaneously by decoupling their parameters. Our experimental results over the Part-of-Speech (POS) tagging task on eight different languages, show that the NCRF-AE model can outperform competitive systems in both supervised and semi-supervised scenarios.

Learning Bayesian network structures under incremental construction curricula

Abstract Bayesian networks have been successfully applied to various tasks for probabilistic reasoning and causal modeling. One major challenge in the application of Bayesian networks is to learn the Bayesian network structures from data. In this paper, we take advantage of the idea of curriculum learning and learn Bayesian network structures by stages. At each stage a subnet is learned over a selected subset of the random variables. The selected subset grows with stages and eventually includes all the variables. We show that in our approach each target subnet is closer to the target Bayesian network than any of its predecessors. The experimental results show that our algorithm outperformed the state-of-the-art heuristic approach in learning Bayesian network structures under several different evaluation metrics.

A Semantic Search Approach by Graph Matching with Negations and Inferences

Research on semantic search has become heated these years. In this paper we propose an approach focusing on searching for resources with descriptions. The knowledge representation we employ is based on conceptual graphs and is expressive with negation. We carry out semantic search by graph matching, which can be performed in polynomial time. Before matching we enrich the resource graphs with background knowledge by a deductive graph inference, so as to improve the search performance. The processing of negat ions and the graph inference method are two important contributions of this paper.