Guolong Wang

Human activities prediction by learning combinatorial sparse representations

Human activities prediction is to enable early recognition of unfinished activities from videos only containing the beginning parts, which is a challenge problem. Prediction of human activities is necessarily applied in particular scenes(e.g. surveillance systems, human-computer interfaces). To solve this problem, we propose a novel framework which classifies videos into activity classes by using combinatorial sparse representations (CSR). The major contributions of our work include: (1) dividing each video into multiple equal-length segments, where the local spatio-temporal features extracted; (2) concatenating combinatorial sparse activity dictionaries, formed by overcomplete dictionary of each segment; (3) computing combinatorial sparse coefficients of each segment, based on activity dictionaries above; (4) formulating probability of each activity to estimate the correct class. The results of our experiments show that the proposed method outperforms existing state-of-the-art comparison methods.

Recognizing Emotions Based on Human Actions in Videos

Systems for automatic analysis of videos are in high demands as videos are expanding rapidly on the Internet and understanding of the emotions carried by the videos (e.g. “anger”, “happiness”) are becoming a hot topic. While existing affective computing model mainly focusing on facial expression recognition, little attempts have been made to explore the relationship between emotion and human action. In this paper, we propose a comprehensive emotion classification framework based on spatio-temporal volumes built with human actions. To each action unit we get before, we use Dense-SIFT as descriptor and K-means to form histograms. Finally, the histograms are sent to the mRVM and recognizing the human emotion. The experiment results show that our method performs well on FABO dataset.

Collision-Free LSTM for Human Trajectory Prediction

Pedestrians have an intuitive ability for navigation to avoid obstacles and nearby pedestrians. If we want to predict future positions of a pedestrian, we should know how the pedestrian adjust his direction to avoid collisions. In this work, we present a simple and effective framework for human trajectory prediction to generate the future sequence based on pedestrian past positions. The method, called Collision-Free LSTM, extends the classical LSTM by adding Repulsion pooling layer to share hidden-states of neighboring pedestrians. The model can learn both the temporal information of trajectories and the interactions between pedestrians, which is in contrast to traditional methods using hand-crafted features such as Social forces. The experiments results on two public datasets show that our model can achieve state-of-the-art performance with assessment metrics.

Classification and Software Culture

Peculiar to a multitude of principles, computer software could be grouped into different classes. The classification is partly based on the culture of developers and target users, and the process itself is also a reflection of the culture. Sticking to the different ages of development of mankind to analyze the different classes of software, at first were database systems. As data amount skyrocketed, those systems expressed transplanting, data redundancy, space waste and update difficulty encumbrances, which hindered their ontogenesis. As a consequence was hierarchical, grid database, relational, and non-relational database systems. Following the problem-solution patterns, DOS system ran and operated disks. Driven by users solicitation, it went through a series of redesigns, and finally its legendary predecessor, the windows OS. Together with its principal rivals, the Linux OS and apple Mac OS, they imparted variegated features in manifold ways, thence storifying the different classes of operating systems. Portable and Mobile operating systems were introduced for the sake of mobile computing. The latter interacted with users meanwhile underlying drivers systems managed device hardware. In response to the multifariousness of software systems emerged different classes of programming languages to dispatch heterogenous prospects such as object-oriented programming, script language, and many more. Compilers conjointly forked into distinctive classes transforming incommensurable codes. Game software shifted from console to networking with the proliferation of the internet. Threatens to security bred assorted target-purposed security software. In the chapter below, we will look into these classifications and how they chaperoned mankind all along.

Fundamentals of Software Culture

Diving back as early as the fourth-century BC, China had been using a prototype of abacus called counting boards. With regard to the current dominance of China as a major contributor to Science and Technology, it is obvious that computing has played a key role in this palmy rise. Nowadays, computing runs throughout the entire of our society, from the granular microscopic level of image processing applied in facial recognition to a bigger macro-standing such as port management automation. Computation has been driving all angles through the whole of our civilization since the apparition of the abacus. Computers have proven to be the best and the pillar of the most top inventions of mankind. They reflected the desire of humans to faster perform heavier and cumbersome computations. Later came the requisition for precision and accuracy of evaluations and as a consequence, computing bolstered up from mechanical and manual where they were engaged in printing manuscripts to digital and finally quantum computers which accommodate humanity in bringing to fruition teleportation and lighting data conveyance from earth to satellites in space. Here and now, computers have grown so powerful and versatile that they impersonate every single corner in our syndicates, hence people call up on I/O peripheral devices and media channels such as joysticks and wireless channels respectively to communicate with them and ease the communication exchange. In the chapter below, we shall sequentially recall how these events made history and shaped our future. 1.1 From History to Culture Philosopher Karl Marx argues that what distinguish human beings from animals are tools, the product of human’s intelligence, experience, and creativity [1]. The evolutionary history of tools is the evolutionary history of human civilization. As the most important tool nowadays, computer, is absolutely a sign of culture. And software, the approaches to control computer, is the crystallization of human wisdom. In this chapter, we will introduce computer, the stage of software, from its history to culture.

The History of Computing

Diving back as early as the fourth-century BC, China had been using a prototype of abacus called counting boards. With regard to the current dominance of China as a major contributor to Science and Technology, it is obvious that computing has played a key role in this palmy rise. Nowadays, computing runs throughout the entire of our society, from the granular microscopic level of image processing applied in facial recognition to a bigger macro-standing such as port management automation. Computation has been driving all angles through the whole of our civilization since the apparition of the abacus. Computers have proven to be the best and the pillar of the most top inventions of mankind. They reflected the desire of humans to faster perform heavier and cumbersome computations. Later came the requisition for precision and accuracy of evaluations and as a consequence, computing bolstered up from mechanical and manual where they were engaged in printing manuscripts to digital and finally quantum computers which accommodate humanity in bringing to fruition teleportation and lighting data conveyance from earth to satellites in space. Here and now, computers have grown so powerful and versatile that they impersonate every single corner in our syndicates, hence people call up on I/O peripheral devices and media channels such as joysticks and wireless channels respectively to communicate with them and ease the communication exchange. In the chapter below, we shall sequentially recall how these events made history and shaped our future.

The Development of Software

Culture can be regarded to be that normative glue withholding a coterie together. It is associated to the standardized approach of reasoning and behavior of our kind. Being engulfed in the hegemony of computing, civilization indispensable vocation narrowed down to how to convey their conjectures to computers so that the latter would by dint of effective logic put together an output, henceforth software came into existence. Two centuries earlier, humanity desired hypersonic productivity, thus shepherding Charles Babbage’s in 1860 with his mechanical computer to chime in machine language. As computing versatility up surged, software exigencies proliferated, thereupon siphoning the birth of FORTRAN language which expunged barricades between manifold hardware and the programming language. Computing versatility came in double edged, unquestionably overloading itself and urging for additions in its magnitude, this led to the birth of languages like BASIC, PASCAL, and C. Following the downright outstretch of software about every single niche of our turf, operating systems such as Microsoft made surface, managing data storages and of all users’ interruptions and treated them with a corresponding adequate and appropriate response. Here and now, software pilot financial management tools and systems, networking systems, browsers, the IoT, and even e-commerce platforms. The appearance of each of these software services has been closely related to the evolution of the needs of mankind and its hunger for improvement and novelty. In the following chapter, we will decorticate this process with relevant examples to understand the flow process in their happening.

Software Applications and Software Culture

As civilization plunged into software and eventually grew coated over generations, society inevitably ran to the “every body” can use age. At that stage of development, software drew closer to mankind. Industries called for software applications to handle their data, leading to an elephantine production of database management systems in the 1970s. Later in the 80s, following the advancement in hardware, software amplified its scope into a network of disciplines such agriculture where it made possible outcome prediction and the medical sector, where software applications monitored medical equipment. Meanwhile software applications engulfed the entertainment market with virtual and augmented reality, the Internet and Mobile applications furnished completeness of services. As of emerging industries involving Space, Scientific Exploration, New energy, Internet of Things and Unmanned Aerial Vehicle areas, software applications provided ways of accelerated explorations and analysis, together with advanced data transformation and interpretation. Currently as we are writing this book, a lot of software simulation and exploration is taking place in the software industry. They are carried out in an attempt to visualize future opportunities. As mankind constantly receives threads from its environment and nature, it has been endlessly seeking for the next station. From these examples of software application, readers will have a better understanding of software culture. This chapter discusses the relationship between software applications and software culture from two aspects: software applications in traditional and emerging industries.

Semantic Sequence Analysis for Human Activity Prediction

The task of long video analysis is challenging, and it is often the case that many human actions occur but only a few contribute to the semantic topic of the video. However, compared with short video human activity studies, long video analysis has its practical utility especially considering the effort of watching a long video for human. In this paper, we propose to learn semantic symbol sequence patterns of complex videos for activity prediction. The prefix method of semantic stream is designed based on the semantic symbol sequence and their time marks. The prediction phase is implemented via matching semantic sequence of incomplete videos and sequence patterns of different activities. We evaluate various prediction methods depending on low-level features or high-level descriptions. The empirical result suggests that when applied to activity prediction, sequence pattern mining can effectively reduce its reliance upon the low level features and improve predicting performance.

Semantic R-CNN for Natural Language Object Detection

In this paper, we present a simple and effective framework for natural language object detection, to localize a target within an image based on description of the target. The method, called semantic R-CNN, extends RPN (Region Proposal Network) [1] by adding LSTM [20] module for processing natural language query text. LSTM [20] module take encoded query text and image descriptors as input and output the probability of the query text conditioned on visual features of candidate box and whole image. Those candidate boxes are generated by RPN and their local features are extracted by ROI pooling. RPN can be initialized from pre-trained Faster R-CNN model [1], transfers object visual knowledge from traditional object detection domain to our task. Experimental results demonstrate that our method significantly outperform previous baseline SCRC (Spatial Context Recurrent ConvNet) [7] model on Referit dataset [8], moreover, our model is simple to train similar to Faster R-CNN.