Zhenke Yang

Analysis of shopping behavior based on surveillance system

Closed Circuit Television systems in shopping malls could be used to monitor the shopping behavior of people. From the tracked path, features can be extracted such as the relation with the shopping area, the orientation of the head, speed of walking and direction, pauses which are supposed to be related to the interest of the shopper. Once the interest has been detected the next step is to assess the shoppers positive or negative appreciation to the focused products by analyzing the (non-verbal) behavior of the shopper. Ultimately the system goal is to assess the opportunities for selling, by detecting if a customer needs support. In this paper we present our methodology towards developing such a system consisting of participating observation, designing shopping behavioral models, assessing the associated features and analyzing the underlying technology. In order to validate our observations we made recordings in our shop lab. Next we describe the used tracking technology and the results from experiments.

Multimodal workbench for automatic surveillance applications

Noticeable developments have lately been achieved on designing automated multimodal smart processes to increase security in every-day life of people. As these developments continue, proper infrastructures and methodologies for the aggregation of various demands that will inevitably arise, such as the huge amount of data and computation, become more important. In this research, we introduce a multimodal framework with support for an automatic surveillance application. The novelty of the attempt resides in the modalities to underpin data manipulation as a natural process but still keeping the overall performance at high levels. At the application level, the typical complexity behind the emerging distributed multimodal systems is reduced in a transparent manner through multimodal frameworks that handle data on different abstraction levels and efficiently accommodate constituent technologies. The proposed specifications includes the use of shared memory spaces (XML data spaces) and smart document-centered content-based data querying mechanisms (XQuery formal language (S. Bong et al., 2006)). We also report on the use of this framework in an application on aggression detection in train compartments.

Automatic aggression detection inside trains

This paper presents work addressing the challenges of video analysis for automatic detection of aggression in a train. Using data from surveillance cameras, the system assists human operators in their work. It is unobtrusive and respects the privacy of passengers. We used existing algorithms to recognize and classify human behavior. While evaluating the algorithms we paid special attention to their ability to cope with environment specific issues, such as varying lighting conditions and (self)occlusions. A passenger behavior model was developed based on many hours of observing and studying professional operators as they analyze and respond to surveillance data. Experiments were conducted in a real train to evaluate the detection system.

Surveillance system using abandoned object detection

Recent increased adaptation of security cameras to survey public areas has caused a digital data explosion that human operators have difficulty to keep up with. In order to support future surveillance systems, this paper describes a recognition and tracking system for abandoned objects. The goal is to design and implement a system which will be able to detect abandoned luggage using video sequences. The system uses high level reasoning to infer the existence of abandoned luggage. The proposed approach was implemented using existing image segmentation, blob creation, labelling and tracking algorithms. Our experiments demonstrate the effectiveness of the approach.

Addressing multimodality in overt aggression detection

Automatic detection of aggressive situations has a high societal and scientific relevance. It has been argued that using data from multimodal sensors as for example video and sound as opposed to unimodal is bound to increase the accuracy of detections. We approach the problem of multimodal aggression detection from the viewpoint of a human observer and try to reproduce his predictions automatically. Typically, a single ground truth for all available modalities is used when training recognizers. We explore the benefits of adding an extra level of annotations, namely audio-only and video-only. We analyze these annotations and compare them to the multimodal case in order to have more insight into how humans reason using multimodal data. We train classifiers and compare the results when using unimodal and multimodal labels as ground truth. Both in the case of audio and video recognizer the performance increases when using the unimodal labels.

A system for adaptive multimodal interaction in crisis environments

In the recent years multimodal interfaces have acquired an important role in human computer interaction applications. Subsequently these interfaces become more and more human-oriented. Humans use multimodality to reduce ambiguity and incompleteness of information. Seemingly they are able to switch easily from one modality to the other and fuse the information from different multimodal sources. The goal of our research was to develop a crisis based human like multimodal system. In particular, we bring into focus the multimodal interaction between human users and the automatic crisis system and its correlation with the adaptability to the human behavior in crisis situations. Our system is capable of conceding for an optimal interaction process by taking into account the major informational human channels while gathering the user inputs and producing the system feedback. In this paper we describe the design of our system which is implemented as a running prototype. We have conducted a simulation of a crisis event to measure the degree of user satisfaction. At last we discuss the drawbacks as well as the premises of our solution in the context of the high level of performance achieved by our approach.

A bottom-up approach of fusion of events in surveillance systems

Human observers are able to fuse events and to integrate them in a unifying theory. An incoming stream of events triggers a hypothesis in an associative way. On the other hand, most automated classification systems require a full list of hypotheses with a specified list of events. An incoming event increases the probability of an associated hypothesis. In this paper we introduce a system which emulates the emergent process of hypothesis generation from human observers. Basically it is a bottom up approach of fusion of events. The starting point is a matrix of correlation coefficients between pairs of events. The system builds up a network of linked events. The largest network of highly salient events is the prevailing hypothesis at a given moment. In this way the system is able to generate hypothesis not defined at start. We describe the design of the proposed system and results of testing it in a surveillance environment of aggression detection.

Context-Aware Multimodal Human–Computer Interaction

Crisis response and management involve the collaboration of many people. To perform and coordinate their activities, they must rely on detailed and accurate information about the crisis, the environment, and many more factors. To ensure collaboration of emergency services and high-quality care for victims, the ability to supply dynamic and contextually correlated information is necessary. However, current approaches to construct globally consistent views of crises suffer from problems identified in [60]: (a) the setting of events is constantly changing, (b) the information is distributed across geographically distant locations, and (c) the complexity of the crisis management organization makes it difficult and time consuming to collaborate and verify obtained information.