Mirela C. Popa

Kinect sensing of shopping related actions

Surveillance systems in shopping malls or supermarkets are usually used for detecting abnormal behavior. We used the distributed video cameras system to design digital shopping assistants which assess the behavior of customers while shopping, detect when they need assistance, and offer their support in case there is a selling opportunity. In this paper we propose a system for analyzing human behavior patterns related to products interaction, such as browse through a set of products, examine, pick products, try on, interact with the shopping cart, and look for support by waiving one hand. We used the Kinect sensor to detect the silhouettes of people and extracted discriminative features for basic action detection. Next we analyzed different classification methods, statistical and also spatio-temporal ones, which capture relations between frames, features, and basic actions. By employing feature level fusion of appearance and movement information we obtained an accuracy of 80% for the mentioned six basic actions.

Semantic assessment of shopping behavior using trajectories, shopping related actions, and context information

Automatic understanding of customers shopping behavior and acting according to their needs is relevant in the marketing domain and is attracting a lot of attention lately. In this work, we propose a multi-level framework for the automatic assessment of customers shopping behavior. The low level input to the framework is obtained from different types of cameras, which are synchronized, facilitating efficient processing of information. A fish-eye camera is used for tracking people, while a high-definition one serves for the action recognition task. The experiments are performed on both laboratory and real-life recordings in a supermarket. From the video recordings, we extract features related to the spatio-temporal behavior of trajectories, the dynamics and the time spent in each region of interest (ROI) in the shop and regarding the customer-products interaction patterns. Next we analyze the shopping sequences using a Hidden Markov Model (HMM). We conclude that it is possible to accurately classify trajectories (93%), discriminate between different shopping related actions (91.6%), and recognize shopping behavioral types by means of our proposed reasoning model in 95% of the cases.

Analysis of shopping behavior based on surveillance system

Closed Circuit Television systems in shopping malls could be used to monitor the shopping behavior of people. From the tracked path, features can be extracted such as the relation with the shopping area, the orientation of the head, speed of walking and direction, pauses which are supposed to be related to the interest of the shopper. Once the interest has been detected the next step is to assess the shoppers positive or negative appreciation to the focused products by analyzing the (non-verbal) behavior of the shopper. Ultimately the system goal is to assess the opportunities for selling, by detecting if a customer needs support. In this paper we present our methodology towards developing such a system consisting of participating observation, designing shopping behavioral models, assessing the associated features and analyzing the underlying technology. In order to validate our observations we made recordings in our shop lab. Next we describe the used tracking technology and the results from experiments.

Detecting customers' buying events on a real-life database

Video Analytics covers a large set of methodologies which aim at automatically extracting information from video material. In the context of retail, the possibility to effortlessly gather statistics on customer shopping behavior is very attractive. In this work, we focus on the task of automatic classification of customer behavior, with the objecting to recognize buying events. The experiments are performed on several hours of video collected in a supermarket. Given the vast effort of the research community on the task of tracking, we assume the existence of a video tracking system capable of producing a trajectory for every individual, and currently manually annotate the input videos with trajectories. From the annotated video recordings, we extract features related to the spatio-temporal behavior of the trajectory, and to the user movement, and analyze the shopping sequences using a Hidden Markov Model (HMM). First results show that it is possible to discriminate between buying and non-buying behavior with an accuracy of 74%.

Products appreciation by facial expressions analysis

Automatic assessment of users appreciation of products represents an important functionality for shops, leading to better fitted products on the customers needs and enabling more efficient marketing strategies. By means of a surveillance system we track customers, make a first interpretation of their behaviour and analyze their facial expressions when they are next to a product. Facial expressions carry relevant information regarding customers opinion of products and can be used to detect if they show interest and also which type (positive or negative). The main contribution of this work resides in the development of a facial expression recognition analyzer that can be used in the product appreciation domain. In our approach we employ the Active Appearance Model to extract the key facial regions (e.g. eyes, nose, and mouth). Around these special regions we define Regions of Interest and extract relevant features using the optical flow estimation method. The classification phase is carried out using Hidden Markov Models. Experiments are conducted on the well-known Cohn-Kanade database and also on our own recorded database of 21 product emotions to show the efficacy of our approach. An average recognition accuracy of 93% is achieved.

Assessment of customers' level of interest

Surveillance systems in shopping malls or supermarkets are usually designated for assuring safety and detecting abnormal behavior. We used the distributed video cameras system to design digital shopping assistants which assess the behavior of customers while shopping, detect when they need assistance, and offer their support in case there is a selling opportunity. In this paper we propose a system for analyzing human behavior patterns related to products interaction, which could reveal the customers level of interest. We extracted discriminative features for basic action detection and analyzed different statistical and spatio-temporal classification methods, which capture relations between frames, features, and basic actions. Our experiments show that it is possible to accurately recognize different shopping related actions (85.7%) and discriminate between the proposed levels of interest in (88%) of the cases.

Shopping behavior recognition using a language modeling analogy

Automatic understanding and recognition of human shopping behavior has many potential applications, attracting an increasing interest inthe market- ing domain. The reliability and performance of the automatic recognition system is highly in uenced by the adopted theoretical model of behavior. In this work, we address the analogy betweenhuman shopping behavior and a natural language. The adopted methodology associates low-level informa- tion extracted from video data with semantic information using the proposed behavior language model. Our contribution on the action recognition level consists of proposing a new feature set which fuses Histograms of Optical Flow (HOF) with directional features. On the behavior level we propose combining smoothed bi-grams with the maximum dependency in a chain of conditional probabilities. The experiments are performed on both laboratory and real-life datasets. The introduced behavior language model achieves an accuracy of 87% on the laboratory data and 76% on the real-life dataset, an improvement of 11% and 8% respectively over the baseline model, by incor- porating semantic knowledge and capturing correlations between the basic actions.

AN INTELLIGENT CO-DRIVER SURVEILLANCE SYSTEM

In recent years many car manufacturers developed digital co-drivers , which are able to monitor the driving behaviour of a car. Sensors in the car measure if a car passes speed limits, leaves its lane, or violates other traffic rules. A new generation of co-drivers is based on sensors in the car which are able to monitor the driver behaviour. Driving a car is a sequence of actions. In case a driver doesn’t show one of the actions the co-driver generates a warning signal. Experiments in the car simulator TORC were performed to extract the actions of a car driver. These actions were used to develop probabilistic models of the driving behaviour. A prototype of a warning system has been developed and tested in the car simulator. The experiments and test results will be reported in this paper.