Håkan Ardö

A Hardware Architecture for Real-Time Video Segmentation Utilizing Memory Reduction Techniques

This paper presents the implementation of a video segmentation unit used for embedded automated video surveillance systems. Various aspects of the underlying segmentation algorithm are explored and modifications are made with potential improvements of segmentation results and hardware efficiency. In addition, to achieve real-time performance with high resolution video streams, a dedicated hardware architecture with streamlined dataflow and memory access reduction schemes are developed. The whole system is implemented on a Xilinx field-programmable gate array platform, capable of real-time segmentation with VGA resolution at 25 frames per second. Substantial memory bandwidth reduction of more than 70% is achieved by utilizing pixel locality as well as wordlength reduction. The hardware platform is intended as a real-time testbench, especially for observations of long term effects with different parameter settings.

Hardware accelerator design for video segmentation with multi-modal background modelling

Among many of the algorithms for video segmentation, one based on a statistical background model (Stauffer, C. and Grimson, W., Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1999) was developed with the unique feature of robustness in multi-modal background scenarios. However, with a large number of calculations due to the pixel-wise processing of each frame, such an algorithm could only achieve a low frame rate, far from real-time requirements, on computers. A hardware accelerator is proposed, with a dedicated architecture aimed at addressing both computation and memory bandwidth demands. The whole system is targeted to an FPGA platform, which serves as a real-time test bench where long term effects caused by fixed point quantization and various parameter settings can be studied. Meanwhile, memory bandwidth as well as memory size are investigated, and reduction by up to 60 percent, through similarity exploitation for neighboring Gaussian parameters, is envisioned. Furthermore, a controller synthesis tool is used to relieve the effort for the manual design of the complex control unit which schedules the operations of the whole system.

Towards safe human-robot interaction in robotic cells: An approach based on visual tracking and intention estimation

Removing the safety fences that separate humans and robots, to allow for an effective human-robot interaction, requires innovative safety control systems. An advanced functionality of a safety controller might be to detect the presence of humans entering the robotic cell and to estimate their intention, in order to enforce an effective safety reaction. This paper proposes advanced algorithms for cognitive vision, empowered by a dynamic model of human walking, for detection and tracking of humans. Intention estimation is then addressed as the problem of predicting online the trajectory of the human, given a set of trajectories of walking people learnt offline using an unsupervised classification algorithm. Results of the application of the presented approach to a large number of experiments on volunteers are also reported.

Real Time Viterbi Optimization of Hidden Markov Models for Multi Target Tracking

In this paper the problem of tracking multiple objects in im- age sequences is studied. A Hidden Markov Model describ- ing the movements of multiple objects is presented. Previ- ously similar models have been used, but in real time sys- tem the standard dynamic programming Viterbi algorithm is typically not used to find the global optimum state se- quence, as it requires that all past and future observations are available. In this paper we present an extension to the Viterbi algorithm that allows it to operate on infinite time sequences and produce the optimum with only a finite de- lay. This makes it possible to use the Viterbi algorithm in real time applications. Also, to handle the large state spaces of these models another extension is proposed. The global optimum is found by iteratively running an approximative algorithm with higher and higher precision. The algorithm can determine when the global optimum is found by main- taining an upper bound on all state sequences not evalu- ated. For real time performance some approximations are needed and two such approximations are suggested. The theory has been tested on three real data experiments, all with promising results.

Development of automatic surveillance of animal behaviour and welfare using image analysis and machine learned segmentation technique.

In this paper the feasibility to extract the proportion of pigs located in different areas of a pig pen by advanced image analysis technique is explored and discussed for possible applications. For example, pigs generally locate themselves in the wet dunging area at high ambient temperatures in order to avoid heat stress, as wetting the body surface is the major path to dissipate the heat by evaporation. Thus, the portion of pigs in the dunging area and resting area, respectively, could be used as an indicator of failure of controlling the climate in the pig environment as pigs are not supposed to rest in the dunging area. The computer vision methodology utilizes a learning based segmentation approach using several features extracted from the image. The learning based approach applied is based on extended state-of-the-art features in combination with a structured prediction framework based on a logistic regression solver using elastic net regularization. In addition, the method is able to produce a probability per pixel rather than form a hard decision. This overcomes some of the limitations found in a setup using grey-scale information only. The pig pen is a difficult imaging environment because of challenging lighting conditions like shadows, poor lighting and poor contrast between pig and background. In order to test practical conditions, a pen containing nine young pigs was filmed from a top view perspective by an Axis M3006 camera with a resolution of 640 × 480 in three, 10-min sessions under different lighting conditions. The results indicate that a learning based method improves, in comparison with greyscale methods, the possibility to reliable identify proportions of pigs in different areas of the pen. Pigs with a changed behaviour (location) in the pen may indicate changed climate conditions. Changed individual behaviour may also indicate inferior health or acute illness.

Real-Time Video Segmentation with VGA Resolution and Memory Bandwidth Reduction

This paper presents the implementation of a video segmentation unit used for embedded automated video surveillance systems. Various aspects of the underlying segmentation algorithm are explored and modifications are made with potential improvements of segmentation results and hardware efficiency. In addition, to achieve real-time performance with high resolution video streams, a dedicated hardware architecture with streamlined dataflow and memory access reduction schemes are developed. The whole system is implemented on a Xilinx FPGA platform, capable of real-time segmentation with VGA resolution at 25 frames per second. Substantial memory bandwidth reduction of more than 70% is achieved by utilizing pixel locality as well as wordlenghth reduction. The hardware platform is intended as a real-time testbench for observations of long term effects with different parameter settings, which is hard to achieve on a PC platform.

Bayesian formulation of image patch matching using cross-correlation

A classical solution for matching two image patches is to use the cross-correlation coefficient. This works well if there is a lot of structure within the patches, but not so well if the patches are close to uniform. This means that some patches are matched with more confidence than others. By estimating this uncertainty more weight can be put on the confident matches than those that are more uncertain. In this paper we present a system that can learn the distribution of the correlation coefficient from a video sequence of an empty scene. No manual annotation of the video is needed. Two distributions functions are learned for two different cases: i) the correlation between an estimated background image and the current frame showing that background and ii) the correlation between an estimated background image and an unrelated patch.

Loop-aware optimizations in PyPy's tracing JIT

One of the nice properties of a tracing just-in-time compiler (JIT) is that many of its optimizations are simple, requiring one forward pass only. This is not true for loop-invariant code motion which is a very important optimization for code with tight kernels. Especially for dynamic languages that typically perform quite a lot of loop invariant type checking, boxed value unwrapping and virtual method lookups. In this paper we explain a scheme pioneered within the context of the LuaJIT project for making basic optimizations loop-aware by using a simple pre-processing step on the trace without changing the optimizations themselves. We have implemented the scheme in RPythons tracing JIT compiler. PyPys Python JIT executing simple numerical kernels can become up to two times faster, bringing the performance into the ballpark of static language compilers.

Feasibility study for the implementation of an automatic system for the detection of social interactions in the waiting area of automatic milking stations by using a video surveillance system

Monitoring of social interactions by use of image segmentation and tracking methods.Three cameras with top-down view were used for recordings and observations.The social interactions were identified based on collision of geometrical shapes.The overall performance of the detector in its early stage is 85.1%. A well-planned waiting area is crucial for automatic milking systems. In an enclosed waiting area, cows of different rank compete for entering the milking station and they are exposed for a variety of social interactions. Such interactions could increase standing time and delay milking, which may result in stress, lameness, impaired welfare and reduced performance. The aim was to monitor the waiting area in a free stall dairy by the use of three video cameras to detect occurrence of social interactions by using improved image segmentation and tracking methods. The surveillance system observed 252 cows having free access to any of four milking stations during 24h over a period of two weeks. A two-step pattern recognition approach was used. In the first step geometric features (distances) were extracted from every pair of cows in every frame. These features form the input of the second step. It consists of a classifier of the behaviour of the cows. A support vector machine was used to realise this classifier. The social interactions were identified based on collision of geometrical shapes segmented from the image and positively identified as cows by experienced observers. The results showed that the proposed system was capable of a fairly accurate detection of social interactions.

Convolutional neural network-based cow interaction watchdog

Animal behaviour and welfare can be studied/assessed by looking at different interactions occurring between the animals. Video recordings of a scene of interest are often made and then watched/evaluated by experts. However, the interactions of interest are often fairly rare. To reduce the amount of time the experts spend on watching the uninteresting video, this paper introduces an automated watchdog system that can discard some of the recorded video material. A pilot study on cows was made where a Convolutional Neural Network (CNN) detector was used to count the number of cows in the scene and discard video where less than two cows were present. This removed 38 % of the recordings while only losing 1 % of the interesting video.In the field of applied animal behaviour, video recordings of a scene of interest are often made and then evaluated by experts. This evaluation is based on different criteria (number of animals present, an occurrence of certain interactions, the proximity between animals and so forth) and aims to filter out video sequences that contain irrelevant information. However, such task requires a tremendous amount of time and resources, making manual approach ineffective. To reduce the amount of time the experts spend on watching the uninteresting video, this study introduces an automated watchdog system that can discard some of the recorded video material based on user-defined criteria. A pilot study on cows was made where a convolutional neural network detector was used to detect and count the number of cows in the scene as well as include distances and interactions between cows as filtering criteria. This approach removed 38% (50% for additional filter parameters) of the recordings while only losing 1% (4%) of the potentially interesting video frames.