Sajid Javed

Decomposition into low-rank plus additive matrices for background/foreground separation

Background/foreground separation is the first step in video surveillance system to detect moving objects. Recent research on problem formulations based on decomposition into low-rank plus sparse matrices shows a suitable framework to separate moving objects from the background. The most representative problem formulation is the Robust Principal Component Analysis (RPCA) solved via Principal Component Pursuit (PCP) which decomposes a data matrix into a low-rank matrix and a sparse matrix. However, similar robust implicit or explicit decompositions can be made in the following problem formulations: Robust Non-negative Matrix Factorization (RNMF), Robust Matrix Completion (RMC), Robust Subspace Recovery (RSR), Robust Subspace Tracking (RST) and Robust Low-Rank Minimization (RLRM). The main goal of these similar problem formulations is to obtain explicitly or implicitly a decomposition into low-rank matrix plus additive matrices. These formulation problems differ from the implicit or explicit decomposition, the loss function, the optimization problem and the solvers. As the problem formulation can be NP-hard in its original formulation, and it can be convex or not following the constraints and the loss functions used, the key challenges concern the design of efficient relaxed models and solvers which have to be with iterations as few as possible, and as efficient as possible. In the application of background/foreground separation, constraints inherent to the specificities of the background and the foreground as the temporal and spatial properties need to be taken into account in the design of the problem formulation. Practically, the background sequence is then modeled by a low-rank subspace that can gradually change over time, while the moving foreground objects constitute the correlated sparse outliers. Although, many efforts have been made to develop methods for the decomposition into low-rank plus additive matrices that perform visually well in foreground detection with reducing their computational cost, no algorithm today seems to emerge and to be able to simultaneously address all the key challenges that accompany real-world videos. This is due, in part, to the absence of a rigorous quantitative evaluation with synthetic and realistic large-scale dataset with accurate ground truth providing a balanced coverage of the range of challenges present in the real world. In this context, this work aims to initiate a rigorous and comprehensive review of the similar problem formulations in robust subspace learning and tracking based on decomposition into low-rank plus additive matrices for testing and ranking existing algorithms for background/foreground separation. For this, we first provide a preliminary review of the recent developments in the different problem formulations which allows us to define a unified view that we called Decomposition into Low-rank plus Additive Matrices (DLAM). Then, we examine carefully each method in each robust subspace learning/tracking frameworks with their decomposition, their loss functions, their optimization problem and their solvers. Furthermore, we investigate if incremental algorithms and real-time implementations can be achieved for background/foreground separation. Finally, experimental results on a large-scale dataset called Background Models Challenge (BMC 2012) show the comparative performance of 32 different robust subspace learning/tracking methods.

OR-PCA with MRF for Robust Foreground Detection in Highly Dynamic Backgrounds

Accurate and efficient foreground detection is an important task in video surveillance system. The task becomes more critical when the background scene shows more variations, such as water surface, waving trees, varying illumination conditions, etc. Recently, Robust Principal Components Analysis (RPCA) shows a very nice framework for moving object detection. The background sequence is modeled by a low-dimensional subspace called low-rank matrix and sparse error constitutes the foreground objects. But RPCA presents the limitations of computational complexity and memory storage due to batch optimization methods, as a result it is difficult to apply for real-time system. To handle these challenges, this paper presents a robust foreground detection algorithm via Online Robust PCA (OR-PCA) using image decomposition along with continuous constraint such as Markov Random Field (MRF). OR-PCA with good initialization scheme using image decomposition approach improves the accuracy of foreground detection and the computation time as well. Moreover, solving MRF with graph-cuts exploits structural information using spatial neighborhood system and similarities to further improve the foreground segmentation in highly dynamic backgrounds. Experimental results on challenging datasets such as Wallflower, I2R, BMC 2012 and Change Detection 2014 dataset demonstrate that our proposed scheme significantly outperforms the state of the art approaches and works effectively on a wide range of complex background scenes.

Spatiotemporal Low-Rank Modeling for Complex Scene Background Initialization

Background modeling constitutes the building block of many computer-vision tasks. Traditional schemes model the background as a low rank matrix with corrupted entries. These schemes operate in batch mode and do not scale well with the data size. Moreover, without enforcing spatiotemporal information in the low-rank component, and because of occlusions by foreground objects and redundancy in video data, the design of a background initialization method robust against outliers is very challenging. To overcome these limitations, this paper presents a spatiotemporal low-rank modeling method on dynamic video clips for estimating the robust background model. The proposed method encodes spatiotemporal constraints by regularizing spectral graphs. Initially, a motion-compensated binary matrix is generated using optical flow information to remove redundant data and to create a set of dynamic frames from the input video sequence. Then two graphs are constructed, one between frames for temporal consistency and the other between features for spatial consistency, to encode the local structure for continuously promoting the intrinsic behavior of the low-rank model against outliers. These two terms are then incorporated in the iterative Matrix Completion framework for improved segmentation of background. Rigorous evaluation on severely occluded and dynamic background sequences demonstrates the superior performance of the proposed method over state-of-the-art approaches.

OR-PCA with dynamic feature selection for robust background subtraction

Background modeling and foreground object detection is the first step in visual surveillance system. The task becomes more difficult when the background scene contains significant variations, such as water surface, waving trees and sudden illumination conditions, etc. Recently, subspace learning model such as Robust Principal Component Analysis (RPCA) provides a very nice framework for separating the moving objects from the stationary scenes. However, due to its batch optimization process, high dimensional data should be processed. As a result, huge computational complexity and memory problems occur in traditional RPCA based approaches. In contrast, Online Robust PCA (OR-PCA) has the ability to process such large dimensional data via stochastic manners. OR-PCA processes one frame per time instance and updates the subspace basis accordingly when a new frame arrives. However, due to the lack of features, the sparse component of OR-PCA is not always ro-bust to handle various background modeling challenges. As a consequence, the system shows a very weak performance, which is not desirable for real applications. To handle these challenges, this paper presents a multi-feature based OR-PCA scheme. A multi-feature model is able to build a ro-bust low-rank background model of the scene. In addition, a very nice feature selection process is designed to dynamically select a useful set of features frame by frame, according to the weighted sum of total features. Experimental results on challenging datasets such as Wallflower, I2R and BMC 2012 show that the proposed scheme outperforms the state of the art approaches for the background subtraction task.

Background Subtraction via Superpixel-Based Online Matrix Decomposition with Structured Foreground Constraints

Background subtraction process plays a very essential role for various computer vision tasks. The process becomes more critical when the input scene contains variation of pixels such as swaying trees, rippling of water, illumination variations, etc. Recent methods of matrix decomposition into low-rank (e.g., corresponds to the background) and sparse (e.g., constitutes the moving objects) components such as Robust Principal Component Analysis (RPCA), have been shown to be very efficient framework for background subtraction. However, when the size of the input data grows and due to the lack of sparsity-constraints, these methods cannot cope with the real-time challenges and always show a weak performance due to the erroneous foreground regions. In order to address the above mentioned issues, this paper presents a superpixel-based matrix decomposition method together with maximum norm (max-norm) regularizations and structured sparsity constraints. The low-rank component estimated from each homogeneous region is more perfect, reliable, and efficient, since each superpixel provides different characteristics with a reduced value of rank. Online max-norm based matrix decomposition is employed on each segmented superpixel to separate the low rank and initial outliers support. And then, the structured sparsity constraints such as the generalized fussed lasso (GFL) are adopted for exploiting structural information continuously as the foreground pixels are both spatially connected and sparse. We propose an online single unified optimization framework for detecting foreground and learning the background model simultaneously. Rigorous experimental evaluations on challenging datasets demonstrate the superior performance of the proposed scheme in terms of both accuracy and computational time.

Robust background subtraction to global illumination changes via multiple features-based online robust principal components analysis with Markov random field

Abstract. Background subtraction is an important task for various computer vision applications. The task becomes more critical when the background scene contains more variations, such as swaying trees and abruptly changing lighting conditions. Recently, robust principal component analysis (RPCA) has been shown to be a very efficient framework for moving-object detection. However, due to its batch optimization process, high-dimensional data need to be processed. As a result, computational complexity, lack of features, weak performance, real-time processing, and memory issues arise in traditional RPCA-based approaches. To handle these, a background subtraction algorithm robust against global illumination changes via online robust PCA (OR-PCA) using multiple features together with continuous constraints, such as Markov random field (MRF), is presented. OR-PCA with automatic parameter estimation using multiple features improves the background subtraction accuracy and computation time, making it attractive for real-time systems. Moreover, the application of MRF to the foreground mask exploits structural information to improve the segmentation results. In addition, global illumination changes in scenes are tackled by using sum of the difference of similarity measure among features, followed by a parameter update process using a low-rank, multiple features model. Evaluation using challenging datasets demonstrated that the proposed scheme is a top performer for a wide range of complex background scenes.

Background–Foreground Modeling Based on Spatiotemporal Sparse Subspace Clustering

Background estimation and foreground segmentation are important steps in many high-level vision tasks. Many existing methods estimate background as a low-rank component and foreground as a sparse matrix without incorporating the structural information. Therefore, these algorithms exhibit degraded performance in the presence of dynamic backgrounds, photometric variations, jitter, shadows, and large occlusions. We observe that these backgrounds often span multiple manifolds. Therefore, constraints that ensure continuity on those manifolds will result in better background estimation. Hence, we propose to incorporate the spatial and temporal sparse subspace clustering into the robust principal component analysis (RPCA) framework. To that end, we compute a spatial and temporal graph for a given sequence using motion-aware correlation coefficient. The information captured by both graphs is utilized by estimating the proximity matrices using both the normalized Euclidean and geodesic distances. The low-rank component must be able to efficiently partition the spatiotemporal graphs using these Laplacian matrices. Embedded with the RPCA objective function, these Laplacian matrices constrain the background model to be spatially and temporally consistent, both on linear and nonlinear manifolds. The solution of the proposed objective function is computed by using the linearized alternating direction method with adaptive penalty optimization scheme. Experiments are performed on challenging sequences from five publicly available datasets and are compared with the 23 existing state-of-the-art methods. The results demonstrate excellent performance of the proposed algorithm for both the background estimation and foreground segmentation.

Motion-Aware Graph Regularized RPCA for background modeling of complex scenes

Computing a background model from a given sequence of video frames is a prerequisite for many computer vision applications. Recently, this problem has been posed as learning a low-dimensional subspace from high dimensional data. Many contemporary subspace segmentation methods have been proposed to overcome the limitations of the methods developed for simple background scenes. Unfortunately, because of the absence of motion information and without preserving intrinsic geometric structure of video data, most existing algorithms do not provide promising nature of the low-rank component for complex scenes. Such as largely occluded background by foreground objects, superfluity in video frames in order to cope with intermittent motion of foreground objects, sudden lighting condition variation, and camera jitter sequences. To overcome these difficulties, we propose a motion-aware regularization of graphs on low-rank component for video background modeling. We compute optical flow and use this information to make a motion-aware matrix. In order to learn the locality and similarity information within a video we compute inter-frame and intra-frame graphs which we use to preserve geometric information in the low-rank component. Finally, we use linearized alternating direction method with parallel splitting and adaptive penalty to incorporate the preceding steps to recover the model of the background. Experimental evaluations on challenging sequences demonstrate promising results over state-of-the-art methods.

Online Stochastic Tensor Decomposition for Background Subtraction in Multispectral Video Sequences

Background subtraction is an important task for visual surveillance systems. However, this task becomes more complex when the data size grows since the real-world scenario requires larger data to be processed in a more efficient way, and in some cases, in a continuous manner. Until now, most of background subtraction algorithms were designed for mono or trichromatic cameras within the visible spectrum or near infrared part. Recent advances in multispectral imaging technologies give the possibility to record multispectral videos for video surveillance applications. Due to the specific nature of these data, many of the bands within multispectral images are often strongly correlated. In addition, processing multispectral images with hundreds of bands can be computationally burdensome. In order to address these major difficulties of multispectral imaging for video surveillance, this paper propose an online stochastic framework for tensor decomposition of multispectral video sequences (OSTD). First, the experimental evaluations on synthetic generated data show the robustness of the OSTD with other state of the art approaches then, we apply the same idea on seven multispectral video bands to show that only RGB features are not sufficient to tackle color saturation, illumination variations and shadows problem, but the addition of six visible spectral bands together with one near infrared spectra provides a better background/foreground separation.

Depth extended online RPCA with spatiotemporal constraints for robust background subtraction

The detection of moving objects is the first step in video surveillance systems. But due to the challenging backgrounds such as illumination conditions, color saturation, and shadows, etc., the state of the art methods do not provide accurate segmentation using only a single camera. Recently, subspace learning model such as Robust Principal Component analysis (RPCA) shows a very nice framework towards object detection. But, RPCA presents the limitations of computational and memory issues due to the batch optimization methods, and hence it cannot process high dimensional data. Recent research on RPCA methods such as Online RPCA (OR-PCA) alleviates the traditional RPCA limitations. However, OR-PCA using only color or intensity features shows a weak performance specially when the background and foreground objects have a similar color or shadows appear in the background scene. To handle these challenges, this paper presents an extension of OR-PCA with the integration of depth and color information for robust background subtraction. Depth is less affected by shadows or background/foreground color saturation issues. However, the foreground object may not be detected when it is far from the camera field as depth is less useful without color information. We show that the OR-PCA including spatiotemporal constraints provides accurate segmentation with the utilization of both color and depth features. Experimental evaluations on a well-defined benchmark dataset with other methods demonstrate that our proposed technique is a top performer using color and range information.