Soon Ki Jung

Decomposition into low-rank plus additive matrices for background/foreground separation

Background/foreground separation is the first step in video surveillance system to detect moving objects. Recent research on problem formulations based on decomposition into low-rank plus sparse matrices shows a suitable framework to separate moving objects from the background. The most representative problem formulation is the Robust Principal Component Analysis (RPCA) solved via Principal Component Pursuit (PCP) which decomposes a data matrix into a low-rank matrix and a sparse matrix. However, similar robust implicit or explicit decompositions can be made in the following problem formulations: Robust Non-negative Matrix Factorization (RNMF), Robust Matrix Completion (RMC), Robust Subspace Recovery (RSR), Robust Subspace Tracking (RST) and Robust Low-Rank Minimization (RLRM). The main goal of these similar problem formulations is to obtain explicitly or implicitly a decomposition into low-rank matrix plus additive matrices. These formulation problems differ from the implicit or explicit decomposition, the loss function, the optimization problem and the solvers. As the problem formulation can be NP-hard in its original formulation, and it can be convex or not following the constraints and the loss functions used, the key challenges concern the design of efficient relaxed models and solvers which have to be with iterations as few as possible, and as efficient as possible. In the application of background/foreground separation, constraints inherent to the specificities of the background and the foreground as the temporal and spatial properties need to be taken into account in the design of the problem formulation. Practically, the background sequence is then modeled by a low-rank subspace that can gradually change over time, while the moving foreground objects constitute the correlated sparse outliers. Although, many efforts have been made to develop methods for the decomposition into low-rank plus additive matrices that perform visually well in foreground detection with reducing their computational cost, no algorithm today seems to emerge and to be able to simultaneously address all the key challenges that accompany real-world videos. This is due, in part, to the absence of a rigorous quantitative evaluation with synthetic and realistic large-scale dataset with accurate ground truth providing a balanced coverage of the range of challenges present in the real world. In this context, this work aims to initiate a rigorous and comprehensive review of the similar problem formulations in robust subspace learning and tracking based on decomposition into low-rank plus additive matrices for testing and ranking existing algorithms for background/foreground separation. For this, we first provide a preliminary review of the recent developments in the different problem formulations which allows us to define a unified view that we called Decomposition into Low-rank plus Additive Matrices (DLAM). Then, we examine carefully each method in each robust subspace learning/tracking frameworks with their decomposition, their loss functions, their optimization problem and their solvers. Furthermore, we investigate if incremental algorithms and real-time implementations can be achieved for background/foreground separation. Finally, experimental results on a large-scale dataset called Background Models Challenge (BMC 2012) show the comparative performance of 32 different robust subspace learning/tracking methods.

Automatic pose estimation of complex 3D building models

3D models of urban sites with geometry and facade textures are needed for many planning and visualization applications. Approximate 3D wireframe model can be derived from aerial images but detailed textures must be obtained from ground level images. Integrating such views with the 3D models is difficult as only small parts of buildings may be visible in a single view. We describe a method that uses two or three vanishing points, and three 3D to 2D line correspondences to estimate the rotational and translational parameters of the ground level cameras. The valid set of multiple combinations of 3D to 2D line pairs is chosen by a hypotheses generation and evaluation Some experimental results are presented.

Calibration-free approach to 3D reconstruction using light stripe projections on a cube frame

This paper presents a new approach based on light striping for reconstructing a 3D model from a real object. The proposed system consists of a light plane projector, camera and cube frame with LEDs attached. As in other light striping systems, the correspondence problem is solved by projecting light plane onto an object inside a frame. However, we use only cross-ratios and vanishing points to set up the world coordinates of the object, while the intrinsic and extrinsic parameters of the camera and the position of light source are not required. As such, the proposed system does not calibrate the camera and the light source. Furthermore, the computed 3D point data does not require by registration process because the data is directly measured based on unified world coordinates. Experimental results proved the accuracy of the measurements and consistency of the outcomes without any knowledge of the camera and light source parameters.

Integrating ground and aerial views for urban site modeling

3D models of urban sites with good geometry and facade textures are needed for many planning and visualization applications. Approximate wireframe can be derived from aerial images but detailed textures must be obtained from ground level images. Integrating such views with the 3D models is difficult as only small parts of buildings may be visible in a single view. We describe a method that uses two or three vanishing points, not necessarily from orthogonal sets of parallel lines, and a small number of point correspondences to estimate the intrinsic and extrinsic parameters of the ground level cameras. Experimental results from some buildings are presented.

A moving planar mirror based approach for cultural reconstruction

Modelling from images is a cost‐effective means of obtaining virtual cultural heritage models. These models can be effectively constructed from classical Structure from Motion algorithm. However, its too difficult to reconstruct whole scenes using SFM method since general oriental historic sites contain a very complex shapes and brilliant colours. To overcome this difficulty, the current paper proposes a new reconstruction method based on a moving planar mirror. We devise the mirror posture instead of scene itself as a cue for reconstructing the geometry. That implies that the geometric cues are inserted into the scene by compulsion. With this method, we can obtain the geometrical details regardless of the scene complexity. For this purpose, we first capture image sequences through the moving mirror containing the interested scene, and then calibrate the camera through the mirrors posture. Since the calibration results are still inaccurate due to the detection error, the camera pose is revised using frame‐correspondence of the corner points that are easily obtained using the initial camera posture. Finally, 3D information is computed from a set of calibrated image sequences. We validate our approach with a set of experiments on some cultural heritage objects. Copyright

OR-PCA with MRF for Robust Foreground Detection in Highly Dynamic Backgrounds

Accurate and efficient foreground detection is an important task in video surveillance system. The task becomes more critical when the background scene shows more variations, such as water surface, waving trees, varying illumination conditions, etc. Recently, Robust Principal Components Analysis (RPCA) shows a very nice framework for moving object detection. The background sequence is modeled by a low-dimensional subspace called low-rank matrix and sparse error constitutes the foreground objects. But RPCA presents the limitations of computational complexity and memory storage due to batch optimization methods, as a result it is difficult to apply for real-time system. To handle these challenges, this paper presents a robust foreground detection algorithm via Online Robust PCA (OR-PCA) using image decomposition along with continuous constraint such as Markov Random Field (MRF). OR-PCA with good initialization scheme using image decomposition approach improves the accuracy of foreground detection and the computation time as well. Moreover, solving MRF with graph-cuts exploits structural information using spatial neighborhood system and similarities to further improve the foreground segmentation in highly dynamic backgrounds. Experimental results on challenging datasets such as Wallflower, I2R, BMC 2012 and Change Detection 2014 dataset demonstrate that our proposed scheme significantly outperforms the state of the art approaches and works effectively on a wide range of complex background scenes.

Spatiotemporal Low-Rank Modeling for Complex Scene Background Initialization

Background modeling constitutes the building block of many computer-vision tasks. Traditional schemes model the background as a low rank matrix with corrupted entries. These schemes operate in batch mode and do not scale well with the data size. Moreover, without enforcing spatiotemporal information in the low-rank component, and because of occlusions by foreground objects and redundancy in video data, the design of a background initialization method robust against outliers is very challenging. To overcome these limitations, this paper presents a spatiotemporal low-rank modeling method on dynamic video clips for estimating the robust background model. The proposed method encodes spatiotemporal constraints by regularizing spectral graphs. Initially, a motion-compensated binary matrix is generated using optical flow information to remove redundant data and to create a set of dynamic frames from the input video sequence. Then two graphs are constructed, one between frames for temporal consistency and the other between features for spatial consistency, to encode the local structure for continuously promoting the intrinsic behavior of the low-rank model against outliers. These two terms are then incorporated in the iterative Matrix Completion framework for improved segmentation of background. Rigorous evaluation on severely occluded and dynamic background sequences demonstrates the superior performance of the proposed method over state-of-the-art approaches.

OR-PCA with dynamic feature selection for robust background subtraction

Background modeling and foreground object detection is the first step in visual surveillance system. The task becomes more difficult when the background scene contains significant variations, such as water surface, waving trees and sudden illumination conditions, etc. Recently, subspace learning model such as Robust Principal Component Analysis (RPCA) provides a very nice framework for separating the moving objects from the stationary scenes. However, due to its batch optimization process, high dimensional data should be processed. As a result, huge computational complexity and memory problems occur in traditional RPCA based approaches. In contrast, Online Robust PCA (OR-PCA) has the ability to process such large dimensional data via stochastic manners. OR-PCA processes one frame per time instance and updates the subspace basis accordingly when a new frame arrives. However, due to the lack of features, the sparse component of OR-PCA is not always ro-bust to handle various background modeling challenges. As a consequence, the system shows a very weak performance, which is not desirable for real applications. To handle these challenges, this paper presents a multi-feature based OR-PCA scheme. A multi-feature model is able to build a ro-bust low-rank background model of the scene. In addition, a very nice feature selection process is designed to dynamically select a useful set of features frame by frame, according to the weighted sum of total features. Experimental results on challenging datasets such as Wallflower, I2R and BMC 2012 show that the proposed scheme outperforms the state of the art approaches for the background subtraction task.

Background Subtraction via Superpixel-Based Online Matrix Decomposition with Structured Foreground Constraints

Background subtraction process plays a very essential role for various computer vision tasks. The process becomes more critical when the input scene contains variation of pixels such as swaying trees, rippling of water, illumination variations, etc. Recent methods of matrix decomposition into low-rank (e.g., corresponds to the background) and sparse (e.g., constitutes the moving objects) components such as Robust Principal Component Analysis (RPCA), have been shown to be very efficient framework for background subtraction. However, when the size of the input data grows and due to the lack of sparsity-constraints, these methods cannot cope with the real-time challenges and always show a weak performance due to the erroneous foreground regions. In order to address the above mentioned issues, this paper presents a superpixel-based matrix decomposition method together with maximum norm (max-norm) regularizations and structured sparsity constraints. The low-rank component estimated from each homogeneous region is more perfect, reliable, and efficient, since each superpixel provides different characteristics with a reduced value of rank. Online max-norm based matrix decomposition is employed on each segmented superpixel to separate the low rank and initial outliers support. And then, the structured sparsity constraints such as the generalized fussed lasso (GFL) are adopted for exploiting structural information continuously as the foreground pixels are both spatially connected and sparse. We propose an online single unified optimization framework for detecting foreground and learning the background model simultaneously. Rigorous experimental evaluations on challenging datasets demonstrate the superior performance of the proposed scheme in terms of both accuracy and computational time.

Robust background subtraction to global illumination changes via multiple features-based online robust principal components analysis with Markov random field

Abstract. Background subtraction is an important task for various computer vision applications. The task becomes more critical when the background scene contains more variations, such as swaying trees and abruptly changing lighting conditions. Recently, robust principal component analysis (RPCA) has been shown to be a very efficient framework for moving-object detection. However, due to its batch optimization process, high-dimensional data need to be processed. As a result, computational complexity, lack of features, weak performance, real-time processing, and memory issues arise in traditional RPCA-based approaches. To handle these, a background subtraction algorithm robust against global illumination changes via online robust PCA (OR-PCA) using multiple features together with continuous constraints, such as Markov random field (MRF), is presented. OR-PCA with automatic parameter estimation using multiple features improves the background subtraction accuracy and computation time, making it attractive for real-time systems. Moreover, the application of MRF to the foreground mask exploits structural information to improve the segmentation results. In addition, global illumination changes in scenes are tackled by using sum of the difference of similarity measure among features, followed by a parameter update process using a low-rank, multiple features model. Evaluation using challenging datasets demonstrated that the proposed scheme is a top performer for a wide range of complex background scenes.