Edgar A. Bernal

Estimating Gaze Direction of Vehicle Drivers Using a Smartphone Camera

Many automated driver monitoring technologies have been proposed to enhance vehicle and road safety. Most existing solutions involve the use of specialized embedded hardware, primarily in high-end automobiles. This paper explores driver assistance methods that can be implemented on mobile devices such as a consumer smartphone, thus offering a level of safety enhancement that is more widely accessible. Specifically, the paper focuses on estimating driver gaze direction as an indicator of driver attention. Input video frames from a smartphone camera facing the driver are first processed through a coarse head pose direction. Next, the locations and scales of face parts, namely mouth, eyes, and nose, define a feature descriptor that is supplied to an SVM gaze classifier which outputs one of 8 common driver gaze directions. A key novel aspect is an in-situ approach for gathering training data that improves generalization performance across drivers, vehicles, smartphones, and capture geometry. Experimental results show that a high accuracy of gaze direction estimation is achieved for four scenarios with different drivers, vehicles, smartphones and camera locations.

Video-based real-time on-street parking occupancy detection system

Abstract. Urban parking management is receiving significant attention due to its potential to reduce traffic congestion, fuel consumption, and emissions. Real-time parking occupancy detection is a critical component of on-street parking management systems, where occupancy information is relayed to drivers via smart phone apps, radio, Internet, on-road signs, or global positioning system auxiliary signals. Video-based parking occupancy detection systems can provide a cost-effective solution to the sensing task while providing additional functionality for traffic law enforcement and surveillance. We present a video-based on-street parking occupancy detection system that can operate in real time. Our system accounts for the inherent challenges that exist in on-street parking settings, including illumination changes, rain, shadows, occlusions, and camera motion. Our method utilizes several components from video processing and computer vision for motion detection, background subtraction, and vehicle detection. We also present three traffic law enforcement applications: parking angle violation detection, parking boundary violation detection, and exclusion zone violation detection, which can be integrated into the parking occupancy cameras as a value-added option. Our experimental results show that the proposed parking occupancy detection method performs in real-time at 5  frames/s and achieves better than 90% detection accuracy across several days of videos captured in a busy street block under various weather conditions such as sunny, cloudy, and rainy, among others.

Computer vision in roadway transportation systems: a survey

Abstract. There is a worldwide effort to apply 21st century intelligence to evolving our transportation networks. The goals of smart transportation networks are quite noble and manifold, including safety, efficiency, law enforcement, energy conservation, and emission reduction. Computer vision is playing a key role in this transportation evolution. Video imaging scientists are providing intelligent sensing and processing technologies for a wide variety of applications and services. There are many interesting technical challenges including imaging under a variety of environmental and illumination conditions, data overload, recognition and tracking of objects at high speed, distributed network sensing and processing, energy sources, as well as legal concerns. This paper presents a survey of computer vision techniques related to three key problems in the transportation domain: safety, efficiency, and security and law enforcement. A broad review of the literature is complemented by detailed treatment of a few selected algorithms and systems that the authors believe represent the state-of-the-art.

Non contact monitoring of respiratory function via depth sensing

Monitoring respiratory events is of clinical importance in the early detection of potentially fatal conditions. Current technologies involve contact sensors the individual must wear constantly. Such a requirement can lead to patient discomfort, and consequently may fail due to a variety of reasons including refusal to wear the monitoring device. Elderly patients and neo-natal infants are even more likely to suffer from the adverse effects of continued monitoring. Unobtrusive, non-contact, remote-sensing-based methods are increasingly needed for monitoring patient respiratory function at homes, which can in turn help to establish patterns over time. We propose to use active-stereo-based depth sensing system for forced flow-volume loop measurements and for semi-automatic and automatic assessment of abnormal breathing patterns.

Parking lot occupancy determination from lamp-post camera images

In recent years, detection of parking space availability has become of great importance worldwide due to its high correlation with fuel consumption and traffic congestion. We propose a novel framework for the automatic detection of vacant parking spaces from a lamp-post camera view of a parking lot. Our method models the 3-D volume parking spaces based on the parking lot geometry. The occupancy of the parking lot is determined based on a vehicle detector and the inferred volume of each space. We evaluate our method on three different datasets and show that its accuracy is close to 80% on a wide variety of test images.

On-the-fly hand detection training with application in egocentric action recognition

We propose a novel approach to segment hand regions in egocentric video that requires no manual labeling of training samples. The user wearing a head-mounted camera is prompted to perform a simple gesture during an initial calibration step. A combination of color and motion analysis that exploits knowledge of the expected gesture is applied on the calibration video frames to automatically label hand pixels in an unsupervised fashion. The hand pixels identified in this manner are used to train a statistical-model-based hand detector. Superpixel region growing is used to perform segmentation refinement and improve robustness to noise. Experiments show that our hand detection technique based on the proposed on-the-fly training approach significantly outperforms state-of-the-art techniques with respect to accuracy and robustness on a variety of challenging videos. This is due primarily to the fact that training samples are personalized to a specific user and environmental conditions. We also demonstrate the utility of our hand detection technique to inform an adaptive video sampling strategy that improves both computational speed and accuracy of egocentric action recognition algorithms. Finally, we offer an egocentric video dataset of an insulin self-injection procedure with action labels and hand masks that can serve towards future research on both hand detection and egocentric action recognition.

Deep Multimodal Representation Learning from Temporal Data

In recent years, Deep Learning has been successfully applied to multimodal learning problems, with the aim of learning useful joint representations in data fusion applications. When the available modalities consist of time series data such as video, audio and sensor signals, it becomes imperative to consider their temporal structure during the fusion process. In this paper, we propose the Correlational Recurrent Neural Network (CorrRNN), a novel temporal fusion model for fusing multiple input modalities that are inherently temporal in nature. Key features of our proposed model include: (i) simultaneous learning of the joint representation and temporal dependencies between modalities, (ii) use of multiple loss terms in the objective function, including a maximum correlation loss term to enhance learning of cross-modal information, and (iii) the use of an attention model to dynamically adjust the contribution of different input modalities to the joint representation. We validate our model via experimentation on two different tasks: video-and sensor-based activity classification, and audio-visual speech recognition. We empirically analyze the contributions of different components of the proposed CorrRNN model, and demonstrate its robustness, effectiveness and state-of-the-art performance on multiple datasets.

Two Algorithms for Compressed Sensing of Sparse Tensors

Compressed sensing (CS) exploits the sparsity of a signal in order to integrate acquisition and compression. CS theory enables exact reconstruction of a sparse signal from relatively few linear measurements via a suitable nonlinear minimization process. Conventional CS theory relies on vectorial data representation, which results in good compression ratios at the expense of increased computational complexity. In applications involving color images, video sequences, and multi-sensor networks, the data is intrinsically of high order, and thus more suitably represented in tensorial form. Standard applications of CS to higher-order data typically involve representation of the data as long vectors that are in turn measured using large sampling matrices, thus imposing a huge computational and memory burden. In this chapter, we introduce Generalized Tensor Compressed Sensing (GTCS)—a unified framework for compressed sensing of higher-order tensors which preserves the intrinsic structure of tensorial data with reduced computational complexity at reconstruction. We demonstrate that GTCS offers an efficient means for representation of multidimensional data by providing simultaneous acquisition and compression from all tensor modes. In addition, we propound two reconstruction procedures, a serial method (GTCS-S) and a parallelizable method (GTCS-P), both capable of recovering a tensor based on noiseless or noisy observations. We then compare the performance of the proposed methods with Kronecker compressed sensing (KCS) and multi-way compressed sensing (MWCS). We demonstrate experimentally that GTCS outperforms KCS and MWCS in terms of both reconstruction accuracy (within a range of compression ratios) and processing speed. The major disadvantage of our methods (and of MWCS as well) is that the achieved compression ratios may be worse than those offered by KCS.

Efficient processing of transportation surveillance videos in the compressed domain

Abstract. Video surveillance is used extensively in intelligent transportation systems to enforce laws, collect tolls, and regularize traffic flow. Benefits to society include reduced fuel consumption and emissions, improved safety, and reduced traffic congestion. These video cameras installed at traffic lights, highways, toll booths, etc., continuously capture video and hence generate a vast amount of data that are stored in large databases. The captured video is typically compressed before being transmitted and/or stored. While all the archived information is present in the compressed video, most current applications operate on uncompressed video. The aim is to improve the efficiency of processing by utilizing features of the compression process and the compressed video stream. Key methods that are employed involve intelligent selection of reference frames (I-frames) and exploitation of the compression motion vectors. Although specific applications in the transportation imaging domain are presented, the methods proposed here can generally impact the ability to mine vast amounts of video data for usable information in many diverse settings. Applications presented include rapid search for target vehicles (Amber Alert, Silver Alert, stolen car, etc.), vehicle counting, stop sign/light enforcement, and vehicle speed estimation.

Monocular vision-based vehicular speed estimation from compressed video streams

This paper introduces a monocular vision-based vehicular speed estimation algorithm that operates in the compressed domain. The algorithm relies on the use of motion vectors associated with video compression to achieve computationally efficient and accurate speed estimation. Building the speed estimation directly into the compression step adds only a small amount of computation which is conducive to real-time performance. We demonstrate the effectiveness of the algorithm on 30 fps video of one hundred and forty vehicles travelling at speeds ranging from 30 to 60 mph. The average speed estimation accuracy of our algorithm across the test set was better than 2.50% at a yield of 100%, with the accuracy increasing as the yield decreases and as the frame rate increases.