Lex Fridman

Active Authentication on Mobile Devices via Stylometry, Application Usage, Web Browsing, and GPS Location

Active authentication is the problem of continuously verifying the identity of a person based on behavioral aspects of their interaction with a computing device. In this paper, we collect and analyze behavioral biometrics data from 200 subjects, each using their personal Android mobile device for a period of at least 30 days. This data set is novel in the context of active authentication due to its size, duration, number of modalities, and absence of restrictions on tracked activity. The geographical colocation of the subjects in the study is representative of a large closed-world environment such as an organization where the unauthorized user of a device is likely to be an insider threat: coming from within the organization. We consider four biometric modalities: 1) text entered via soft keyboard, 2) applications used, 3) websites visited, and 4) physical location of the device as determined from GPS (when outdoors) or WiFi (when indoors). We implement and test a classifier for each modality and organize the classifiers as a parallel binary decision fusion architecture. We are able to characterize the performance of the system with respect to intruder detection time and to quantify the contribution of each modality to the overall performance.

Learning Human Identity From Motion Patterns

We present a large-scale study exploring the capability of temporal deep neural networks to interpret natural human kinematics and introduce the first method for active biometric authentication with mobile inertial sensors. At Google, we have created a first-of-its-kind dataset of human movements, passively collected by 1500 volunteers using their smartphones daily over several months. We (1) compare several neural architectures for efficient learning of temporal multi-modal data representations, (2) propose an optimized shift-invariant dense convolutional mechanism (DCWRNN), and (3) incorporate the discriminatively-trained dynamic features in a probabilistic generative framework taking into account temporal characteristics. Our results demonstrate that human kinematics convey important information about user identity and can serve as a valuable component of multi-modal authentication systems.

What Can Be Predicted from Six Seconds of Driver Glances

We consider a large dataset of real-world, on-road driving from a 100-car naturalistic study to explore the predictive power of driver glances and, specifically, to answer the following question: what can be predicted about the state of the driver and the state of the driving environment from a 6-second sequence of macro-glances? The context-based nature of such glances allows for application of supervised learning to the problem of vision-based gaze estimation, making it robust, accurate, and reliable in messy, real-world conditions. So, its valuable to ask whether such macro-glances can be used to infer behavioral, environmental, and demographic variables? We analyze 27 binary classification problems based on these variables. The takeaway is that glance can be used as part of a multi-sensor real-time system to predict radio-tuning, fatigue state, failure to signal, talking, and several environment variables.

Automated synchronization of driving data using vibration and steering events

A passive synchronization method for driving data is proposedSynchronization of vehicle sensors uses vibration and steering events.Dense optical flow of video is used to capture significant car vibrations events.Cross correlation of vehicle sensor pairs achieves 13.5ms synchronization accuracy. We propose a method for automated synchronization of vehicle sensors useful for the study of multi-modal driver behavior and for the design of advanced driver assistance systems. Multi-sensor decision fusion relies on synchronized data streams in (1) the offline supervised learning context and (2) the online prediction context. In practice, such data streams are often out of sync due to the absence of a real-time clock, use of multiple recording devices, or improper thread scheduling and data buffer management. Cross-correlation of accelerometer, telemetry, audio, and dense optical flow from three video sensors is used to achieve an average synchronization error of 13 milliseconds. The insight underlying the effectiveness of the proposed approach is that the described sensors capture overlapping aspects of vehicle vibrations and vehicle steering allowing the cross-correlation function to serve as a way to compute the delay shift in each sensor. Furthermore, we show the decrease in synchronization error as a function of the duration of the data stream.

Detecting road surface wetness from audio: A deep learning approach

We introduce a recurrent neural network architecture for automated road surface wetness detection from audio of tire-surface interaction. The robustness of our approach is evaluated on 785,826 bins of audio that span an extensive range of vehicle speeds, noises from the environment, road surface types, and pavement conditions including international roughness index (IRI) values from 25 in/mi to 1400 in/mi. The training and evaluation of the model are performed on different roads to minimize the impact of environmental and other external factors on the accuracy of the classification. We achieve an unweighted average recall (UAR) of 93.2% across all vehicle speeds including 0 mph. The classifier still works at 0 mph because the discriminating signal is present in the sound of other vehicles driving by. Removing audio segments at speeds below 2.9 mph from consideration improves the UAR to 100%. In the case when the vehicle speed is below 2.9 mph, we were able to discriminate between wet and dry road surfaces from ambient noises and achieve 74.5% UAR.

Investigating the correspondence between driver head position and glance location

The relationship between a drivers glance orientation and corresponding head rotation is highly complex due to its nonlinear dependence on the individual, task, and driving context. This paper presents expanded analytic detail and findings from an effort that explored the ability of head pose to serve as an estimator for driver gaze by connecting head rotation data with manually coded gaze region data using both a statistical analysis approach and a predictive (i.e., machine learning) approach. For the latter, classification accuracy increased as visual angles between two glance locations increased. In other words, the greater the shift in gaze, the higher the accuracy of classification. This is an intuitive but important concept that we make explicit through our analysis. The highest accuracy achieved was 83% using the method of Hidden Markov Models (HMM) for the binary gaze classification problem of (a) glances to the forward roadway versus (b) glances to the center stack. Results suggest that although there are individual differences in head-glance correspondence while driving, classifier models based on head-rotation data may be robust to these differences and therefore can serve as reasonable estimators for glance location. The results suggest that driver head pose can be used as a surrogate for eye gaze in several key conditions including the identification of high-eccentricity glances. Inexpensive driver head pose tracking may be a key element in detection systems developed to mitigate driver distraction and inattention.

Deep Learning for Understanding the Human

We will explore how deep learning approaches can be used for perceiving and interpreting the state and behavior of human beings in images, video, audio, and text data. The course will cover how convolutional, recurrent and generative neural networks can be used for applications of face recognition, eye tracking, cognitive load estimation, emotion recognition, natural language processing, voice-based interaction, and activity recognition. The course is open to beginners and is designed for those who are new to deep learning, but it can also benefit advanced researchers in the field looking for a practical overview of deep learning methods and their application.

Cognitive Load Estimation in the Wild

Cognitive load has been shown, over hundreds of validated studies, to be an important variable for understanding human performance. However, establishing practical, non-contact approaches for automated estimation of cognitive load under real-world conditions is far from a solved problem. Toward the goal of designing such a system, we propose two novel vision-based methods for cognitive load estimation, and evaluate them on a large-scale dataset collected under real-world driving conditions. Cognitive load is defined by which of 3 levels of a validated reference task the observed subject was performing. On this 3-class problem, our best proposed method of using 3D convolutional neural networks achieves 86.1% accuracy at predicting task-induced cognitive load in a sample of 92 subjects from video alone. This work uses the driving context as a training and evaluation dataset, but the trained network is not constrained to the driving environment as it requires no calibration and makes no assumptions about the subjects visual appearance, activity, head pose, scale, and perspective.

Machine Learning and Human Factors: Status, Applications, and Future Directions

As machine learning approaches ubiquity in industrial systems and consumer products, human factors research must attend to machine learning, specifically on how intelligent systems built on machine learning are different from early generations of automated systems, and what these differences mean for human-system interaction, design, evaluation and training. This panel invites five researchers in different domains to discuss how human factors can contribute to machine learning research and applications, as well as how machine learning presents both challenges and contributions for human factors.

Linking the Detection Response Task and the AttenD Algorithm Through Assessment of Human–Machine Interface Workload

Multitasking related demands can adversely affect drivers’ allocation of attention to the roadway, resulting in delays or missed responses to roadway threats and to decrements in driving performance. Robust methods for obtaining evidence and data about demands on and decrements in the allocation of driver attention are needed as input for design, training, and policy. The detection response task (DRT) is a commonly used method (ISO 17488) for measuring the attentional effects of cognitive load. The AttenD algorithm is a method intended to measure driver distraction through real-time glance analysis, in which individual glances are converted into a scalar value using simple rules considering glance duration, frequency, and location. A relationship between the two tools is explored. A previous multitasking driving simulation study, which used the remote form of the DRT to differentiate the demands of a primary visual–manual human–machine interface from alternative primary auditory–vocal multimodal human–machine interfaces, was reanalyzed using AttenD, and the two analyses compared. Results support an association between DRT performance and AttenD algorithm output. Summary statistics produced from AttenD profiles differentiate between the demands of the human–machine interfaces considered with more power than analyses of DRT response time and miss rate. Among discussed implications is the possibility that AttenD taps some of the same attentional effects as the DRT. Future research paths, strategies for analyses of past and future data sets, and possible application for driver state detection are also discussed.