Thi Ngoc Tho Nguyen

Leave Your Phone at the Door: Side Channels that Reveal Factory Floor Secrets

From pencils to commercial aircraft, every man-made object must be designed and manufactured. When it is cheaper or easier to steal a design or a manufacturing process specification than to invent ones own, the incentive for theft is present. As more and more manufacturing data comes online, incidents of such theft are increasing. In this paper, we present a side-channel attack on manufacturing equipment that reveals both the form of a product and its manufacturing process, i.e., exactly how it is made. In the attack, a human deliberately or accidentally places an attack-enabled phone close to the equipment or makes or receives a phone call on any phone nearby. The phone executing the attack records audio and, optionally, magnetometer data. We present a method of reconstructing the products form and manufacturing process from the captured data, based on machine learning, signal processing, and human assistance. We demonstrate the attack on a 3D printer and a CNC mill, each with its own acoustic signature, and discuss the commonalities in the sensor data captured for these two different machines. We compare the quality of the data captured with a variety of smartphone models. Capturing data from the 3D printer, we reproduce the form and process information of objects previously unknown to the reconstructors. On average, our accuracy is within 1 mm in reconstructing the length of a line segment in a fabricated objects shape and within 1 degree in determining an angle in a fabricated objects shape. We conclude with recommendations for defending against these attacks.

Robust speech recognition using beamforming with adaptive microphone gains and multichannel noise reduction

This paper presents a robust speech recognition system using a microphone array for the 3rd CHiME Challenge. A minimum variance distortionless response (MVDR) beamformer with adaptive microphone gains is proposed for robust beamforming. Two microphone gain estimation methods are studied using the speech-dominant time-frequency bins. A multichannel noise reduction (MCNR) postprocessing is also proposed to further reduce the interference in the MVDR processed signal. Experimental results for the ChiME-3 challenge show that both the proposed MVDR beamformer with microphone gains and the MCNR postprocessing improve the speech recognition performance significantly. With the state-of-the-art deep neural network (DNN) based acoustic model, our system achieves a word error rate (WER) of 11.67% on the real test data of the evaluation set.

Large region acoustic source mapping using movable arrays

Mapping environmental noise with high resolution on a large scale (such as a city) is prohibitively expensive with current approaches, which use a large, dense array spanning the entire region of interest, or sequential noise measurements at thousands of locations on a dense grid. We propose instead a new acoustic measurement scheme using a small movable array (for example, mounted on a vehicle driving along the streets of a city) to rapidly acquire measurements at many different locations. A multiple-point sparse constrained deconvolution approach for the mapping of acoustic sources (MPSC-DAMAS) and a multiple-point covariance matrix fitting (MP-CMF) approach are developed to accurately estimate the locations and powers of stationary noise sources across the region of interest. Computer simulations of large region acoustic mapping demonstrate that superior resolution and much lower power estimation errors are achieved by the proposed approaches compared to the state-of-the-art SC-DAMAS approach and CMF approach.

An expectation-maximization eigenvector clustering approach to direction of arrival estimation of multiple speech sources

This paper presents an eigenvector clustering approach for estimating the direction of arrival (DOA) of multiple speech signals using a microphone array. Existing clustering approaches usually only use low frequencies to avoid spatial aliasing. In this study, we propose a probabilistic eigenvector clustering approach to use all frequencies. In our work, time-frequency (TF) bins dominated by only one source are first detected using a combination of noise-floor tracking, onset detection and coherence test. For each selected TF bin, the largest eigenvector of its spatial covariance matrix is extracted for clustering. A mixture density model is introduced to model the distribution of the eigenvectors, where each component distribution corresponds to one source and is parameterized by the source DOA. To use eigenvectors of all frequencies, the steering vectors of all frequencies of the sources are used in the distribution function. The DOAs of the sources can be estimated by maximizing the likelihood of the eigenvectors using an expectation-maximization (EM) algorithm. Simulation and experimental results show that the proposed approach significantly improves the root-mean-square error (RMSE) for DOA estimation of multiple speech sources compared to the MUSIC algorithm implemented on the single-source dominated TF bins and our previous clustering approach.

Large region acoustic source mapping: A generalized sparse constrained deconvolution approach

This paper presents a generalized multiple-point sparse constrained deconvolution approach for mapping acoustic noise sources in large regions using a movable array. Extended from our previous MPSC-DAMAS approach, we first derive a generalized inverse problem relating to the source powers and the array manifold using a generic beamformer and an explicit measurement noise model. We then propose a generalized MPSC-DAMAS (GMPSC-DAMAS) approach for resolving the inverse problem. A new parameter setting method based on a multiple-point minimum-variance-distortionless-response (MVDR) beamformer is also presented. The realizations of the GMPSC-DAMAS approach using the delay- and-sum (DAS) beamformer and the MVDR beamformer are evaluated. Simulation results show the proposed GMPSC-DAMAS approach achieves much lower absolute power estimation errors and processing time than the MPSC-DAMAS approach in terms of number of sources and robustness to measurement noise.

Drive-by large-region acoustic noise-source mapping via sparse beamforming tomography

Environmental noise is a risk factor for human physical and mental health, demanding an efficient large-scale noise-monitoring scheme. The current technology, however, involves extensive sound pressure level (SPL) measurements at a dense grid of locations, making it impractical on a city-wide scale. This paper presents an alternative approach using a microphone array mounted on a moving vehicle to generate two-dimensional acoustic tomographic maps that yield the locations and SPLs of the noise-sources sparsely distributed in the neighborhood traveled by the vehicle. The far-field frequency-domain delay-and-sum beamforming output power values computed at multiple locations as the vehicle drives by are used as tomographic measurements. The proposed method is tested with acoustic data collected by driving an electric vehicle with a rooftop-mounted microphone array along a straight road next to a large open field, on which various pre-recorded noise-sources were produced by a loudspeaker at different locations. The accuracy of the tomographic imaging results demonstrates the promise of this approach for rapid, low-cost environmental noise-monitoring.

Wideband compressive beamforming tomography for drive-by large-scale acoustic source mapping

Noise-mapping is an effective sound visualization tool for the identification of urban noise hotspots, which is crucial to taking targeted measures to tackle environmental noise pollution. This paper develops a high-resolution wideband acoustic source mapping methodology using a portable microphone array, where the joint localization and power spectrum estimation of individual sources sparsely distributed over a large region are achieved by tomographic imaging with the multi-frequency delay-and-sum beamforming power outputs from multiple array positions. Exploiting the fact that a wideband source has a common spatial signal-support across the frequency spectrum, two-dimensional tomographic maps are produced by applying compressive sensing techniques including group least absolute shrinkage selection operator formulation and sparse Bayesian learning to promote group sparsity over multiple frequency bands. The high-resolution mapping is demonstrated with experimental data recorded with a microphone array mounted atop an electric vehicle driven along a road while playing audio clips from a loudspeaker positioned within the adjacent open field.

A novel sparse model for multi-source localization using distributed microphone array

When distances between microphone pairs are larger than the half-wavelength of signals, source localization methods using cross-correlation such as time-difference-of-arrival (TDOA), steered response power (SRP) are commonly used in practice. We present here a novel model that expresses microphone pairwise cross-correlations as a sum of autocorrelations of source signals shifted by the relative delays of the signals arriving at the microphone pairs, and weighted by the source power and the distances between the sources and the microphone pairs. The model is formulated as a linear inverse problem and is sparse with respect to the source power map. The source power map, which directly shows the locations of all the sound sources, can be reconstructed using ℓ1-norm minimization algorithms. We demonstrate the effectiveness of our model in a wildlife monitoring application, where the goal is to locate multiple frogs in a dense chorus.

Large-region acoustic source mapping using a movable array and sparse covariance fitting

Large-region acoustic source mapping is important for city-scale noise monitoring. Approaches using a single-position measurement scheme to scan large regions using small arrays cannot provide clean acoustic source maps, while deploying large arrays spanning the entire region of interest is prohibitively expensive. A multiple-position measurement scheme is applied to scan large regions at multiple spatial positions using a movable array of small size. Based on the multiple-position measurement scheme, a sparse-constrained multiple-position vectorized covariance matrix fitting approach is presented. In the proposed approach, the overall sample covariance matrix of the incoherent virtual array is first estimated using the multiple-position array data and then vectorized using the Khatri-Rao (KR) product. A linear model is then constructed for fitting the vectorized covariance matrix and a sparse-constrained reconstruction algorithm is proposed for recovering source powers from the model. The user parameter settings are discussed. The proposed approach is tested on a 30 m × 40 m region and a 60 m × 40 m region using simulated and measured data. Much cleaner acoustic source maps and lower sound pressure level errors are obtained compared to the beamforming approaches and the previous sparse approach [Zhao, Tuna, Nguyen, and Jones, Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP) (2016)].

Sparse tomographic acoustic imaging for environmental noise-source mapping

Environmental noise has become a major problem in large cities, increasing health risks for the urban population. Current methods are overly expensive for city-wide noise-monitoring as they generally require dense deployment of fixed microphones for noise-level measurements. We present here alternative sparse tomographic acoustic imaging techniques using arrays with relatively few number of microphones for large-region acoustic noise mapping. We first demonstrate that the locations and sound pressure levels of fixed noise sources sparsely located in a large field are recovered by collecting acoustic data at multiple locations with a portable microphone array for tomographic reconstruction. We then introduce a nonstationary tomographic imaging approach using fixed microphone arrays, which can also capture the intermittent changes in the acoustic field due to transient and/or moving noise sources. We test both the sparse static and dynamic imaging models with acoustic measurements collected with a circular ...