Luís M. Silva

Improving Deep Neural Network Performance by Reusing Features Trained with Transductive Transference

Transfer Learning is a paradigm in machine learning to solve a target problem by reusing the learning with minor modifications from a different but related source problem. In this paper we propose a novel feature transference approach, especially when the source and the target problems are drawn from different distributions. We use deep neural networks to transfer either low or middle or higher-layer features for a machine trained in either unsupervised or supervised way. Applying this feature transference approach on Convolutional Neural Network and Stacked Denoising Autoencoder on four different datasets, we achieve lower classification error rate with significant reduction in computation time with lower-layer features trained in supervised way and higher-layer features trained in unsupervised way for classifying images of uppercase and lowercase letters dataset.

Can artificial neural networks be used to predict the origin of ozone episodes

Tropospheric ozone is a secondary pollutant having a negative impact on health and environment. To control and minimize such impact the European Community established regulations to promote a clean air all over Europe. However, when an episode is related with natural mechanisms as Stratosphere-Troposphere Exchanges (STE), the benefits of an action plan to minimize precursor emissions are inefficient. Therefore, this work aims to develop a tool to identify the sources of ozone episodes in order to minimize misclassification and thus avoid the implementation of inappropriate air quality plans. For this purpose, an artificial neural network model - the Multilayer Perceptron - is used as a binary classifier of the source of an ozone episode. Long data series, between 2001 and 2010, considering the ozone precursors, (7)Be activity and meteorological conditions were used. With this model, 2-7% of a mean error was achieved, which is considered as a good generalization. Accuracy measures for imbalanced data are also discussed. The MCC values show a good performance of the model (0.65-0.92). Precision and F1-measure indicate that the model specifies a little better the rare class. Thus, the results demonstrate that such a tool can be used to help authorities in the management of ozone, namely when its thresholds are exceeded due natural causes, as the above mentioned STE. Therefore, the resources used to implement an action plan to minimize ozone precursors could be better managed avoiding the implementation of inappropriate measures.

Using Different Cost Functions to Train Stacked Auto-Encoders

Deep neural networks comprise several hidden layers of units, which can be pre-trained one at a time via an unsupervised greedy approach. A whole network can then be trained (fine-tuned) in a supervised fashion. One possible pre-training strategy is to regard each hidden layer in the network as the input layer of an auto-encoder. Since auto-encoders aim to reconstruct their own input, their training must be based on some cost function capable of measuring reconstruction performance. Similarly, the supervised fine-tuning of a deep network needs to be based on some cost function that reflects prediction performance. In this work we compare different combinations of cost functions in terms of their impact on layer-wise reconstruction performance and on supervised classification performance of deep networks. We employed two classic functions, namely the cross-entropy (CE) cost and the sum of squared errors (SSE), as well as the exponential (EXP) cost, inspired by the error entropy concept. Our results were based on a number of artificial and real-world data sets.

Improving transfer learning accuracy by reusing Stacked Denoising Autoencoders

Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study transference on deep architectures, known to directly extract the features from the input data. A Stacked Denoising Autoencoder (SDA) is a deep model able to represent the hierarchical features needed for solving classification problems. In this paper we study the performance of SDAs trained on one problem and reused to solve a different problem not only with different distribution but also with a different tasks. We propose two different approaches: 1) unsupervised feature transference, and 2) supervised feature transference using deep transfer learning. We show that SDAs using the unsupervised feature transference outperform randomly initialized machines on a new problem. We achieved 7% relative improvement on average error rate and 41% on average computation time to classify typed uppercase letters. In the case of supervised feature transference, we achieved 5.7% relative improvement in the average error rate, by reusing the first and second hidden layer, and 8.5% relative improvement for the average error rate and 54% speed up w.r.t the baseline by reusing all three hidden layers for the same data. We also explore transfer learning between geometrical shapes and canonical shapes, we achieved 7.4% relative improvement on average error rate in case of supervised feature transference approach.

Thermochemical and structural studies of Cu(II) and Ni(II) complexes with N, N-diethyl-N'-pivaloylthiourea

Abstract The standard (p0=0.1 MPa) molar enthalpies of formation of crystalline bis(N,N-diethyl-N′-pivaloylthioureato)copper(II), Cu(PVET)2, and bis(N,N-diethyl-N′-pivaloylthioureato)nickel(II), Ni(PVET)2, were measured, at T=298.15 K, by solution–reaction isoperibol calorimetry. The standard molar enthalpies of sublimation, at T=298.15 K, of both complexes were obtained using a Knudsen effusion technique. These values were used to derive the standard molar enthalpy of formation of Cu(PVET)2 and Ni(PVET)2 in gaseous phase, and to evaluate the difference between the mean metal–ligand and the hydrogen–ligand bond dissociation enthalpies, in these compounds. The NH homolytic bond dissociation enthalpy in N,N-diethyl-N′-pivaloylthiourea ligand (HPVET) was calculated by high-level density functional theory based calculations. The three-dimensional structures of Cu(PVET)2 and Ni(PVET)2 are presented and show a planar coordination around the metal in both molecules.

Application of Artificial Neural Networks to Predict the Impact of Traffic Emissions on Human Health

Artificial Neural Networks (ANN) have been essentially used as regression models to predict the concentration of one or more pollutants usually requiring information collected from air quality stations. In this work we consider a Multilayer Perceptron (MLP) with one hidden layer as a classifier of the impact of air quality on human health, using only traffic and meteorological data as inputs. Our data was obtained from a specific urban area and constitutes a 2-class problem: above or below the legal limits of specific pollutant concentrations. The results show that an MLP with 40 to 50 hidden neurons and trained with the cross-entropy cost function, is able to achieve a mean error around 11%, meaning that air quality impacts can be predicted with good accuracy using only traffic and meteorological data. The use of an ANN without air quality inputs constitutes a significant achievement because governments may therefore minimize the use of such expensive stations.

Multi-source deep transfer learning for cross-sensor biometrics

Deep transfer learning emerged as a new paradigm in machine learning in which a deep model is trained on a source task and the knowledge acquired is then totally or partially transferred to help in solving a target task. In this paper, we apply the source–target–source methodology, both in its original form and an extended multi-source version, to the problem of cross-sensor biometric recognition. We tested the proposed methodology on the publicly available CSIP image database, achieving state-of-the-art results in a wide variety of cross-sensor scenarios.

Transfer Learning for the Recognition of Immunogold Particles in TEM Imaging

We present a (TL) framework based on (SDA) for the recognition of immunogold particles. These particles are part of a high-resolution method for the selective localization of biological molecules at the subcellular level only visible through (TEM). Four new datasets were acquired encompassing several thousands of immunogold particles. Due to the particles size (for a particular dataset a particle has a radius of 4 pixels in an image of size 4008\(\times \)2670) the annotation of these datasets is extremely time taking. Thereby, we apply a (TL) approach by reusing the learning model that can be used on other datasets containing particles of different (or similar) sizes. In our experimental study we verified that our (TL) framework outperformed the baseline (not involving TL) approach by more than 20% of accuracy on the recognition of immunogold particles.

Deep Transfer Learning Ensemble for Classification

Transfer learning algorithms typically assume that the training data and the test data come from different distribution. It is better at adapting to learn new tasks and concepts more quickly and accurately by exploiting previously gained knowledge. Deep Transfer Learning (DTL) emerged as a new paradigm in transfer learning in which a deep model offer greater flexibility in extracting high-level features. DTL offers selective layer based transference, and it is problem specific. In this paper, we propose the Ensemble of Deep Transfer Learning (EDTL) methodology to reduce the impact of selective layer based transference and provide optimized framework to work for three major transfer learning cases. Empirical results on character, object and biomedical image recognition tasks achieves that the proposed method indicate statistically significant classification accuracy over the other established transfer learning method.

Source-Target-Source Classification Using Stacked Denoising Autoencoders

Deep Transfer Learning (DTL) emerged as a new paradigm in machine learning in which a deep model is trained on a source task and the knowledge acquired is then totally or partially transferred to help in solving a target task. Even though DTL offers a greater flexibility in extracting high-level features and enabling feature transference from a source to a target task, the DTL solution might get stuck at local minima leading to performance degradation-negative transference-, similar to what happens in the classical machine learning approach. In this paper, we propose the Source-Target-Source (STS) methodology to reduce the impact of negative transference, by iteratively switching between source and target tasks in the training process. The results show the effectiveness of such approach.