D. Garabato

Analysis and Knowledge Discovery by Means of Self-Organizing Maps for Gaia Data Releases

A billion stars: this is the approximate amount of visible objects estimated to be observed by the Gaia satellite, representing roughly 1 % of the objects in the Galaxy. It constitutes the biggest amount of data gathered to date: by the end of the mission, the data archive will exceed 1 Petabyte. Now, in order to process this data, the Gaia mission conceived the Data Processing and Analysis Consortium, which will apply data mining techniques such as Self-Organizing Maps. This paper shows a useful technique for source clustering, focusing on the development of an advanced visualization tool based on this technique.

On the estimation of stellar parameters with uncertainty prediction from Generative Artificial Neural Networks: application to Gaia RVS simulated spectra

Aims. We present an innovative artificial neural network (ANN) architecture, called Generative ANN (GANN), that computes the forward model, that is it learns the function that relates the unknown outputs (stellar atmospheric parameters, in this case) to the given inputs (spectra). Such a model can be integrated in a Bayesian framework to estimate the posterior distribution of the outputs. Methods. The architecture of the GANN follows the same scheme as a normal ANN, but with the inputs and outputs inverted. We train the network with the set of atmospheric parameters (Teff, logg, [Fe/H] and [alpha/Fe]), obtaining the stellar spectra for such inputs. The residuals between the spectra in the grid and the estimated spectra are minimized using a validation dataset to keep solutions as general as possible. Results. The performance of both conventional ANNs and GANNs to estimate the stellar parameters as a function of the star brightness is presented and compared for different Galactic populations. GANNs provide significantly improved parameterizations for early and intermediate spectral types with rich and intermediate metallicities. The behaviour of both algorithms is very similar for our sample of late-type stars, obtaining residuals in the derivation of [Fe/H] and [alpha/Fe] below 0.1dex for stars with Gaia magnitude Grvs<12, which accounts for a number in the order of four million stars to be observed by the Radial Velocity Spectrograph of the Gaia satellite. Conclusions. Uncertainty estimation of computed astrophysical parameters is crucial for the validation of the parameterization itself and for the subsequent exploitation by the astronomical community. GANNs produce not only the parameters for a given spectrum, but a goodness-of-fit between the observed spectrum and the predicted one for a given set of parameters. Moreover, they allow us to obtain the full posterior distribution...

Distributed Fast Self-Organized Maps for Massive Spectrophotometric Data Analysis †

Analyzing huge amounts of data becomes essential in the era of Big Data, where databases are populated with hundreds of Gigabytes that must be processed to extract knowledge. Hence, classical algorithms must be adapted towards distributed computing methodologies that leverage the underlying computational power of these platforms. Here, a parallel, scalable, and optimized design for self-organized maps (SOM) is proposed in order to analyze massive data gathered by the spectrophotometric sensor of the European Space Agency (ESA) Gaia spacecraft, although it could be extrapolated to other domains. The performance comparison between the sequential implementation and the distributed ones based on Apache Hadoop and Apache Spark is an important part of the work, as well as the detailed analysis of the proposed optimizations. Finally, a domain-specific visualization tool to explore astronomical SOMs is presented.

Distributed Unsupervised Clustering for Outlier Analysis in the Biggest Milky Way Survey: ESA Gaia Mission

The Gaia mission (ESA) is collecting huge amounts of information about the objects that populate our Galaxy and beyond. Such data must be processed and analyzed before being released, and this work is carried out by the Data Processing and Analysis Consortium (DPAC) through several work packages. One of these packages is Outlier Analysis, devoted to the study, by means of unsupervised clustering, of all the objects that cannot be fitted into any of the existent models. An algorithm based on optimized Self-Organized Maps (SOM) is proposed and implemented for taking advantage of distributed computing platforms, such as the MapReduce paradigm for Apache Hadoop and Apache Spark. Finally, the processing times of the sequential implementation of the algorithm is compared to the Hadoop and Spark based ones.