Nausicaa Delmotte

Data products in the ESO Science Archive Facility

The European Southern Observatory Science Archive Facility is evolving from an archive containing predominantly raw data into a resource also offering science-grade data products for immediate analysis and prompt interpretation. New products originate from two different sources. On the one hand Principal Investigators of Public Surveys and other programmes reduce the raw observational data and return their products using the so-called Phase 3 - a process that extends the Data Flow System after proposal submission (Phase 1) and detailed specification of the observations (Phase 2). On the other hand raw data of selected instruments and modes are uniformly processed in-house, independently of the original science goal. Current data products assets in the ESO science archive facility include calibrated images and spectra, as well as catalogues, for a total volume in excess of 16 TB and increasing. Images alone cover more than 4500 square degrees in the NIR bands and 2400 square degrees in the optical bands; over 85000 individually searchable spectra are already available in the spectroscopic data collection. In this paper we review the evolution of the ESO science archive facility content, illustrate the data access by the community, give an overview of the implemented processes and the role of the associated data standard.

Data interface control at the European Southern Observatory

The European Southern Observatory (ESO) manages numerous telescopes which use various types of instruments and readout detectors. The data flow process at ESOs observatories involves several steps: telescope setup, data acquisition (science, calibration and test), pipeline processing, quality control, archivisation, distribution of data to the users. Well defined interfaces are vital for the smooth operation of such complex structures. Also, the future expansion of ESO operations - such as development of new observatories (e.g. ALMA) and supporting the Virtual Observatory (VO) - will make maintenance of data interfaces even more critical. In this paper we present the overview of the current status of the Data Interface Control process at ESO and discuss the future expansion plans.

The ESO science archive: supporting and enhancing science from the La Silla Paranal Observatory

The archive of the La Silla Paranal Observatory is a powerful science resource for the ESO astronomical community. It stores both the raw data generated by all ESO instruments and selected processed (science-ready) data. We present the new capabilities and user services that have recently been developed in order to enhance data discovery and usage in the face of the increasing volume and complexity of the archive holdings. Future plans to extend the new services to processed data from the Atacama Large Millimeter/submillimeter Array (ALMA) are also discussed.

Enhanced Data Discovery Services for the ESO Science Archive

The Messenger 172 – June 2018 the characteristics and limitations of each collection of processed data. This is particularly important, as it enables users to decide whether the data are suitable for their specific science goals. The systematic archive publication of such processed data dates back to 25 July 2011, with the first products produced by the Public Surveys conducted with the Visible and Infrared Survey Telescope for Astronomy (VISTA) infrared camera VIRCAM (Arnaboldi & Retzlaff, 2011). Processed data that were generated at ESO have been available since September 2013. An up-to-date overview of the released data is available online for contributed and pipeline processed data 2, 3.

ESO Public Surveys at VISTA: Lessons learned from Cycle 1 Surveys and the start of Cycle 2

The ESO Public Surveys on VISTA serve the science goals of the survey teams while increasing the legacy value of ESO programmes, thanks to their homogeneity and the breadth of their sky coverage in multiple bands. These projects address a variety of research areas: from the detection of planets via microlensing, to stars, the Milky Way and Local Group galaxies, to extragalactic astronomy, galaxy evolution, the highredshift Universe and cosmology. In 2015, as the first generation of imaging surveys was nearing completion, a second call for Public Surveys was opened to define a coherent scientific programme for VISTA until the commissioning of the wide-field multi-fibre spectrograph, 4MOST, in 2020. This article presents the status of the Cycle 1 surveys as well as an overview of the seven new programmes in Cycle 2, including their science goals, coverage on the sky and observing strategies. We conclude with a forward look at the Cycle 2 data releases and the timelines for their release.

Validation of ESO Phase 3 data submissions

The data validation phase is an essential step of the Phase 3 process at ESO that is defining and providing an infrastructure to deal with interactions between the data producers and the archive. We are using a controlled process to systematically review all Phase 3 data submissions to ensure a homogeneous and consistent science archive with well traceable and characterised data products, to the benefits of archive users. How the Phase 3 data validation plan is defined and how its results are subsequently managed will be described in the presentation. For a description of its technical implementation, please refer to the contribution by L. Mascetti.

ESO science data product standard for 1D spectral products

The ESO Phase 3 process allows the upload, validation, storage, and publication of reduced data through the ESO Science Archive Facility. Since its introduction, ~2 million data products have been archived and published; 80% of them are one-dimensional extracted and calibrated spectra. Central to Phase3 is the ESO science data product standard that defines metadata and data format of any product. This contribution describes the ESO data standard for 1d-spectra, its adoption by the reduction pipelines of selected instrument modes for in-house generation of reduced spectra, the enhanced archive legacy value. Archive usage statistics are provided.

ESO Phase 3 automatic data validation: groovy-based tool to assure the compliance of the reduced data with the Science Data Product Standard

The ESO Phase 3 infrastructure provides a channel to submit reduced data products for publication to the astronomical community and long-term data preservation in the ESO Science Archive Facility. To be integrated into Phase 3, data must comply to the ESO Science Data Product Standard regarding format (one unique standard data format is associated to each type of product, like image, spectrum, IFU cube, etc.) and required metadata. ESO has developed a Groovy based tool that carries out an automatic validation of the submitted reduced products that is triggered when data are uploaded and then submitted. Here we present how the tool is structured and which checks are implemented.

Publication of science data products through the ESO archive: lessons learned and future evolution

Phase 3 denotes the process of preparation, submission, validation and ingestion of science data products for storage in the ESO Science Archive Facility and subsequent publication to the scientific community. In this paper we will review more than four years of Phase 3 operations at ESO and we will discuss the future evolution of the Phase 3 system.

Public Surveys at ESO

ESO has a strong mandate to survey the Southern Sky. In this article, we describe the ESO telescopes and instruments that are currently used for ESO Public Surveys, and the future plans of the community with the new wide-field-spectroscopic instruments. We summarize the ESO policies governing the management of these projects on behalf of the community. The on-going ESO Public Surveys and their science goals, their status of completion, and the new projects selected during the second ESO VISTA call in 2015/2016 are discussed. We then present the impact of these projects in terms of current numbers of refereed publications and the scientific data products published through the ESO Science Archive Facility by the survey teams, including the independent access and scientific use of the published survey data products by the astronomical community.