Michele Peron

VLT Data Flow System: from concepts to operations

In order to realize the optimal scientific return from the VLT, ESO has undertaken to develop an end-to-end data flow system from proposal entry to science archive. The VLT Data Flow System (DFS) is being designed and implemented by the ESO Data Management and Operations Division in collaboration with VLT and Instrumentation Divisions. Tests of the DFS started in October 1996 on ESOs New Technology Telescope. Since then, prototypes of the Phase 2 Proposal Entry System, VLT Control System Interface, Data Pipelines, On-line Data Archive, Data Quality Control and Science Archive System have been tested. Several major DFS components have been run under operational conditions since February 1997. This paper describes the current status of the VLT DFS, the technological and operational challenges of such a system and the planing for VLT operations beginning in early 1999.

Data reduction pipelines for the Very Large Telescope

With the completion of the first generation instrumentation set on the Very Large Telescope, a total of eleven instruments are now provided at the VLT/VLTI for science operations. For each of them, ESO provides automatic data reduction facilities in the form of instrument pipelines developed in collaboration with the instrument consortia. The pipelines are deployed in different environments, at the observatory and at the ESO headquarters, for on-line assessment of observations, instruments and detector monitoring, as well as data quality control and products generation. A number of VLT pipelines are also distributed to the user community together with front-end applications for batch and interactive usage. The main application of the pipeline is to support the Quality Control process. However, ESO also aims to deliver pipelines that can generate science ready products for a major fraction of the scientific needs of the users. This paper provides an overview of the current developments for the VLT/VLTI next generation of instruments and of the prototyping studies of new tools for science users.

Design of the VLT data flow model

The basic objective of modern observatories is to globally maximize their efficiency and ensure a high, constant and predictable data quality. These challenges can only be met if the scientific operation of such facilities, from the submission of observing programs to the archiving of all information, is carried out in a consistent and well controlled manner. The size, complexity and long operational lifetime of such systems make it difficult to predict and control their behavior with the necessary accuracy. Moreover they are subject to changes and are cumbersome to maintain. We present in this paper an object-oriented end-to-end operations model which describes the flow of science data associated with the operation of the VLT. The analysis model helped us to get a clear understanding of the problem domain. We were able in the design phase to partition the system into subsystems, each of them being allocated to a team for detailed design and implementation. Each of these subsystems is addressed in this paper. Prototypes will be implemented in the near future and tested on the new technology telescope (NTT). They will allow us to clarify the astronomical requirements and check the new operational concepts introduced to meet the ambitious goals of the VLT.

VLT Interferometer Data Flow System: from observation preparation to data processing

In this article we present the Data Flow System (DFS) for the Very Large Telescope Interferometer (VLTI). The Data Flow System is the VLT end-to-end software system for handling astronomical observations from the initial observation proposal phase through the acquisition, processing and control of the astronomical data. The Data Flow system is now in the process of installation and adaptation for the VLT Interferometer. The DFS was first installed for VLTI first fringes utilising the siderostats together with the VINCI instrument and is constantly being upgraded in phase with the VLTI commissioning. When completed the VLT Interferometer will make it possible to coherently combine up to three beams coming from the four VLT 8.2m telescopes as well as from a set of initially three 1.8m Auxiliary Telescopes, using a Delay Line tunnel and four interferometry instruments. Observations of objects with some scientific interest are already being carried out in the framework of the VLTI commissioning using siderostats and the VLT Unit Telescopes, making it possible to test tools under realistic conditions. These tools comprise observation preparation, pipeline processing and further analysis systems. Work is in progress for the commissioning of other VLTI science instruments such as MIDI and AMBER. These are planned for the second half of 2002 and first half of 2003 respectively. The DFS will be especially useful for service observing. This is expected to be an important mode of observation for the VLTI, which is required to cope with numerous observation constraints and the need for observations spread over extended periods of time.

Data flow system for the very large telescope interferometer

The Data Flow System is the VLT end-to-end system for handling astronomical observations from the initial observation proposal phase through the acquisition, processing and control of the astronomical data. The VLT Data Flow System has been in place since the opening of the first VLT Unit Telescope in 1998. When completed the VLT Interferometer will make it possible to coherently combine up to three beams coming from the four VLT 8.2m telescopes as well as from a set of initially three 1.8m Auxiliary Telescopes, using a Delay Line tunnel and four interferometry instruments. The Data Flow system is now in the process of installation and adaptation for the VLT Interferometer. Observation preparation for a multi-telescope system, handling large data volume of several tens of gigabytes per night are among the new challenges offered by this system. This introduction paper presents the VLTI Data Flow system installed during the initial phase of VLTI commissioning. Observation preparation, data archival, and data pipeline processing are addressed.

VLT End-to-End Science Operations: The First Three Years

The end-to-end operations of the ESO VLT has now seen three full years of service to the ESO community. During that time its capabilities have grown to four 8.2m unit telescopes with a complement of four optical and IR multimode instruments being operated in a mixed Service Mode and Visitor Mode environment. The input and output of programs and data to the system is summarized over this period together with the growth in operations manpower. We review the difficulties of working in a mixed operations and development environment and the ways in which the success of the end-to-end approach may be measured. Finally we summarize the operational lessons learned and the challenges posed by future developments of VLT instruments and facilities such as interferometry and survey telescopes.

Moor: web access to end-to-end data flow information at ESO

All ESO Science Operations teams operate on Observing Runs, loosely defined as blocks of observing time on a specific instrument. Observing Runs are submitted as part of an Observing Proposal and executed in Service or Visitor Mode. As an Observing Run progresses through its life-cycle, more and more information gets associated to it: Referee reports, feasibility and technical evaluations, constraints, pre-observation data, science and calibration frames, etc. The Manager of Observing Runs project (Moor) will develop a system to collect operational information in a database, offer integrated access to information stored in several independent databases, and allow HTML-based navigation over the whole information set. Some Moor services are also offered as extensions to, or complemented by, existing desktop applications.

ESO data flow system in operations: closing the data loop

On 1 April 1999, the first unit telescope (ANTU) of the ESO VLT began science operations. Two new instruments (FORS-1 for optical imaging and spectroscopy and ISAAC for IR imaging and spectroscopy) were offered in a mix of 50% visitor mode and 50% service mode. A Phase-I and Phase-II proposal and observation preparation process was conducted from 1 October 1998 until the middle of March 1999 involving approximately 280 proposals. A total of 1768 Observation Blocks for 83 approved service mode programs were scheduled and executed between 1 April and 1 October 1999. The resultant raw science and calibration data were subjected to quality control in Garching and released to the ESO user community starting from 15 June 1999 along with pipeline processed data products for a subset of instrument modes. The data flow loop for the first LT telescope is closed. The current operational VLT data flow system and the developments for the remainder of the VLT will be presented in the light of the first year of operational experience.

VLT Data Flow System: the NTT prototype experience

The data flow system (DFS) for the ESO VLT provides a global system approach to the flow of science related data in the VLT environment. It includes components for preparation and scheduling of observations, archiving of data, pipeline data reduction and quality control. Standardized data structures serve as carriers for the exchange of information units between the DFS subsystems and VLT users and operators. Prototypes of the system were installed and tested at the New Technology Telescope. They helped us to clarify the astronomical requirements and check the new concepts introduced to meet the ambitious goals of the VLT. The experience gained from these tests is discussed.

Diversity at ESO: Paranal Observatory

As diversity continues to grow in astronomy, creating working environments that are equally beneficial to all employees is imperative. Diversity in astronomical observatories is evident in a number of employee characteristics, including gender, race/ethnicity, age. Since June 2017, ESO has created its Diversity and Inclusion Committee gathering a variety of employees from the different sites, with different backgrounds. We will focus here on the status of the diversity and the strategies to develop a skilled and diverse operational workforce in the ESO observatories.