Alberto Maurizio Chavan

VLT Data Flow System: from concepts to operations

In order to realize the optimal scientific return from the VLT, ESO has undertaken to develop an end-to-end data flow system from proposal entry to science archive. The VLT Data Flow System (DFS) is being designed and implemented by the ESO Data Management and Operations Division in collaboration with VLT and Instrumentation Divisions. Tests of the DFS started in October 1996 on ESOs New Technology Telescope. Since then, prototypes of the Phase 2 Proposal Entry System, VLT Control System Interface, Data Pipelines, On-line Data Archive, Data Quality Control and Science Archive System have been tested. Several major DFS components have been run under operational conditions since February 1997. This paper describes the current status of the VLT DFS, the technological and operational challenges of such a system and the planing for VLT operations beginning in early 1999.

VLT Interferometer Data Flow System: from observation preparation to data processing

In this article we present the Data Flow System (DFS) for the Very Large Telescope Interferometer (VLTI). The Data Flow System is the VLT end-to-end software system for handling astronomical observations from the initial observation proposal phase through the acquisition, processing and control of the astronomical data. The Data Flow system is now in the process of installation and adaptation for the VLT Interferometer. The DFS was first installed for VLTI first fringes utilising the siderostats together with the VINCI instrument and is constantly being upgraded in phase with the VLTI commissioning. When completed the VLT Interferometer will make it possible to coherently combine up to three beams coming from the four VLT 8.2m telescopes as well as from a set of initially three 1.8m Auxiliary Telescopes, using a Delay Line tunnel and four interferometry instruments. Observations of objects with some scientific interest are already being carried out in the framework of the VLTI commissioning using siderostats and the VLT Unit Telescopes, making it possible to test tools under realistic conditions. These tools comprise observation preparation, pipeline processing and further analysis systems. Work is in progress for the commissioning of other VLTI science instruments such as MIDI and AMBER. These are planned for the second half of 2002 and first half of 2003 respectively. The DFS will be especially useful for service observing. This is expected to be an important mode of observation for the VLTI, which is required to cope with numerous observation constraints and the need for observations spread over extended periods of time.

Data flow system for the very large telescope interferometer

The Data Flow System is the VLT end-to-end system for handling astronomical observations from the initial observation proposal phase through the acquisition, processing and control of the astronomical data. The VLT Data Flow System has been in place since the opening of the first VLT Unit Telescope in 1998. When completed the VLT Interferometer will make it possible to coherently combine up to three beams coming from the four VLT 8.2m telescopes as well as from a set of initially three 1.8m Auxiliary Telescopes, using a Delay Line tunnel and four interferometry instruments. The Data Flow system is now in the process of installation and adaptation for the VLT Interferometer. Observation preparation for a multi-telescope system, handling large data volume of several tens of gigabytes per night are among the new challenges offered by this system. This introduction paper presents the VLTI Data Flow system installed during the initial phase of VLTI commissioning. Observation preparation, data archival, and data pipeline processing are addressed.

The last mile of the ALMA software development: lessons learned

At the end of 2012, ALMA software development will be completed. While new releases are still being prepared following an incremental development process, the ALMA software has been in daily use since 2008. Last year it was successfully used for the first science observations proposed by and released to the ALMA scientific community. This included the whole project life cycle from proposal preparation to data delivery, taking advantage of the software being designed as an end-to-end system. This presentation will report on software management aspects that became relevant in the last couple of years. These include a new feature driven development cycle, an improved software verification process, and a more realistic test environment at the observatory. It will also present a forward look at the planned transition to full operations, given that upgrades, optimizations and maintenance will continue for a long time.

Moor: web access to end-to-end data flow information at ESO

All ESO Science Operations teams operate on Observing Runs, loosely defined as blocks of observing time on a specific instrument. Observing Runs are submitted as part of an Observing Proposal and executed in Service or Visitor Mode. As an Observing Run progresses through its life-cycle, more and more information gets associated to it: Referee reports, feasibility and technical evaluations, constraints, pre-observation data, science and calibration frames, etc. The Manager of Observing Runs project (Moor) will develop a system to collect operational information in a database, offer integrated access to information stored in several independent databases, and allow HTML-based navigation over the whole information set. Some Moor services are also offered as extensions to, or complemented by, existing desktop applications.

Proposal and observing preparation for ALMA

A number of tools exist to aid in the preparation of proposals and observations for large ground and space-based observatories (VLT, Gemini, HST being examples). These tools have transformed the way in which astronomers use large telescopes. The ALMA telescope has a strong need for such a tool, but its scientific and technical requirements, and the nature of the telescope, provide some novel challenges. In addition to the common Phase I (Proposal) and Phase II (Observing) preparation the tool must support the needs of the novice alongside the needs of those who are expert in millimetre/sub-millimetre aperture synthesis astronomy. We must also provide support for the reviewing process, and must interface with and use the technical architecture underpinning the design of the ALMA Software System. In this paper we describe our approach to meeting these challenges.

VLT Data Flow System: the NTT prototype experience

The data flow system (DFS) for the ESO VLT provides a global system approach to the flow of science related data in the VLT environment. It includes components for preparation and scheduling of observations, archiving of data, pipeline data reduction and quality control. Standardized data structures serve as carriers for the exchange of information units between the DFS subsystems and VLT users and operators. Prototypes of the system were installed and tested at the New Technology Telescope. They helped us to clarify the astronomical requirements and check the new concepts introduced to meet the ambitious goals of the VLT. The experience gained from these tests is discussed.

The ALMA Snooping Project Interface (SnooPI)

In order to provide ALMA users with a comprehensive view of their observing projects, we developed the ALMA Snooping Project Interface (SnooPI) application. The simple and intuitive interface allows scientists to follow the status of their projects, broken down into observing unit sets and scheduling blocks. The application itself contains two separate parts: a Java back-end server and a JavaScript front-end client application. The application interacts with REST interfaces of other ALMA software components to get the necessary project reports, certain details describing the observations and to access statistics of the user’s ALMA Helpdesk tickets. All this information allows to successfully trace all stages of observations, processing and delivery of the ALMA science projects.

ESO scalable architecture for operational databases

The European Organisation for Astronomical Research in the Southern Hemisphere (ESO), headquartered in Garching, Germany, operates different state-of-the-art observing sites in Chile. To manage observatory operations and observation transfer, ESO developed an end-to-end Data Flow System, from Phase I proposal preparation to the final archiving of quality-controlled science, calibration and engineering data. All information pertinent to the data flow is stored in the central databases at ESO headquarters and replicated to and from the observatory database servers. In the ESOs data flow model one can distinguish two groups of databases; the front-end databases, which are replicated from the ESO headquarters to the observing sites, and the back-end databases, where replication is directed from the observations to the headquarters. A part of the front-end database contains the Observation Blocks (OBs), which are sequences of operations necessary to perform an observation, such as instrument setting, target, filter and/or grism ID, exposure time, etc. Observatory operations rely on fast access to the OB database and quick recovery strategies in case of a database outage. After several years of operations, those databases have grown considerably. There was a necessity in reviewing the database architecture to find a solution that support scalability of the operational databases. We present the newly developed concept of distributing the OBs between two databases, containing operational and historical information. We present the architectural design in which OBs in operational databases will be archived periodically at ESO headquarters. This will remedy the scalability problems and keep the size of the operational databases small. The historical databases will only exist in the headquarters, for archiving purposes.