Jorge Ibsen

ALMA software regression tests: the evolution under an operational environment

The ALMA software is a large collection of modules, which implements the functionality needed for the observatory day-to-day operations, including among others Array/Antenna Control, Correlator, Telescope Calibration and Data Archiving. Many software patches must periodically be applied to fix problems detected during operations or to introduce enhancements after a release has been deployed and used under regular operational conditions. Under this scenery, it has been imperative to establish, besides a strict conguration control system, a weekly regression test to ensure that modications applied do not impact system stability and functionality. A test suite has been developed for this purpose, which reflects the operations performed by the commissioning and operations groups, and that aims to detect problems associated to the changes introduced at different versions of ALMA software releases. This paper presents the evolution of the regression test suite, which started at the ALMA Test Facility, and that has been adapted to be executed in the current operational conditions. Topics about the selection of the tests to be executed, the validation of the obtained data and the automation of the test suite are also presented.

First year of ALMA site software deployment: where everything comes together

Starting 2009, the ALMA project initiated one of its most exciting phases within construction: the first antenna from one of the vendors was delivered to the Assembly, Integration and Verification team. With this milestone and the closure of the ALMA Test Facility in New Mexico, the JAO Computing Group in Chile found itself in the front line of the projects software deployment and integration effort. Among the groups main responsibilities are the deployment, configuration and support of the observation systems, in addition to infrastructure administration, all of which needs to be done in close coordination with the development groups in Europe, North America and Japan. Software support has been the primary interaction key with the current users (mainly scientists, operators and hardware engineers), as the software is normally the most visible part of the system. During this first year of work with the production hardware, three consecutive software releases have been deployed and commissioned. Also, the first three antennas have been moved to the Array Operations Site, at 5.000 meters elevation, and the complete end-to-end system has been successfully tested. This paper shares the experience of this 15-people group as part of the construction team at the ALMA site, and working together with Computing IPT, on the achievements and problems overcomed during this period. It explores the excellent results of teamwork, and also some of the troubles that such a complex and geographically distributed project can run into. Finally, it approaches the challenges still to come, with the transition to the ALMA operations plan.

ALMA operation support software and infrastructure

The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least 66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000 m. Each antenna contains instruments capable of receiving radio signals from 31.3 GHz up to 950 GHz. These signals are correlated inside a Correlator and the spectral data are finally saved into the Archive system together with the observation metadata. This paper describes the progress in the development of the ALMA operation support software, which aims to increase the efficiency of the testing, distribution, deployment and operation of the core observing software. This infrastructure has become critical as the main array software evolves during the construction phase. In order to support and maintain the core observing software, it is essential to have a mechanism to align and distribute the same version of software packages across all systems. This is achieved rigorously with weekly based regression tests and strict configuration control. A build farm to provide continuous integration and testing in simulation has been established as well. Given the large amount of antennas, it is imperative to have also a monitoring system to allow trend analysis of each component in order to trigger preventive maintenance activities. A challenge for which we are preparing this year consists in testing the whole ALMA software performing complete end-to-end operation, from proposal submission to data distribution to the ALMA Regional Centers. The experience gained during deployment, testing and operation support will be presented.

Exploring No-SQL Alternatives for ALMA Monitoring System

The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least 66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000 m. This paper describes the experience gained after several years working with the monitoring system, which has a strong requirement of collecting and storing up to 150K variables with a highest sampling rate of 20.8 kHz. The original design was built on top of a cluster of relational database server and network attached storage with fiber channel interface. As the number of monitoring points increases with the number of antennas included in the array, the current monitoring system demonstrated to be able to handle the increased data rate in the collection and storage area (only one month of data), but the data query interface showed serious performance degradation. A solution based on no-SQL platform was explored as an alternative to the current long-term storage system. Among several alternatives, mongoDB has been selected. In the data flow, intermediate cache servers based on Redis were introduced to allow faster streaming of the most recently acquired data to web based charts and applications for online data analysis.

Virtualization in network and servers infrastructure to support dynamic system reconfiguration in ALMA

ALMA is the first astronomical project being constructed and operated under industrial approach due to the huge amount of elements involved. In order to achieve the maximum through put during the engineering and scientific commissioning phase, several production lines have been established to work in parallel. This decision required modification in the original system architecture in which all the elements are controlled and operated within a unique Standard Test Environment (STE). The advance in the network industry and together with the maturity of virtualization paradigm allows us to provide a solution which can replicate the STE infrastructure without changing their network address definition. This is only possible with Virtual Routing and Forwarding (VRF) and Virtual LAN (VLAN) concepts. The solution allows dynamic reconfiguration of antennas and other hardware across the production lines with minimum time and zero human intervention in the cabling. We also push the virtualization even further, classical rack mount servers are being replaced and consolidated by blade servers. On top of them virtualized server are centrally administrated with VMWare ESX. Hardware costs and system administration effort will be reduced considerably. This mechanism has been established and operated successfully during the last two years. This experience gave us confident to propose a solution to divide the main operation array into subarrays using the same concept which will introduce huge flexibility and efficiency for ALMA operation and eventually may simplify the complexity of ALMA core observing software since there will be no need to deal with subarrays complexity at software level.

A code generation framework for the ALMA common software

Code generation helps in smoothing the learning curve of a complex application framework and in reducing the number of Lines Of Code (LOC) that a developer needs to craft. The ALMA Common Software (ACS) has adopted code generation in specific areas, but we are now exploiting the more comprehensive approach of Model Driven code generation to transform directly an UML Model into a full implementation in the ACS framework. This approach makes it easier for newcomers to grasp the principles of the framework. Moreover, a lower handcrafted LOC reduces the error rate. Additional benefits achieved by model driven code generation are: software reuse, implicit application of design patterns and automatic tests generation. A model driven approach to design makes it also possible using the same model with different frameworks, by generating for different targets. The generation framework presented in this paper uses openArchitectureWare1 as the model to text translator. OpenArchitectureWare provides a powerful functional language that makes this easier to implement the correct mapping of data types, the main difficulty encountered in the translation process. The output is an ACS application readily usable by the developer, including the necessary deployment configuration, thus minimizing any configuration burden during testing. The specific application code is implemented by extending generated classes. Therefore, generated and manually crafted code are kept apart, simplifying the code generation process and aiding the developers by keeping a clean logical separation between the two. Our first results show that code generation improves dramatically the code productivity.

Exploring remote operation for ALMA Observatory

The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least 66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000 m. The observatory has another office located in Santiago of Chile, 1600 km from the Chajnantor plain. In the Atacama desert, the wonderful observing conditions imply precarious living conditions and extremely high operation costs: i.e: flight tickets, hospitality, infrastructure, water, electricity, etc. It is clear that a purely remote operational model is impossible, but we believe that a mixture of remote and local operation scheme would be beneficial to the observatory, not only in reducing the cost but also in increasing the observatory overall efficiency. This paper describes the challenges and experience gained in such experimental proof of the concept. The experiment was performed over the existing 100 Mbps bandwidth, which connects both sites through a third party telecommunication infrastructure. During the experiment, all of the existent capacities of the observing software were validated successfully, although room for improvement was clearly detected. Network virtualization, MPLS configuration, L2TPv3 tunneling, NFS adjustment, operational workstations design are part of the experiment.

A web-based dashboard for the high-level monitoring of ALMA

The ALMA radio-telescope’s operations depend on the availability of high-level, easy-to-understand status information about all of its components. The ALMA Dashboard aims at providing an all-in-one-place near-real-time overview of the observatory’s key elements and figures to both line and senior management. The Dashboard covers a wide range of elements beyond antennas, such as pads, correlator and central local oscillator. Data can be displayed in multiple ways, including: a table view, a compact view fitting on a single screen, a timeline showing detailed information over time, a logbook, a geographical map.

Communication to remote observatories is a science enabler

The environmental conditions that allow optimal astronomical observations are often coupled with sites that are far away from human settlements and of difficult access, implying limited infrastructure availability that translates in excessive costs and limited bandwidth. With the availability and more affordability of optical based technologies, the astronomical scientific community, alone or joining forces with other actors, has managed in the last decade to boost the communication capability available to several of the astronomical installations in the northern Chilean region, the Atacama Desert, and to successfully increase the efficiency and effectivity of the existing Observatories and setting the basis for the coming ones. The paper, after providing a short summary of the projects developed to enable better communications and the future initiatives currently foreseen, focuses on the following show‐cases, from users that differ in size and aims, in served communities and in geographical locations: a) the observation of the First Light from Gravitational Wave Source (ESO, ALMA, et Al.); b) the use of virtual presence to bring the observer where things happen (ESO/PARANAL); c) remote operations for robotic installation (OCA); d) Contributing to develop the local environment (REUNA); e) provide the “muscle” for the current and future data challenge (ALMA). These examples, by illustrating how communication transformed the way research and education are done, demonstrate that improved communication is paramount in achieving better and, in some case, new astonishing results, both in terms of science and as well as enriching the communities, both scientific and in general.

Introducing hardware in the loop and model based simulation concepts in the ALMA observatory for software testing

The Atacama Large Millimeter /sub-millimeter Array (ALMA) has been working in operations phase regime since 2013. The transition to the operations phase has changed the priorities within the observatory, in which, most of the available time will be dedicated to science observations at the expense of technical time required for testing newer version of ALMA software. Therefore, a process to design and implement a new simulation environment, which must be comparable - or at least- be representative of the production environment was started in 2017. Concepts of model in the loop and hardware in the loop were explored. In this paper we review and present the experiences gained and lessons learned during the design and implementation of the new simulation environment.