A. Alexov

Learning from FITS: Limitations in use in modern astronomical research

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. n nThere are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

Ten years of software sustainability at the Infrared Processing and Analysis Center

This paper presents a case study of an approach to sustainable software architecture that has been successfully applied over a period of 10 years to astronomy software services at the NASA Infrared Processing and Analysis Center (IPAC), Caltech (http://www.ipac.caltech.edu). The approach was developed in response to the need to build and maintain the NASA Infrared Science Archive (http://irsa.ipac.caltech.edu), NASAs archive node for infrared astronomy datasets. When the archive opened for business in 1999 serving only two datasets, it was understood that the holdings would grow rapidly in size and diversity, and consequently in the number of queries and volume of data download. It was also understood that platforms and browsers would be modernized, that user interfaces would need to be replaced and that new functionality outside of the scope of the original specifications would be needed. The changes in scientific functionality over time are largely driven by the archive user community, whose interests are represented by a formal user panel. The approach has been extended to support four more major astronomy archives, which today host data from more than 40 missions and projects, to support a complete modernization of a powerful and unique legacy astronomy application for co-adding survey data, and to support deployment of Montage, a powerful image mosaic engine for astronomy. The approach involves using a component-based architecture, designed from the outset to support sustainability, extensibility and portability. Although successful, the approach demands careful assessment of new and emerging technologies before adopting them, and attention to a disciplined approach to software engineering and maintenance. The paper concludes with a list of best practices for software sustainability that are based on 10 years of experience at IPAC.

LOFAR and HDF5: Toward a new radio data standard

For decades now, scientific data volumes have experienced relentless, exponential growth. As a result, legacy astronomical data formats are straining under a burden not conceived when these formats were first introduced. With future astronomical projects ensuring this trend, ASTRON and the LOFAR project are exploring the use of the Hierarchical Data Format, version 5 (HDF5), for LOFAR radio data encapsulation. Most of LOFAR’s standard data products will be stored using the HDF5 format. In addition, HDF5 analogues for traditional radio data structures such as visibility data and spectral image cubes are also being developed. The HDF5 libraries allow for the construction of distributed, entirely unbounded files. The nature of the HDF5 format further provides the ability to custom design a data encapsulation format, specifying hierarchies, content and attributes. The LOFAR project has designed several data formats that will accommodate and house all LOFAR data products, the primary styles and kinds of which are presented in this paper. With proper development and support, it is hoped that these data formats will be adopted by other astronomical projects as they, too, attempt to grapple with a future filled with mountains of data.

Initial deep LOFAR observations of epoch of reionization windows

Aims. The aim of the LOFAR epoch of reionization (EoR) project is to detect the spectral fluctuations of the redshifted HI 21 cm signal. This signal is weaker by several orders of magnitude than the astrophysical foreground signals and hence, in order to achieve this, very long integrations, accurate calibration for stations and ionosphere and reliable foreground removal are essential. Methods. One of the prospective observing windows for the LOFAR EoR project will be centered at the north celestial pole (NCP). We present results from observations of the NCP window using the LOFAR highband antenna (HBA) array in the frequency range 115 MHz to 163 MHz. The data were obtained in April 2011 during the commissioning phase of LOFAR. We used baselines up to about 30 km. The data was processed using a dedicated processing pipeline which is an enhanced version of the standard LOFAR processing pipeline. Results. With about 3 nights, of 6 h each, effective integration we have achieved a noise level of about 100 mu Jy/PSF in the NCP window. Close to the NCP, the noise level increases to about 180 mu Jy/PSF, mainly due to additional contamination from unsubtracted nearby sources. We estimate that in our best night, we have reached a noise level only a factor of 1.4 above the thermal limit set by the noise from our Galaxy and the receivers. Our continuum images are several times deeper than have been achieved previously using the WSRT and GMRT arrays. We derive an analytical explanation for the excess noise that we believe to be mainly due to sources at large angular separation from the NCP. We present some details of the data processing challenges and how we solved them. Conclusions. Although many LOFAR stations were, at the time of the observations, in a still poorly calibrated state we have seen no artefacts in our images which would prevent us from producing deeper images in much longer integrations on the NCP window which are about to commence. The limitations present in our current results are mainly due to sidelobe noise from the large number of distant sources, as well as errors related to station beam variations and rapid ionospheric phase fluctuations acting on bright sources. We are confident that we can improve our results with refined processing.

Wide-band simultaneous observations of pulsars

Dispersion in the interstellar medium is a well known phenomenon that follows a simple relationship, which has been used to predict the time delay of dispersed radio pulses since the late 1960s. We performed wide-band simultaneous observations of four pulsars with LOFAR (at 40-190 MHz), the 76-m Lovell Telescope (at 1400 MHz) and the Effelsberg 100-m Telescope (at 8000 MHz) to test the accuracy of the dispersion law over a broad frequency range. In this paper we present the results of these observations which show that the dispersion law is accurate to better than 1 part in 100000 across our observing band. We use this fact to constrain some of the properties of the ISM along the line-of-sight and use the lack of any aberration or retardation effects to determine upper limits on emission heights in the pulsar magnetosphere. We also discuss the effect of pulse profile evolution on our observations, and the implications that it could have for precision pulsar timing projects such as the detection of gravitational waves with pulsar timing arrays.