Norman Gray

Learning from 25 years of the extensible N-Dimensional Data Format ☆

The extensible N-Dimensional Data Format (NDF) was designed and developed in the late 1980s to provide a data model suitable for use in a variety of astronomy data processing applications supported by the UK Starlink Project. Starlink applications were used extensively, primarily in the UK astronomical community, and form the basis of a number of advanced data reduction pipelines today. This paper provides an overview of the historical drivers for the development of NDF and the lessons learned from using a defined hierarchical data model for many years in data reduction software, data pipelines and in data acquisition systems.

Can RDB2RDF Tools Feasibily Expose Large Science Archives for Data Integration

Many science archive centres publish very large volumes of image, simulation, and experiment data. In order to integrate and analyse the available data, scientists need to be able to (i) identify and locate all the data relevant to their work; (ii) understand the multiple heterogeneous data models in which the data is published; and (iii) interpret and process the data they retrieve. rdf has been shown to be a generally successful framework within which to perform such data integration work. It can be equally successful in the context of scientific data, if it is demonstrably practical to expose that data as rdf . In this paper we investigate the capabilities of rdf to enable the integration of scientific data sources. Specifically, we discuss the suitability of sparql for expressing scientific queries, and the performance of several triple stores and rdbrdf tools for executing queries over a moderately sized sample of a large astronomical data set. We found that more research and improvements are required into sparql and rdbrdf tools to efficiently expose existing science archives for data integration.

Learning from FITS: Limitations in use in modern astronomical research

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

Microlensing of extended stellar sources

Abstract We investigate the feasibility of reconstructing the radial intensity profile of extended stellar sources by inverting their microlensed light curves. Using a simple, linear, limb darkening law as an illustration, we show that the intensity profile can be accurately determined, at least over the outer part of the stellar disc, with realistic light curve sampling and photometric errors. The principal requirement is that the impact parameter of the lens be less than or equal to the stellar radius. Thus, the analysis of microlensing events provides a powerful method for testing stellar atmosphere models.

Data Management and Preservation Planning for Big Science

‘Big Science’ - that is, science which involves large collaborations with dedicated facilities, and involving large data volumes and multinational investments – is often seen as different when it comes to data management and preservation planning. Big Science handles its data differently from other disciplines and has data management problems that are qualitatively different from other disciplines. In part, these differences arise from the quantities of data involved, but possibly more importantly from the cultural, organisational and technical distinctiveness of these academic cultures. Consequently, the data management systems are typically and rationally bespoke, but this means that the planning for data management and preservation (DMP) must also be bespoke. These differences are such that ‘just read and implement the OAIS specification’ is reasonable Data Management and Preservation (DMP) advice, but this bald prescription can and should be usefully supported by a methodological ‘toolkit’, including overviews, case-studies and costing models to provide guidance on developing best practice in DMP policy and infrastructure for these projects, as well as considering OAIS validation, audit and cost modelling. In this paper, we build on previous work with the LIGO collaboration to consider the role of DMP planning within these big science scenarios, and discuss how to apply current best practice. We discuss the result of the MaRDI-Gross project (Managing Research Data Infrastructures – Big Science), which has been developing a toolkit to provide guidelines on the application of best practice in DMP planning within big science projects. This is targeted primarily at projects’ engineering managers, but intending also to help funders collaborate on DMP plans which satisfy the requirements imposed on them.

Finding the right term: Retrieving and exploring semantic concepts in astronomical vocabularies

Astronomy, like many domains, already has several sets of terminology in general use, referred to as controlled vocabularies. For example, the keywords for tagging journal articles, or the taxonomy of terms used to label image files. These existing vocabularies can be encoded into skos, a W3C proposed recommendation for representing vocabularies on the Semantic Web, so that computer systems can help users to search for and discover resources tagged with vocabulary concepts. However, this requires a search mechanism to go from a user-supplied string to a vocabulary concept. In this paper, we present our experiences in implementing the Vocabulary Explorer, a vocabulary search service based on the Terrier Information Retrieval Platform. We investigate the capabilities of existing document weighting models for identifying the correct vocabulary concept for a query. Due to the highly structured nature of a skos encoded vocabulary, we investigate the effects of term weighting (boosting the score of concepts that match on particular fields of a vocabulary concept), and query expansion. We found that the existing document weighting models provided very high quality results, but these could be improved further with the use of term weighting that makes use of the semantic evidence.

Finding Data Resources in a Virtual Observatory Using SKOS Vocabularies

One problem faced by astronomers using the virtual observatory is finding which of the multitude of data resources is relevant for them. The current tool, VOExplorer, relies on matching searches against tags provided by the resources. This paper shows how skos encoded vocabularies can be used to improve the search results. The techniques are general and applicable to any loose collaborations sharing their resources.

Inversion of polarimetric data from eclipsing binaries

We describe a method for determining the limb polarization and limb darkening of stars in eclipsing binary systems, by inverting photometric and polarimetric light curves. Because of the ill-conditioning of the problem, we use the Backus-Gilbert method to control the resolution and stability of the recovered solution, and to make quantita- tive estimates of the maximum accuracy possible. Using this method we conrm that the limb polarization can in- deed be recovered, and demonstrate this with simulated data, thus determining the level of observational accuracy required to achieve a given accuracy of reconstruction. This allows us to set out an optimal observational strat- egy, and to critcally assess the claimed detection of limb polarization in the Algol system. The use of polarization in stars has been proposed as a diagnostic tool in microlensing surveys by Simmons et al. (1995), and we discuss the extension of this work to the case of microlensing of extended sources.

Automatic reduction of elliptic integrals using Carlson's relations

In a series of papers, B. C. Carlson produced tables of elliptic integrals, evaluating them in terms of easily computed symmetrical functions. using a group of multivariate recurrence relations. These relations are, however, cumbersome to use by hand and, in the absence of a specific reductive algorithm, difficult to use with computer algebra. This paper presents such an algorithm, guaranteed to reduce a general elliptic integral to a set of fundamental ones.

Photography at relativistic speeds

In a photo taken with a camera moving at relativistic speed, the world appears distorted. That much has long been clear, but the details of the distortion were slow to emerge correctly. We recently added relativistic raytracing capability to our custom raytracer, Dr TIM, resulting in unique combinations of capabilities. Here we discuss a few observations. In particular, photos can be sharp only if the shutter is placed correctly. A hypothetical window that changes light-ray direction like a change of inertial frame, when combined with suitable shutter placement, can correct for all relativistic-aberration effects.