Kian-Tat Lim

A demonstration of SciDB: a science-oriented DBMS

In CIDR 2009, we presented a collection of requirements for SciDB, a DBMS that would meet the needs of scientific users. These included a nested-array data model, science-specific operations such as regrid, and support for uncertainty, lineage, and named versions. In this paper, we present an overview of SciDBs key features and outline a demonstration of the first version of SciDB on data and operations from one of our lighthouse users, the Large Synoptic Survey Telescope (LSST).

The Hyper Suprime-Cam software pipeline

In this paper, we describe the optical imaging data processing pipeline developed for the Subaru Telescopes Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey Telescopes Data Management system, adding customizations for HSC, large-scale processing capabilities, and novel algorithms that have since been reincorporated into the LSST codebase. While designed primarily to reduce HSC Subaru Strategic Program (SSP) data, it is also the recommended pipeline for reducing general-observer HSC data. The HSC pipeline includes high level processing steps that generate coadded images and science-ready catalogs as well as low-level detrending and image characterizations.

Report from the 3rd Workshop on Extremely Large Databases

Industrial and scientific datasets have been growing enormously in size and complexity in recent years. The largest transactional databases and data warehouses can no longer be hosted cost-effectively in off-the-shelf commercial database management systems. There are other forums for discussing databases and data warehouses, but they typically deal with problems occurring at smaller scales and do not always focus on practical solutions or influencing DBMS vendors. Given the relatively small (but highly influential and growing) number of users with these databases and the relatively small number of opportunities to exchange practical information related to DBMSes at extremely large scale, a workshop on extremely large databases was organized. This paper is the final report of the discussions and activities at the workshop.

LSST control software component design

Construction of the Large Synoptic Survey Telescope system involves several different organizations, a situation that poses many challenges at the time of the software integration of the components. To ensure commonality for the purposes of usability, maintainability, and robustness, the LSST software teams have agreed to the following for system software components: a summary state machine, a manner of managing settings, a flexible solution to specify controller/controllee relationships reliably as needed, and a paradigm for responding to and communicating alarms. This paper describes these agreed solutions and the factors that motivated these.

Investigating interoperability of the LSST data management software stack with Astropy

The Large Synoptic Survey Telescope (LSST) will be an 8.4m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and is based on a C++ core with a Python control layer. The software consists of nearly a quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution. The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will expect it to be fully interoperable with Astropy. In this paper we describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affliated packages.

Data management cyberinfrastructure for the Large Synoptic Survey Telescope

The Large Synoptic Survey Telescope (LSST) project is a proposed large-aperture, wide-field, ground-based telescope that will survey half the sky every few nights in six optical bands. LSST will produce a data set suitable for answering a wide range of pressing questions in astrophysics, cosmology, and fundamental physics. The 8.4-meter telescope will be located in the Andes mountains near La Serena, Chile. The 3.2 Gpixel camera will take 6.4 GB images every 15 seconds, resulting in 15 TB of new raw image data per night. An estimated 2 million transient alerts per night will be generated within 60 seconds of when the camera’s shutter closes. Processing such a large volume of data, converting the raw images into a faithful representation of the universe, automated data quality assessment, automated discovery of moving or transient sources, and archiving the results in useful form for a broad community of users is a major challenge. We present an overview of the planned computing infrastructure for LSST. The cyberinfrastructure required to support the movement, storing, processing, and serving of hundreds of petabytes of image and database data is described. We also review the sizing model that was developed to estimate the hardware requirements to support this environment beginning during project construction and continuing throughout the 10 years of operations.