MeerKATHI -- an end-to-end data reduction pipeline for MeerKAT and other radio telescopes
Gyula I. G. Józsa, Sarah V. White, Kshitij Thorat, Oleg M. Smirnov, Paolo Serra, Mpati Ramatsoku, Athanaseus J. T. Ramaila, Simon J. Perkins, Dániel Cs. Molnár, Sphesihle Makhathini, Filippo M. Maccagni, Dane Kleiner, Peter Kamphuis, Benjamin V. Hugo, W. J. G. de Blok, Lexy A. L. Andati
aa r X i v : . [ a s t r o - ph . I M ] J un MeerKATHI - an end-to-end data reduction pipeline for MeerKATand other radio telescopes
MeerKATHI collaborationGyula I. G. Józsa , , (delegate), Sarah V. White , , Kshitij Thorat , , , Oleg M.Smirnov , , Paolo Serra , Mpati Ramatsoku , , Athanaseus J. T. Ramaila ,Simon J. Perkins , Dániel Cs. Molnár , Sphesihle Makhathini , Filippo M.Maccagni , Dane Kleiner , Peter Kamphuis , Benjamin V. Hugo , , W. J. G. deBlok , , , and Lexy A. L. Andati South African Radio Astronomy Observatory, Cape Town, Western Cape,South Africa Department of Physics and Electronics, Rhodes University, Makhanda,Eastern Cape, South Africa Argelander-Institut für Astronomie, Auf dem Hügel 71, Bonn, Germany Department of Physics, University of Pretoria, Pretoria, Gauteng, SouthAfrica INAF - Osservatorio Astronomico di Cagliari, Selargius, Cagliari, Italy Ruhr-Universität Bochum, Faculty of Physics and Astronomy, AstronomicalInstitute, Bochum, Germany Netherlands Institute for Radio Astronomy (ASTRON), Dwingeloo, TheNetherlands Dept. of Astronomy, Univ. of Cape Town, Cape Town, Western Cape, SouthAfrica Kapteyn Astronomical Institute, University of Groningen, Groningen, TheNetherlands
Abstract. M eer KATHI is the current development name for a radio-interferometricdata reduction pipeline, assembled by an international collaboration. We create a pub-licly available end-to-end continuum- and line imaging pipeline for MeerKAT and otherradio telescopes. We implement advanced techniques that are suitable for producinghigh-dynamic-range continuum images and spectroscopic data cubes. Using container-ization, our pipeline is platform-independent. Furthermore, we are applying a stan-dardized approach for using a number of di ff erent of advanced software suites, partlydeveloped within our group. We aim to use distributed computing approaches through-out our pipeline to enable the user to reduce larger data sets like those provided by radiotelescopes such as MeerKAT. The pipeline also delivers a set of imaging quality metricsthat give the user the opportunity to e ffi ciently assess the data quality. eer KATHI collaboration
1. Introduction
The data rate and the size of radio-interferometric data sets has increased to such a levelthat it is impossible to apply very interactive data reduction strategies, in which the userintervenes frequently in the data reduction process. Instead, automated pipelines arebecoming a requirement, in which all data reduction steps (i.e. editing, calibration,imaging, and data quality assessment) are integrated in a single pipeline, with auto-mated data analysis and archiving of data products being a potential extension. Such apipeline should ideally be able to process the data in data reduction loops, which willbe controlled by an automated data assessment, reducing the necessity for interventionby the user to a minimum.The usage of a data reduction pipeline is often restricted to a specific instrument ora certain software package. While this approach eases the implementation, it is in manycases not the best solution. Optimally, a radio-interferometric data reduction pipelinewould make use of the most recent software, based on the most modern techniques,and work regardless of the radiointerferometer providing the data or the origin of thesoftware.This is the goal of our international M eer
KATHI collaboration in building theM eer
KATHI pipeline. While M eer
KATHI originates from an e ff ort to create an imag-ing pipeline suited for the reduction of data generated by the MeerKAT radio telescope,for the purpose of H i imaging, the focus shifted from this more-restricted case to itsapplication as a generic pipeline. This empowers the user to automatically reduce datafrom instruments similar to MeerKAT (like the Very Large Array, VLA, or the GiantMetrewave Radio Telescope, GMRT). In the following, we describe how the pipelineis implemented, its current scope, first results using the pipeline, and future plans.
2. Architecture M eer KATHI is a (large) Python script making use of S timela (the IsiZulu word fora train), a platform-independent radio interferometry scripting framework based onPython and a choice of Linux containerization technologies. It enables the user touse a suite of astronomical software to reduce radio astronomical imaging data. Bothsoftware and containerization packages are listed in Table 1. In addition to container-ized (and therefore) platform-independent versions of the software, it provides a stan-dard syntax, allowing the user to access all implemented software in a similar man-ner. Stimela supports the containerization platforms P odman , D ocker , S ingularity ,and u D ocker .S timela provides access to standard software as CASA (McMullin et al. 2007,generic data reduction and analysis), P y BDSM (Mohan & Ra ff erty 2015, see also vanWeeren et al. 2012, source finding), AOF lagger (O ff ringa 2010, radio interference de-tection and elimination), M eq T rees (Noordam & Smirnov 2010, data simulation andcalibration), WSC lean (O ff ringa et al. 2014, imaging and deconvolution), and S o F i A(Serra et al. 2015, source finding). In addition, it wraps high-end software developedwithin our group, using the same standard syntax: • CUBI cal (Kenyon et al. 2018, prediction and calibration) • AI m FA s T (Diagnostics / flow control) • RFI nder (RFI visualisation) • R adio PADRE (Results examiner)eerKATHI 3 • RAG a V i (Data visualisation) • T ricolour (Parallel flagger) • S unblocker (Solar RFI mitigation) • C rystalball (Parallel predict) • SHARP ener (Spectral analysis) • C odex A fricanus (parallel radio software API) Software repositoriesSoftware Repository ReferenceM eer
KATHI https://meerkathi.readthedocs.io AI m FA s T https://github.com/Athanaseus/aimfast/ C odex A fricanus https://github.com/ska-sa/codex-africanus/ C rystalball https://github.com/paoloserra/crystalball/ CUBI cal https://github.com/ratt-ru/CubiCal/ (1)M eq T rees http://meqtrees.net/ (2)R adio PADRE https://github.com/ratt-ru/radiopadre/
RAG a V i https://github.com/ratt-ru/ragavi/ RFI nder https://github.com/Fil8/RFInder/
SHARP ener https://github.com/Fil8/SHARPener/ S o F i A https://github.com/SoFiA-Admin/SoFiA/ (3)S unblocker https://github.com/gigjozsa/sunblocker/ T ricolour https://github.com/ska-sa/tricolour/ AOF lagger https://sourceforge.net/p/aoflagger/wiki/Home/ (4)CASA https://casa.nrao.edu/ (5)P y BDSM https://github.com/lofar-astron/PyBDSF/ (6)WSC lean https://github.com/lofar-astron/PyBDSF/ (7)D ocker P odman https://podman.io S ingularity https://sylabs.io/singularity/ u D ocker https://github.com/indigo-dc/udocker/ Table 1. Repositories of software used in M eer
KATHI. Top: software with con-tributions by M eer
KATHI members. Middle: other radio-astronomical software.Bottom: containerization software. References: (1) Kenyon et al. (2018), (2) Noor-dam & Smirnov (2010), (3) Serra et al. (2015), (4) O ff ringa (2010), (5) McMullinet al. (2007), (6) Mohan & Ra ff erty (2015), (7) O ff ringa et al. (2014).
3. Work flow M eer KATHI has a flexible layout and gets configured via a configuration file. Its sim-plest function is to flag RFI and to perform cross-calibration (using calibration sources),followed by an iterative series of continuum imaging and self-calibration steps (re-ducing image artifacts), line imaging, and source finding. Automated data ingest (de-pending on the telescope), several stages of source characterization, and archiving areplanned, but not yet implemented. Figure 1 shows an example work flow. A suite offast, interactive, remotely accessible diagnostic tools has been developed to assess dataquality and to identify potential problems. Copyright SDP / RARG, 2018-2019. South African Radio Astronomy Observatory (SARAO)
Józsa representing the M eer
KATHI collaboration
Figure 1. Exemple M eer
KATHI workflow.
4. Status and Outlook
After a development phase of less than two years, M eer
KATHI is able to generatescience-ready output from multiple instruments (MeerKAT, VLA, GMRT – see Serraet al. 2019; Ramatsoku et al. 2019; Michałowski et al. 2019; Józsa et al. 2019; Maccagniet al. 2019). We interpret this as an indication that M eer
KATHI can become a widelyused method to reduce radio-interferometric data. While distributed computing is notcompletely implemented, MeerKATHI is able to process relatively large data sets ofseveral TB. A public release of the software is currently pending verification and doc-umentation, but we give interested users access on a shared-risk policy.Our attempt to create a radio-interferometric data reduction pipeline, which canprocess data from multiple instruments using a large suite of available software, hasso far been successful. With M eer
KATHI we provide a working pipeline, which isnow available upon request, but will be made public in the near future. Further de-velopment will concentrate on improved techniques, distributed computing, automatedingest, analysis, and archiving.
References
Józsa, G. I. G., Thorat, K., & Kamphuis, P. , et al. 2019, submitted to MNRASKenyon, J. S., Smirnov, O. M., Grobler, T. L., & Perkins, S. J. 2018, MNRAS, 478, 2399Maccagni, F. M., Murgia, M., & Serra, P. , et al. 2019, submitted to A&AMcMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, Astronomical DataAnalysis Software and Systems XVI, 376, 127Michałowski, M. J., Kamphuis, P., & Hjorth, J. , et al. 2019, A&A, 627, A106.
Mohan, N., & Ra ff erty, D. 2015, Astrophysics Source Code Library, ascl:1502.007Noordam, J. E., & Smirnov, O. M. 2010, A&A, 524, A61O ff ringa, A. R. 2010, Astrophysics Source Code Library, ascl:1010.017O ff ringa, A. R., McKinley, B., & Hurley-Walker, N. , et al. 2014, MNRAS, 444, 606Ramatsoku, M., Serra, P., & Poggianti, B. M. , et al. 2019, MNRAS, 487, 4580Serra, P., Maccagni, F. M., & Kleiner, D. , et al. 2019, A&A, 628, A122Serra, P., Westmeier, T., & Giese, N. , et al. 2015, MNRAS, 448, 1922van Weeren, R. J., RÃ˝uttgering, H. J. A., & Ra ff erty, D. A. , et al. 2012, A&A, 543, A43 Acknowledgements:
The MeerKATHI team acknowledges support from the Starting Grant679629 "FORNAX" of the European Research Council (ERC), MAECI Grant ZA18GR02 fromthe Italian Ministry of Foreign A ffff