Toward a Public MAGIC Gamma-Ray Telescope Legacy Data Portal
Michele Doro, Cosimo Nigro, Elisa Prandini, Andrea Tramacere, Manuel Delfino, Jordi Delgado, Elia do Souto, Lea Jouvin, Javier Rico
TToward a Public MAGIC Gamma-Ray TelescopeLegacy Data Portal
M. Doro a , C. Nigro e , E.Prandini a , A. Tramacere b , M. Delfino c , d , J. Delgado c , d , E. doSouto c , L. Jouvin c , J. Rico c for the MAGIC Collaboration ∗ a University of Padova and INFN Padova, I-35131 Padova (Italy) b Department of Astronomy, University of Geneva, Chemin d’Ecogia 16 - 1290 - Versoix -Switzerland e Deutsches Elektronen-Synchrotron (DESY), D-15738 Zeuthen,Germany c Institut de Fisica d’Altes Energies (IFAE), The Barcelona Institute 954 of Science andTechnology (BIST), E-08193 Bellaterra (Barcelona), 955 Spain d also at Port d’Informació Cientifica (PIC) E-08193 Bellaterra (Barcelona) Spain The MAGIC telescopes are one of the three major IACTs (Imaging Atmospheric Cherenkov Tele-scopes) for observation of gamma rays in the TeV regime currently operative. MAGIC functionssince 2003, and has published data from more than 60 sources, mostly blazars. MAGIC alreadyprovides astronomical .fits files with basic final scientific products such as spectral energydistributions, light curves and skymaps from published results. In future, the format of the filescan be complemented with further relevant information to the community: a) by including the fullmulti-wavelength dataset enclosed in a publication, b) providing data in alternative easy-to-useformats such as ASCII or ECSV, which are accessible with other commonly used packages suchas astropy or gammapy . Finally, besides high level products, activities have started to providephoton event lists and instrument response functions in a format such that scientists within andoutside the community are allowed to perform higher level analysis. A second aim is to providea full legacy of MAGIC data. This contribution will illustrate the achievements and plans of thisactivity. ∗ https://magic.mpp.mpg.de/ . For collaboration list see PoS(ICRC2019)1177 c (cid:13) Copyright owned by the author(s) under the terms of the Creative CommonsAttribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). http://pos.sissa.it/ a r X i v : . [ a s t r o - ph . I M ] S e p AGIC data portal
1. The MAGIC Dataset
Figure 1:
The MAGIC stereo system composed of two 17 m diameter dishes operating simultane-ously. Telescopes are located in the Roque de los Muchachos Observatory in the Northern Hemisphere(28 . ◦ N , . ◦ W). Courtesy of D. Lopes (IAC).
The Major Atmospheric Gamma-ray Imaging Cherenkov (MAGIC) telescopes are a pair of17 m-diameter telescopes operating in stereo mode and sensitive to cosmic gamma rays with en-ergies between 0 . ÷
50 TeV [1] (see Fig. 1). The first MAGIC telescope operated in standalonemode from 2003 to 2009, when the construction of a second telescope marked the beginning ofstereoscopic observations. The imaging technique is based on the detection of Cherenkov lightfrom the charged component of the extended atmospheric showers of particles generated in thehigh atmosphere (10 −
20 km a.s.l.) by cosmic gamma rays impinging the Earth (as well as anycharged cosmic ray). The Cherenkov light propagates as a narrow (few ns wide) front of photonstravelling towards the ground that generate an ellipsoidal image on the MAGIC cameras, as shownin Fig. 2.The Cherenkov light developed by the shower results in a compact cluster of triggered photo-multipliers in the pixelised cameras of both telescopes. If a coincidence of such signals is producedin a few ∼ ns window the signal of all the pixels of both cameras is read out and stored. Such in-formation constitutes an event at the detector level. To extract the physical observables (incomingdirection, time, and energy) of the gamma ray producing the event, several algorithms are applied.After subtracting the uniform illumination due to the night sky background ( image cleaning ) theinformation contained in the aforementioned cluster of triggered pixels is parametrised as an el-lipsoidal image [2], the images of the two cameras are then combined ( stereo reconstruction ) for the assignment of of direction and energy , performed with a Random Forest (RF) algorithm anda Monte Carlo look-up table, respectively. The images produced by CR air showers are rejectedthrough a RF classification algorithm [3] that utilizes Hillas parameters and stereoscopic informa-tion returning a single variable “hadronness” (close to 0 for gamma-like events). Each step allowsfor a significant reduction of the data volume. Roughly, of ∼ ∼ AGIC data portal are used for the reduced images (per night, ∼
100 MB per observational run), and just ∼
100 kB ofDL3 data (event lists plus IRFs) per night.
Figure 2:
Basic reconstruction steps for raw MAGIC events.
From the above steps one has a list of events characterised by a time of arrival, a direction inthe sky and an energy. Several selection cuts are further applied to the data, in energy bins, in orderto properly select the signal and background control region.However, in order to reconstruct a photon flux from these events, one need to know the re-sponse of the system: the
Instrument Response Function (IRF) , that contains the effective area,migration matrices between instrument-derived estimated and Monte Carlo matched coordinatesand true energies. IRF components are generated from Monte Carlo simulated events, subject tothe same analysis criteria/cuts optimized for the particular observing condition (zenith and azimuthof the observation, the sky quality and brightness, the mirror reflectivity etc...). The event listaccompained by IRFs is currently dubbed Data Level 3 (DL3) in the IACT community. Ideally,depending the IRF on the observing conditions that continuously change with time, its componentsshould also change for each detected event. As the change is minimal it is usual practice to generateIRFs for the entire observational run.It is then clear that besides an event list (time energy direction) ancillary information mustbe annexed to enable scientists from within and outside the community to reproduce scientificresults with MAGIC data. As an intermediate step to releasing the full event list, an activity that isongoing within MAGIC [4], DL3 MAGIC data were produced for the project in Ref. [4]. MAGICcan share high level products from publications, these comprise e.g. light curves, spectral energydistributions, fits and models, and so on, which can be used for example to build multi-wavelengthsdata collection. The sharing of these products is extremely simplified with respect to the event list,and it is the main subject of this contribution. However, the possibilities to share the informationcontained in a publication are also a matter of high debate in our community regarding the technicalimplementation (format of files, portal server creation, . . . ) and specific information therein. WhileDL3 data have been already produced and shared for projects as the joint-Crab, these proceedingsare concerned with sharing the higher level products, also called Data Level 4 (DL4).Section 2 reports the current status of data sharing activities; Section 3 reports the future plansand Section 4 closes this report. 2
AGIC data portal
2. Status of data sharing
The MAGIC Collaboration currently provides a recollection of shared data accessible fromthe starting page https://magic.mpp.mpg.de/index.php?id=139. Two products are linked:1. High-level FITS files repository: http://vobs.magic.pic.es/fits/2. Low-level open data repository: http://opendata.magic.pic.esThe High-level FITS repository contains a single .fits file for each publication , which againcontains the high-level products mentioned above as well as skymaps, detection plots and variousother pieces of information. The low-level open data contains various lower level events from onlya fraction of selected publications.All of the above represent only an initial step for MAGIC data dissemination, which is goingto significantly improve in the near future in several directions: Disseminate event list (DL3).
Because of a relative large number of background irreducible events,MAGIC shares candidates event list comprising energy, direction and arrival time. Such listis sometimes dubbed Data Level 3 (DL3) format.
Disseminate High Level products (DL4)
MAGIC produces spectral energy distribution and lightcurves which are often used in a much larger multi-wavelength framework. Currently .fits files present MAGIC data alone. This data will be eventually augmented by adding multi-wavelength datasets published in MAGIC papers as well as the possibility of performingqueries via API, for example using a jupyter notebook, as in the case of the astroquery [7]package. We aim at improving this situation by providing a novel format of high-level data.This is the subject of the next section.
3. A novel high-level product file
One of the main focus of our project is the capability to provide high-level products to-gether with all the metadata regarding the multi-wavelength campaings, that usually are presentedin MAGIC papers. In general, multi-wavelength data can be retrieved via front-end-based webqueries, directly accessing the web service, such as those hosted by the Italian Space Agency (ASI)Science Data Center (SSDC) [8]; the European Southern Observatory (ESO) Archive Science Por-tal [10]; The Online Data Analysis ODA) [5] hosted at the University of Geneva; or the Centre deDonnees astronomiques de Strasbourg [11]. Another approach, more suitable for the automatedprocessing and for processing of large datasets, is the access to the data through API, as in the caseof the ODA-API package [6] (see for example this tutorial), or the astroquery package. In this casethe query can be performed using few command lines as in the following example from a s t r o p y . c o o r d i n a t e s i m p o r t
SkyCoord from a s t r o q u e r y . s d s s i m p o r t
SDSS The content of the .fits is specified in a note accessible at the following link:http://vobs.magic.pic.es/fits/mfits/tdas/tdas-fits.pdf AGIC data portal p o s = SkyCoord ( ’ 0h8m05 . 6 3 s +14 d50m23 . 3 s ’ , f r a m e = ’ i c r s ’ )x i d = SDSS . q u e r y _ r e g i o n ( pos , s p e c t r o = T r u e ) p r i n t ( x i d )There are also some TeV-specific web portal for catalogs of sources such as the TeVCAT [13],the TeGeV catalogues [14]. SSDC data for example are exported in ASCII format in the followingformat: −
55 keV ) ( i d = 5 4 ) − −
1) N u f n u _ e r r o r T S t a r t T S t o p − −
50 keV ) ( i d = 5 3 ) − −
1) N u f n u _ e r r o r T S t a r t T S t o p − − −
1) N u f n u _ e r r o r T S t a r t T S t o p − − − Since MAGIC data need some assumptions and ancillary information, the final high-levelproducts cannot be distributed as a single object, as in the case of the SDSS example above, wherethe query is returning a single astropy
Table. In order to explain the details, we take as anexample one specific MAGIC publication AA617(2018)A91 [16] , related to the study of the flaringactivity of the active galaxy NGC 1275 in 2016-2017.In Fig. 3 we report four of five figures of AA617(2018)A91: Fig. 3A a MWL LC, Fig. 3B azoom on this LC, Fig. 3C several MAGIC SEDs for different states, Fig. 3D a MWL SED. The fifthfigure in the paper investigates the correlation between wavelengths and it is not of interest here.Of the above four, cleary the second is a zoom of the first and not of interest. But the remainingthree are of interest. Fig. 3A reports data of MAGIC,
Fermi /LAT and KVA, Fig. 3D of MAGIC and
Fermi /LAT. In the following paragraphs, we try to investigate the formats of the above mentionedinformation that the MAGIC Collaboration may distribute to the community.The format in which high-level data should be distributed in still a matter of debate withinthe TeV community. There is a forum for discussion in github [15], in which specific tag andfields are discussed, however, a consensus is not yet obtained. Our proposal is to store the metadataregarding each paper into a .yaml file, and the spectral/temporal information into astropy tables, using the extended ecsv format. Both these formats, are largely used in the astronomicalcommunity, and easily convertible (eg. .yaml to .json , and .ecsv to .fits ) using wellestablished libraries. 4 AGIC data portal
Figure 3:
Figures of Ref.[16].
As an example, for AA617(2018)A91 we propose to have a .yaml file in the following for-mat:
F i l e n a m e : : m a g i c _ 1 8 f . yamlF i l e _ i n f o :F d a t e = 20190315
F v e r s = 1
Fgen = M i c h e l e Doro , m i c h e l e . doro@unipd . i t
F m a i l = magic_sapo@mpp . mpg . de
F l i n k = XXXX
P a p e r _ i n f o :P r e f : A s t r o n . A s t r o p h y s . 617 ( 2 0 1 8 ) A91P d o i : h t t p s : / / d o i . o r g / 1 0 . 1 0 5 1 / 0 0 0 4 − P c a u t h o r : XX, YY
P a d s : 2018A&A . . . 6 1 7 A . . 9 1M
P i n s p i r e : A n s o l d i : 2 0 1 8 s q g
T a r g e t s i n f i l eTpname : NGC1275
Taname : 3C84 BZUJ0319 +4130 1H0316 +413 4C+ 4 1 . 0 7 . . .
F i l e l i s t MAGIC : m a g i c _ 1 8 f _ l c 1 _ f i g 1 . e c s vm a g i c _ 1 8 f _ l c 2 _ f i g 1 . e c s v AGIC data portal m a g i c _ 1 8 f _ s e d 1 _ f i g 3 . e c s vm a g i c _ 1 8 f _ s e d 2 _ f i g 3 . e c s vm a g i c _ 1 8 f _ s e d 3 _ f i g 3 . e c s vm a g i c _ 1 8 f _ s e d 1 _ f i g 4 . e c s vF i l e l i s t MWL: m a g i c _ 1 8 f _ l c 1 _ f i g 1 _ l a t . e c s vm a g i c _ 1 8 f _ l c 2 _ f i g 1 _ l a t . e c s vm a g i c _ 1 8 f _ s e d 1 _ f i g 4 _ l a t . e c s vF i l e on demands ( a v a i l a b l e on r e q u e s t t o F m a i l ) m a g i c _ 1 8 f _ s e d 1 _ f i g 3 _ f i t . e c s vm a g i c _ 1 8 f _ s e d 2 _ f i g 3 _ f i t . e c s vm a g i c _ 1 8 f _ s e d 3 _ f i g 3 _ f i t . e c s vm a g i c _ 1 8 f _ s e d 1 _ f i g 4 _ f i t . e c s vm a g i c _ 1 8 f _ s e d 1 _ f i g 4 _ m o d e l . e c s vm a g i c _ 1 8 f _ s e d 2 _ f i g 4 _ m o d e l . e c s vComments : None
For example magic_18f_sed1_fig3.ecsv would look like: −−− − { name : en , u n i t : GeV , E n e r g y } − { name : en_wlo , u n i t : GeV , E n e r g y b i n w i d t h low } − { name : en_wup , u n i t : GeV , E n e r g y b i n w i d t h up } − { name : n u f n u , u n i t : TeV cm − − − { name : n u f n u _ e l o , u n i t : TeV cm − − − { name : n u f n u _ e u p , u n i t : TeV cm − − − { name : t s t a r t , u n i t : mjd , MJD s t a r t } − { name : t s t o p , u n i t : mjd , MJD s t o p } − { name : t e x p o , u n i t : h , O b s e r v a t i o n t i m e } − { name : comments , u n i t : l a t e x , Comments } − { F i l e n a m e : m a g i c _ 1 9 e _ s e d _ f i g 1 _ t a r g e t 0 1 . e c s v } − { S o u r c e : TXS0210515 } − { Comments : } en en_wlo en_wup n u f n u n u f n u _ e l o n u f n u _ e u p t s t a r t t s t o p t e x p o comments0 . 1 5 6 7 0 . 0 3 0 8 0 . 3 5 6 1 1 . 5 2 9 e −
10 1 . 4 2 9 e −
10 1 . 7 2 9 e −
10 5 7 6 3 7 . 1 5 7 8 1 1 . 9 630 . 2 4 8 4 0 . 0 4 8 9 0 . 5 6 4 6 4 . 7 1 9 e −
11 4 . 6 1 9 e −
11 4 . 8 1 9 e −
11 5 7 6 3 7 . 1 5 7 8 1 1 . 9 630 . 3 9 3 7 0 . 0 7 7 5 0 . 8 9 4 9 1 . 4 5 9 e −
11 1 . 3 5 9 e −
11 1 . 6 5 9 e −
11 5 7 6 3 7 . 1 5 7 8 1 1 . 9 630 . 6 2 3 9 0 . 1 2 2 6 1 . 4 1 8 2 4 . 6 0 9 e −
12 4 . 5 0 9 e −
12 4 . 8 0 9 e −
12 5 7 6 3 7 . 1 5 7 8 1 1 . 9 630 . 9 8 8 8 0 . 1 9 4 4 2 . 2 4 7 7 1 . 1 3 9 e −
12 1 . 0 3 9 e −
12 1 . 2 3 9 e −
12 5 7 6 3 7 . 1 5 7 8 1 1 . 9 631 . 5 6 7 0 0 . 3 0 8 1 3 . 5 6 2 3 3 . 8 3 9 e −
13 3 . 7 3 9 e −
13 4 . 1 3 9 e −
13 5 7 6 3 7 . 1 5 7 8 1 1 . 9 63
The above example file is significantly based on the format defined for the gamma-cat project [17]. This project aims at collecting all publications from current IACTs, and at being6
AGIC data portal considered a reference for future projects. Our proposal provides a wider dataset that can bestraightforwardly exported into the gamma-cat standards.There are several possibilities to store such files. Considering we would like to provide an API,our server could be based in one of the MAGIC Collaboration institutes, and mirrored elsewhere.The space requirement would be minor. The most critical part of the project is that of iterating allpublished papers and transfer all the high-level products into the proposed format. However, fornovel published papers this would not constitute a problem.
4. Discussion and conclusions
We believe that providing high-level product data from MAGIC publications may be of interestto a wider astronomical community. We want to facilitate the access to these data by building aneasily accessible dataset for all MAGIC published paper as well as papers to come. This papercatalogue will be released in 2019 and will be filled constantly.Besides this effort, the MAGIC Collaboration is also working on the dissemination of the dataat DL3-level, with high level data products relative to individual observations. A working group isactive in MAGIC and results are expected soon.
Acknowledgement
This proposal comes from discussion with several people, specially those at-tending the ASTERICS-OBELICS PyGamma19 meeting in Heidelberghttps://indico.cern.ch/event/783425/overview in particular C. Boisson, C. Deil, G. Maier and R. Zanin.This project has received funding from the European Union’s Horizon2020 research and innovationprogramme under the Marie Sklodowska-Curie grant agreement no 664931. The MAGIC Collab-oration acknowledges support of institutes as inhttps://magic.mpp.mpg.de/acknowledgments_ICRC2019/.
References [1] J. Aleksi´c, et al. [MAGIC collaboration], Astropart. Phys. (2016) 76doi:10.1016/j.astropartphys.2015.02.005 [arXiv:1409.5594 [astro-ph.IM]].[2] Hillas, A. M. 1985, International Cosmic Ray Conference, 3,[3] J. Albert et al. [MAGIC Collaboration], Nucl. Instrum. Meth. A (2008) 424doi:10.1016/j.nima.2007.11.068 [arXiv:0709.3719 [astro-ph]].[4] C. Nigro et al. , Astron. Astrophys. , A10 (2019) doi:10.1051/0004-6361/201834938[arXiv:1903.06621 [astro-ph.HE]].[5] ODA: [6] ODA-API: https://oda-api.readthedocs.io/en/latest/index.html [7] astroquery: https://astroquery.readthedocs.io/en/latest/ [8] SSDC: [9] https://github.com/cta-observatory/ctapipe [10] SSDC: ESP: archive.eso.org AGIC data portal [11] CDS: http://cds.u-strasbg.fr/[12] Astropy: [13] TeVCAT: http://tevcat2.uchicago.edu [14] TeGeV: [15] Gamma Astro Data Format: https://gamma-astro-data-formats.readthedocs.io/ [16] S. Ansoldi et al. [MAGIC Collaboration], Astron. Astrophys. (2018) A91doi:10.1051/0004-6361/201832895 [arXiv:1806.01559 [astro-ph.HE]].[17] Gamma-cat project: https://github.com/gammapy/gamma-cat ..