Using negative detections to estimate source finder reliability
UUsing Negative Detections to Estimate Source Finder Reliability
P. Serra A , D , R. Jurek B and L. Fl¨oer C A Netherlands Institute for Radio Astronomy (ASTRON), Postbus 2, 7990 AA Dwingeloo, TheNetherlands B CSIRO Astronomy & Space Science, Australia Telescope National Facility, P.O. Box 76, EppingNSW 1710, Australia C Argelander-Institut f¨ur Astronomie, Auf dem H¨ugel 71, 53121 Bonn, Germany D Email: [email protected]
Abstract
We describe a simple method to determine the reliability of source finders based on the detection ofsources with both positive and negative total flux. Under the assumption that the noise is symmetricand that real sources have positive total flux, negative detections can be used to assign to each positivedetection a probability of being real. We discuss this method in the context of upcoming, interferometric
H i surveys.
Keywords: methods: data analysis
In the coming years, a number of interferometric neutral-hydrogen (
H i ) surveys will begin (e.g. Koribalski &Staveley-Smith 2009; Verheijen et al. 2009). They willobserve
H i within large cosmic volumes and detecttens of thousands of sources, many of which will beresolved both on the sky and in velocity.These surveys will rely on automated source findersto detect objects present in the data. In particular, thedetection of the faintest objects will require detectioncriteria close to the noise level. However, as fainterand fainter “true” objects are detected an increasingnumber of “false” detections will be included in thesource catalogue (i.e., detections that are, in fact, noisepeaks). Quantifying this effect is crucial to enable aproper scientific exploitation of the final
H i catalogues.A quantity often used for this purpose is the relia-bility R of a source catalogue. This is defined as: R = TT + F , (1)where T and F are the number of true and false de-tections, respectively. Normally, the price to pay fordetecting faint objects is a decrease in R .In some cases, a single value of R may be used tocharacterise an entire source catalogue. However, it ismore informative to study the n -dimensional function R ( p , p , ..., p n ) where the p i ’s are a set of source pa-rameters. For example, R may be given as a functionof objects’ total flux and line width.There are many ways of measuring R . Zwaan et al.(2004) estimate the reliability of the HIPASS catalogue(Meyer et al. 2004) as a function of source total flux,peak flux and line-width by re-observing a sub-sampleof the detected objects. They label confirmed detec-tions as true and non-confirmed detection as false, andadopt a formalism equivalent to Eq. 1 to estimate R . Unfortunately, this empirical procedure may notalways be practical and it requires sources to be re-observed with at least the same data quality of theoriginal observations.Another technique is to create a dataset wheremodel sources are injected on top of modelled (or ob-served) noise and run a source finder as one woulddo with the real data (e.g. Kim et al. 2007; Saintonge2007). Detections corresponding to an input sourceare labelled as true and detections not correspondingto an input source are labelled as false. This approachgives a correct estimate of R only if the model noiseis a good approximation of the real noise and if modelsources are representative of the objects actually con-tained in the data.Here we discuss yet another method to measure R based on the detection of “negative” sources, i.e.,sources with negative total flux. This technique hasbeen used in various forms by several authors workingin different fields (e.g. Dickinson et al. 2004; Yan &Windhorst 2004; Kovaˇc et al. 2009). In this paper wedevelop it further with the aim of making it useful forfuture H i surveys.The main idea is to assume that true sources are“positive” (i.e., they have positive total flux) and thatthe noise is symmetric (we discuss the applicability ofthese assumptions in Section 5). It follows that thenumber of false positive detections equals the numberof negative detections. The reliability can then be de-fined as: R = P − NP , (2)where P and N are the number of positive and nega-tive detections, respectively. It is trivial to verify thatEq. 2 is equivalent to Eq. 1 under the aforementionedassumptions.The advantage of this method is that R is measured a r X i v : . [ a s t r o - ph . I M ] D ec Publications of the Astronomical Society of Australia directly from the data with no additional observationalor modelling effort. In what follows we demonstratethis technique by applying it to a test
H i cube. Wedescribe the cube and the source finder used for thispurpose in Sections 2 and 3, respectively. In Section4 we illustrate the results. In Section 5 we discusspossible caveats and improvements of this technique.We draw conclusions in Section 6.
We test the negative-source method on a data cubewhich is the sum of a noise cube and cubes containingonly
H i sources. We build the noise cube by imag-ing in Stokes Q the continuum-subtracted visibilitydata obtained from a WSRT observation of the galaxyNGC 3941 (Serra et al. 2011).
H i signal is unpolarisedso the Stokes Q cube contains only noise (and imag-ing artefacts). We Fourier transform the visibilitiesusing uniform weighting and 30-arcsec FWHM taper-ing. The resulting Gaussian beam has a FWHM of ∼ ×
30 arcsec . The noise cube covers a sky areaof 1 deg and the recessional velocity range ∼ ∼ − ( z ∼ . z of galax-ies detected by the WALLABY survey is expected tobe ∼ .
03, see Koribalski & Staveley-Smith 2009). Thechannel width is ∼ . − , and we scale the cubeto obtain a root-mean-square (r.m.s.) noise level of 1.6mJy beam − , as per WALLABY specifications.We add ∼ H i cubes available in the WHISPdatabase (van der Hulst 2002) to the noise cube. Todo so we make use of the cubes’ clean componentsderived as part of the WHISP data reduction. Eachset of clean components representing an observed fieldis randomly redshifted within the z range covered bythe noise cube (using a triangular parent distributionfor z ), convolved with a ∼ ×
30 arcsec Gaussianbeam, and placed at a random sky position within thenoise cube (in a few cases this results in a position closeto the edge of the cube). We note that some WHISPcubes contain more than one
H i source, so the numberof input
H i sources is slightly larger than the numberof WHISP cubes used.This data cube is also used by Jurek (2011) to de-velop and refine the CNHI source finder and by West-meier et al. (2011) to test the Duchamp source finder.Compared to other test data cubes discussed in this is-sue (e.g. Popping et al. 2011), this cube has the advan-tage of including real interferometer noise (and there-fore imaging artefacts) and real
H i sources. For exam-ple, the left panel in Figure 1 shows a right ascension-velocity plane of the noise cube. Imaging artefacts arevisible as vertical stripes on this projection. The rightpanel in the figure shows that the distribution of vol-umetric pixel (voxel) values is Gaussian.
We look for objects in the test data cube by running amodified version of the
H i source-finder used in Serraet al. (2011). This finder smooths the data with avariety of kernels and, for each smoothed version of the cube, detects signal above a specified threshold. In thisway we attempt to optimise the signal-to-noise ratio ofobjects present in the data using a limited number offilters. In practice, we look for sources in the original
H i cube and in the cubes obtained by smoothing theoriginal cube either on the sky, or in velocity, or alongall three axes. In this study we use a Gaussian filterof FWHM=60 arcsec for smoothing on the sky, anda box filter of width 2, 4, 8, 16, and 32 channels forsmoothing in velocity.For each smoothed version of the cube we build amask including all voxels brighter (in absolute value)than 4 σ , where σ is the r.m.s. noise in that cube.The final mask is the sum of all masks (i.e., a voxelis included in the total mask if it is included in atleast one of the individual masks). We size-filter themask by performing morphological opening with the scipy.ndimage Python package. We perform the open-ing using a 3 × × × × H i disc model sources. Here we makeuse of the positive and negative source catalogue toestimate the reliability R as a function of source pa-rameters, and demonstrate how R can be used to selectsamples of true detections. The top panels of Figure 2 show the distribution of pos-itive (blue) and negative (red) detections on three pro-jections on the parameter space defined by source totalflux F tot , peak flux F max , and number of voxels N vox1 .Positive detections are shown again in the middle pan-els of Figure 2, where black circles and grey crossesrepresent true and false detections, respectively. A de-tection is labeled true if its mask has non-zero overlapwith an input source in the cube. Input sources are de-fined taking all voxels brighter than 0.16 mJy beam − in the noise-less cube (1/10 of the noise level – see Sec-tion 2) and merging them as in Section 3. This resultsin 137 input sources.We find 303 positive detections. Of these, 63 aretrue. We have verified that undetected input sourcesare too faint and occupy a different region of parameter For negative detections F tot and F max are obtained af-ter multiplying all voxels by −
1. Both F tot and F max aregiven in Jy beam − . ww.publish.csiro.au/journals/pasa space than detected ones. Figure 2 shows that, a poste-riori , it would be easy to define a criterion to efficientlyseparate true from false detections for this particularcombination of data cube and source finder. For ex-ample, all 41 detections with log F tot > − . F max > − . a priori knowledge about thesources and can therefore be applied to any observeddata cube.We compute the density field of positive and nega-tive detections by convolving their distribution shownin Figure 2 with a Gaussian kernel of width σ = 0 . F tot , F max , N vox ) space (we comment on the ker-nel choice below). We use the density fields to calculatethe value of P and N at the location of each detectedsource, and apply Eq. 2 to estimate the reliability R at that location.The bottom panels of Figure 2 show the same dis-tribution of points as in the middle panels, but fordetections with R > .
99 only. Blue and red contoursrepresent constant-surface-density contours of positiveand negative sources, respectively. The figure showsthat red and blue contours lay on top of each otherin the noise-dominated region of the parameter space.Deviations occur in regions hosting true detections.We find 41 detections with
R > .
99. Of these, 40are true, in excellent agreement with the a posteri-ori selection mentioned above. In fact, the only falsedetection (grey cross in the bottom panels) could bediscarded based on its position in the parameter space.We note that the choice of kernel made for theabove calculation can influence the result of our anal-ysis. A larger sample of detections would allow us touse a smaller kernel and, therefore, sample the func-tion R ( F tot , F max , N vox ) in a finer way. We attempt tomake an objective choice of the kernel as follows.We study the quantity P − N estimated from pos-itive and negative density fields at the location of neg-ative detections. We assume that the noise dominatesat these locations, so that the majority of detectionsare false. Given our assumptions (Section 1), it thenfollows that P = N . Assuming that P and N follow aPoissonian distribution the quantity ( P − N ) / √ P + N should follow a Skellam distribution centred on zeroand with variance 1 (Irwin 1937). However, for smallkernels, the guaranteed presence of a negative sourcepushes the distribution to negative values. Only whenthe kernel is sufficiently large does the mean of thedistribution move towards the expected value of zero.We therefore choose the smallest kernel which resultsin a P − N distribution centred on zero. This method works under two basic assumptions: thattrue sources have positive flux and that the noise issymmetric (i.e., its distribution and morphology aresymmetric about zero). The first assumption is not satisfied by data cubes where
H i absorption systemsare also present. However, absorption is only detectableat the location of sufficiently bright continuum sources.Therefore, we believe that these systems could be eas-ily excluded from an analysis like that presented inSection 4.Deviations from noise symmetry may be a moreserious issue. Real data can be thought as a super-position of
H i sources, perfect interferometer noise,and imaging artefacts resulting from faulty calibration,continuum subtraction, cleaning of bright sources andRFI removal. Such artefacts may represent a challengefor this method. The data cube analysed in Section4 contains real WSRT noise and includes some mi-nor artefacts such as stripes visible in right ascension-velocity and declination-velocity projections (see Fig-ure 1). However, it is a relatively clean case and doesnot allow us to assess the impact of imaging artefactson the negative-source method.To test the impact of RFI we analyse an
H i cubewhere RFI is present on short baselines. This cubeis derived from a WSRT observation of NGC 3665taken by Serra et al. (2011). Previous analysis hasshown that no
H i emission is present in this cube.We run the same source finder described in Section 3with the same settings, and perform the same anal-ysis discussed in Section 4. The only difference isthat we now use a Gaussian kernel of width σ = 0 . F tot , F max , N vox ) space. This is the smallest kernel forwhich the mean of all P − N values at the location ofnegative sources equals zero (see Section 4).The result is shown in Figure 3. Only 2 detectionshave R > .
99 (grey crosses). These are very large onthe sky and their moment-0 image shows clearly thatthey are artefacts. We conclude that the method dis-cussed here gives satisfactory results also in this partic-ular case of RFI-contaminated data. The reason whyour method may be able to deal with imaging artefactsis that they are usually symmetric in interferometricimages, so that the method incorporates them as extranoise (in fact, the total flux of an interferometric dirtyimage is always zero because of the lack of data at zerospacing). A more thorough investigation of this aspectis beyond the scope of this paper and requires the anal-ysis of a large number of data cubes including varioustypes of imaging artefacts (e.g., RFI on different base-lines and timescales, cleaning residual) in presence oftrue
H i sources.This technique can be improved by working on amore appropriate parameter space. For example, wehave characterised detected sources with the number ofvoxels they occupy. We could however consider moreparameters describing the shape of a source. For ex-ample, the number of channels occupied by the source,and the major-to-minor axis ratio of the moment-0 im-age of the source. These parameters may be useful toseparate spurious detections caused by imaging arte-facts (which, for example, may be very elongated) fromreal sources. Analysis including more parameters willbe possible with datasets larger than the one analysedhere.
Publications of the Astronomical Society of Australia
We discuss a method to determine the reliability ofsources detected in
H i cubes. We assume that truesources are positive and that the noise is symmetric.It follows that the number of negative detections equalsthe number of positive false detections. Negative de-tections can therefore be used to estimate the relia-bility R of positive detections as a function of theirposition in a chosen source parameter space.We demonstrate this method by running a smooth-and-clip source finder on a test H i cube containing realinterferometer noise and real
H i sources. We showthat sources with
R > .
99 are true. The volumeof parameter space where this simple method gives
R > .
99 is essentially the same which we would haveselected knowing which source is true and which falsein this test cube.We discuss the applicability of this method to
H i cubes with artefacts. We show that at least in theanalysed case of a cube with RFI the method performswell. The reason is that artefacts in interferometricimages tend to be both positive and negative, so thatthey do not necessarily invalidate the noise symmetryassumption.
Acknowledgments
The authors would like to thank Gyula J´ozsa for usefuldiscussion and Tom Oosterloo for a critical reading ofthe paper before submission.
References ww.publish.csiro.au/journals/pasa c h a nn e l − . − . − . − . − . . . . . . . v a l u e ( m J y / b e a m ) − − nu m b e r o f v o x e l s Figure 1: Properties of the test noise cube.
Left:
Right ascension-velocity slice going through the centreof the cube. Artefacts are visible as faint vertical stripes.
Right:
Histogram of voxel values for the entirecube (black line) and a Gaussian distribution with σ = 1 . − (red line). . . . . . . . . N vox − . − . − . − . − . . . . . . l og F t o t . . . . . . . . N vox − . − . − . − . − . − . − . − . l og F m a x − . − . − . − . − . . . . . . F tot − . − . − . − . − . − . − . − . l og F m a x . . . . . . . . N vox − . − . − . − . − . . . . . . l og F t o t . . . . . . . . N vox − . − . − . − . − . − . − . − . l og F m a x − . − . − . − . − . . . . . . F tot − . − . − . − . − . − . − . − . l og F m a x . . . . . . . . N vox − . − . − . − . − . . . . . . l og F t o t . . . . . . . . N vox − . − . − . − . − . − . − . − . l og F m a x − . − . − . − . − . . . . . . F tot − . − . − . − . − . − . − . − . l og F m a x Figure 2: Distribution of detections in the test
H i cube on all projections of the adopted parameter space(see text).
Top : Positive (blue) and negative (red) detections.
Middle : True (black circles) and false (greycrosses) positive detections.
Bottom : Same as middle panels, but showing detections with
R > .
99 only.We also show constant surface-density contours of positive (blue) and negative (red) detections estimatedfrom the distributions in the top panels as described in the text.
Publications of the Astronomical Society of Australia . . . . . . . . . N vox − − − l og F t o t . . . . . . . . . N vox − . − . − . − . − . − . − . − . l og F m a x − − − F tot − . − . − . − . − . − . − . − . l og F m a x Figure 3: Constant surface-density contours of positive (blue) and negative (red) detections for the dat-acube with RFI. Grey crosses indicate sources with
R > ..