Removing Atmospheric Fringes from Zwicky Transient Facility i-Band Images using Principal Component Analysis
Michael S. Medford, Peter Nugent, Danny Goldstein, Frank J. Masci, Igor Andreoni, Ron Beck, Michael W. Coughlin, Dmitry A. Duev, Ashish A. Mahabal, Reed L. Riddle
DDraft version February 23, 2021
Typeset using L A TEX twocolumn style in AASTeX62
Removing Atmospheric Fringes from Zwicky Transient Facility i-Band Images using Principal Component Analysis
Michael S. Medford,
1, 2
Peter Nugent, Danny Goldstein, Frank J. Masci, Igor Andreoni, Ron Beck, Michael W. Coughlin, Dmitry A. Duev, Ashish A. Mahabal,
6, 7 and Reed L . Riddle Department of Astronomy, University of California, Berkeley, Berkeley, CA 94720 Lawrence Berkeley National Laboratory, 1 Cyclotron Rd., Berkeley, CA 94720 IPAC, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA 91125, USA Division of Physics, Mathematics, and Astronomy, California Institute of Technology, Pasadena, CA 91125, USA School of Physics and Astronomy, University of Minnesota, Minneapolis, Minnesota 55455, USA Division of Physics, Mathematics and Astronomy, California Institute of Technology, Pasadena, CA 91125, USA Center for Data Driven Discovery, California Institute of Technology, Pasadena, CA 91125, USA Caltech Optical Observatories, California Institute of Technology, Pasadena, CA 91125, USA
ABSTRACTThe Zwicky Transient Facility is a time-domain optical survey that has substantially increased ourability to observe and construct massive catalogs of astronomical objects by use of its 47 square degreecamera that can observe in multiple filters. However the telescope’s i-band filter suffers from significantatmospheric fringes that reduce photometric precision, especially for faint sources and in multi-epochco-additions. Here we present a method for constructing models of these atmospheric fringes usingPrincipal Component Analysis that can be used to identify and remove these image artifacts fromscience images. In addition, we present the Uniform Background Indicator as a quantitative measure-ment of correlated background noise and its relationship to reduced photometric error after removingfringes. We conclude by evaluating the effect of our method on measuring faint sources through theinjection and recovery of artificial faint sources in both single-image epochs and co-additions. Ourmethod for constructing atmospheric fringe models, and using those models for cleaning images, isavailable for public download in the open source python package fringez. INTRODUCTIONLarge scale synoptic surveys have made the generationof optical images efficient to obtain and consequentlyubiquitous. Surveys such as the Palomar Transient Fac-tory (Law et al. 2009), Catalina Real-Time TransientSurvey (Drake et al. 2009), Zwicky Transient Facility(Bellm et al. 2019b; Graham et al. 2019; Masci et al.2018) and others have revolutionized our understandingof the universe through massive transient datasets thatrequire advances in computational techniques to processthe ever growing stockpile of data they generate. TheVera C. Rubin Observatory will generate 20 terabytesof data per night, totalling over 500 petabytes of imag-ing data over the 10 years of the survey (Ivezi´c et al.2019). Such data flows necessitate that the reduction ofraw images into science images must be systematic andrequire minimal human intervention, all while removingtransient and significant sources of noise.One such significant source of noise in long wavelengthoptical imaging are atmospheric emission lines. Theseemission lines are brought about by highly non-thermalatomic and molecular transitions (primarily O II andOH) and are influenced by the temperature and density of the upper and lower atmosphere as well as the currentsolar activity. Thus the strength of these lines can varythroughout the night and are also proportional to theairmass through which the telescope is pointed. Fringepatterns appear when these wavelengths fall onto thinCCDs due to the self-interference caused from light re-flecting off of the back of the imaging instrument beforeit is absorbed by the CCD itself Bernstein et al. (2017).Thick CCDs rarely see this effect save at the longestwavelengths. For thin CCDs, these fringe patterns canintroduce significant background noise into i-band andz-band images, rendering photometry and image sub-traction ineffective at faint magnitudes without a cali-bration correction that successfully removes these atmo-spheric fringes. These fringes also appear in interferenceimage spectrometers where attempts have been made toremove them using wavelet transformations (Ren et al.2017).Principal component analysis (PCA) is a statisticalmethod for reducing data to a set of orthogonal compo-nents by finding directions of minimal variance withinthe data (Jolliffe & Cadima 2016). PCA has been shownto be an effective method for modeling and removing a r X i v : . [ a s t r o - ph . I M ] F e b Medford et al. these atmospheric fringes from optically red and near-IR images. These methods work by building a set of or-thogonal component images, extracted from a large sam-ple of representative images, which can approximate thefringes of an image through linear combination. Individ-ual images are dotted against these orthogonal compo-nent images resulting in eigenvalues that, when used asweights to these orthogonal images, generate bias imagesof the atmospheric fringes. These bias images correctthe photometric inaccuracy for an image’s astrophysicalsources caused by fringes which, when removed, increasephotometric precision considerably.Previous PCA methods for atmospheric fringe sub-traction constructed orthogonal images by down-sampling science images into a lower resolution beforere-parameterization of the data. Bernstein et al. (2017)compress an image into a sparse set of features beforeattempting reduction into orthogonal components. Thismethod has had strong results but sacrifices the powerto potentially resolve contributions from individual at-mospheric lines due to the compression. PCA performedon a per-pixel basis is more computationally expensivebut has the power to capture these individual fringeeffects, resulting in more accurate image photometry,subtraction and co-additions.The computational resources dedicated to the reduc-tion and calibration of astronomical images has grownover the decades to keep up with the increasing flow ofdata. The development of highly optimized PCA algo-rithms, along with additional computational resources,have now made it possible to perform per-pixel PCAanalysis on science images for atmospheric fringe mod-eling. Here we present the implementation of such amethod on full resolution Zwicky Transient Facility i-band data. In section 2, we outline the Zwicky Tran-sient Facility instrument and dataset. In Section 3, wepresent our method for implementing per-pixel PCA at-mospheric fringe modeling and removal, as well as theUniform Background Indicator as a quantitative mea-sure of correlated background noise. In Section 4, weanalyze the results of applying our method, includingincreased photometric precision on faint sources and theability to detect otherwise undetectable faint sources inmulti-epoch co-additions. We discuss and conclude inSection 5. THE ZWICKY TRANSIENT FACILITYINSTRUMENTThe Zwicky Transient Facility (ZTF) is an opticaltime-domain survey that has been operating on the 48-inch Samuel Oschin Telescope at Palomar Observatorysince March 2018 (Bellm et al. 2019b; Graham et al. 2019). ZTF’s camera covers 47 square degrees in asingle exposure, enabling coverage of the entire visibleNorthern sky every few nights in ZTF g-band, r-bandand i-band filters with an average 2 . (cid:48)(cid:48) FWHM on aplate scale of 1 . (cid:48)(cid:48) pixel − . The ZTF camera is dividedinto 16 CCDs, each covered with an anti-reflective (AR)coating, with each CCD split into four separate readoutchannels for a total of 64 readout channels (Dekany et al.2020). Surveys for the telescope over its first severalyears of operations (Bellm et al. 2019a) have a standard30 second exposures that achieve median limiting magni-tudes of r ≈ . g ≈ .
25, and i ≈ . METHOD FOR REMOVING ATMOSPHERICFRINGESIn this paper, we remove atmospheric fringes from op-tical images in two steps. First the PCA eigen-vectorsfor a readout-channel are extracted from a large set ofimages with significant fringing. Second each image isprocessed through this model to generate a unique biasimage that, when subtracted from the original image,removes the atmospheric fringes. Figure 1 displays a vi- emoving ZTF Atmospheric Fringes ⭑⭑ ⭑ P C A T r a i n i n g ⭑⭑ ⭑ ⭑⭑ ⭑ Fringe Bias
Subtraction By
Fringe Model
Clip & Scale
Single Image Single Fringe Map
Estimation
Clean Image ⭑⭑ ⭑ ⭑⭑ ⭑⭑⭑ ⭑ ⭑⭑ ⭑ ⭑⭑ ⭑ ⭑⭑ ⭑
Set of Images Set of Fringe Maps
Clip & Scale fringez-generatefringez-clean
Figure 1.
The process for removing atmospheric fringes isdone in two steps. A fringe model for each readout channelis constructed from processing a set of fringe maps throughPCA training, where fringe maps are a clipped and scaledversion of single-epoch images. This step is only performedonce. Cleaning every single-epoch image thereafter is per-formed by using the eigen-vectors in the fringe model togenerate a fringe bias image for each fringed image. Thisfringe bias is subtracted from the science image to createa clean image. Functions for generating fringe models andcleaning fringed images can be found in the open sourcefringez package under the executables fringez-generate and fringez-clean respectively.
ZTF’s reduction pipeline is designed to process each ofthe camera’s 64 readout-channels separately. We there-fore begin by gathering images of a common readout-channel together for model generation, with the goal ofgenerating 64 fringe models. Training images were se-lected by gathering all i-band images between 2019-04-01 and 2020-04-01 and removing images with a limitingmagnitude less than 19. This cut on limiting magnituderemoves cloudy and other images from our sample thatwould fail to have a photometric measurement of thenight sky and thus its atmospheric lines. By includingall images within these dates we ensure a representa-tive sample of atmospheric conditions and airmass whichare correlated with the strength of atmospheric fringes.There are 550,365 i-band images included in total, witheach fringe model trained on between 8,062 and 8,898images.The location and strength of atmospheric fringes areindependent of the astrophysical sources. Thereforethese sources must be removed from each image priorto training in order to reduce the confusion in the PCAvariance reduction process. The preferred method foridentifying pixels containing astrophysical sources is touse the quality masks produced by IPAC that labelspixels containing a source in each image’s source cata- log. In cases where this pixel mask cannot be obtained,pixels containing sources were identified by calculatingthe median absolute deviation for the image and flag-ging pixels that were 5 standard deviations above orbelow the image’s median value. All pixels flagged ascontaining sources were replaced with the value of theimage’s global median, removing stars from our imageswithout changing the value of the vast majority of pix-els. This image must now be transformed to a stan-dard scale so that it can be trained with other imagestaken at different airmass and limiting magnitude. Thisscaling was done by subtracting the median from theimage, and dividing the result by the median absolutedeviation. We call this training image a fringe map,as it traces the relative location and strength of onlythe atmospheric fringes. Each fringe map in the train-ing set was then transformed into a (1 , N col × N row )array simply by flattening the original 2-dimensional ar-ray. A reasonable assumption would be that collaps-ing the spatial correlation present in the 2-dimensionalimage would make our method less effective. Howeverthis concern proved unfounded, as experimentation withfitting sub-images of different sizes resulted in no im-provement on our method. Each pixel in the flattenedfringe map was treated as a separate feature in our PCAanalysis, resulting in 9,461,760 features (3080 rows by3072 columns) for each model. The eigenvalues for eachfeature was determined using the randomized SingularValue Decomposition method (Martinsson et al. (2011),Halko et al. (2011)). Each 1-dimensional eigen-vectorwas reconstructed into the original image shape to cre-ate an eigen-image. The final fringe model is a set ofeigen-images capturing orthogonal contributions to thevariance in each pixel.Individual science images can have their atmosphericfringes removed using the eigen-images in these models.First, a fringe map is generated for the science imageusing the above procedure. This fringe map is dottedagainst the eigen-images of the fringe model and eachcomponent in the vector is divided by the square root ofthat component’s explained variance. The resulting ar-ray of weights is then dotted back into the square root ofthe model’s explained variance multiplied by the com-ponents. This single image, constructed from a linearcombination of the model’s eigen-images, is multipliedby the science image’s median absolute deviation andnamed a fringe bias. A clean image is finally generatedby subtracting the fringe bias from the fringed scienceimage.An example of each of the images in this process areshown in Figure 2. This figure visually demonstrates theresults of our method. The original image is a 90 second Medford et al. i-band exposure that contains a representative amountof atmospheric fringes. The fringe map is almost entirelydevoid of individual sources, although the source pixelidentification method does struggle around particularlybright stars. The fringe bias generated from process-ing the fringe map through the fringe model successfullyidentifies the location and strength of each atmosphericfringe. The clean image has nearly all of the atmosphericfringing pattern removed while retaining most all of theastrophysical sources.
Figure 2.
An example of an i-band image in the vari-ous steps of our fringe removal method. Science images areclipped and scaled to produce a fringe map. The fringe mapis processed through a fringe model to generate a fringe bias.This bias image is subtracted from the original fringed im-age to create a cleaned image. While the fringe model isconstructed only once, each single-epoch image has a uniquefringe bias determined by the linear combination of eigen-images in the fringe model that best reconstructs the fringemap. This ensures that the fringe model can successfullyremove fringes arising from a variety of airmass and seeingconditions.
The arithmetic average of the fringe maps used totrain the fringe model for each readout-channel is shownin Figure 3, with each readout-channel placed on a com-mon gray-scale. The pattern that clearly emerges is onthe level not of the readout-channel but on the CCD,where the fringe patterns connect into large circularpatterns. The etching process for creating a thinnedCCD uses a circular buffer that removes layers of thechip to create a uniformly thick device. The average of the training fringe maps identifies the residual thicknessvariations for each CCD. Work is ongoing to use thesepsuedo-measurements of the thickness to improve theZTF data quality pipeline. The inner 32 readout chan-nels (16 ≤ rcid <
48) and the outer 32 readout channels(0 ≤ rcid <
16, 48 ≤ rcid <
64) have distinctly differentamounts of atmospheric fringing present in their images.The inner readout channels have two layers of AR coat-ing while the top and bottom rows of CCDs have a singlelayer coating. This causes the outer readout channelsto have a higher reflectivity at the longer wavelengthswhere fringing occurs and thus a larger contrast of thefringing pattern.
Figure 3.
The average of the training fringe maps for eachreadout channel’s fringe model on the ZTF camera, all placedonto a common gray-scale. These circular patterns roughlytrace the thickness variations in the CCD and show the cir-cular pattern resulting from flattening the device. The in-ner 32 readout channels show significantly less atmosphericfringing due to an additional layer of anti-reflective coating,as compared to the outer 32 readout channels.
The eigen-images for readout channel 13 are shown inFigure 4 as a representative example of a fringe model.Each pixel in the image is trained as a separate featureand yet the fringe patterns across pixels remain coher-ent after reconstruction into the original image dimen-sions. This confirms that the components are calculat-ing correlated eigen-vectors for the different features. Inaddition to the atmospheric fringes, the PCA trainingappears to be capturing global flux backgrounds that re-main after flat fielding and these backgrounds dominate emoving ZTF Atmospheric Fringes fringez (Michael Medford 2021). The fringez pack-age includes command line executables and pythonfunctions for downloading the fringe models com-puted as a result of this work, producing fringebias images, and cleaning fringed science images.This package is available for download and installa-tion at https://github.com/MichaelMedford/fringez,and all fringe models are available for download at
Figure 4.
Each readout channel has a distinct fringe modelwith six eigen-images constructed from the reduction of thou-sands of training fringe maps. Here is an example of theeigen-images from readout channel 13 (top). While the fringemodels are trained on a 1-dimensional array of pixels, the 2-dimensional fringes remain intact. The fringe pattern of thereadout channel is clearly evident with slight variations inposition and strength amongst the first four eigen-images.However the last two components (and to some extent thefirst four) primarily capture smooth global gradients thatwould ideally be removed by flat fielding. The fractionalexplained variance for each component across all 64 fringemodels (bottom) confirms that the fifth and sixth compo-nents are contributing far less variance to the overall varia-tion, as shown by their relatively small explained variance.The first component captures 64.6% of the pixel variancewhile the sixth component captures only 5.0% of the pixelvariance. In total six PCA components reconstructs 95.0%of the variance seen in our pixel sample. This justifies thechoice of training our models on no more than six compo-nents. https://portal.nersc.gov/project/ptf/iband. In October2019, fringez was implemented into the ZTF IPACdata reduction pipeline with fringe models built on 500training images per readout channel. All i-band imagestaken up to this point were also reprocessed to removeatmospheric fringes. In November 2020, the versionof fringe models used for generating fringe bias imageswere updated to models built on the 550,365 i-bandimages described and investigated in this paper. Thisimplementation of fringez and the current version of
Medford et al. fringe models will continue to be a part of the IPACdata reduction pipeline in ZTF-II. MEASURING IMPROVED PHOTOMETRICPRECISION4.1.
The Uniform Background Indicator
In order to assess the effect of our de-fringing method,we require a quantitative method for measuring the ad-ditional correlated background noise caused by atmo-spheric fringes. We here define the Uniform BackgroundIndicator (UBI) as Ψ:Ψ = std( { F background } )median( { σ background } ) (1)The UBI is calculated by performing aperture pho-tometry at random locations on the background of animage and dividing the standard deviation of the mea-sured background fluxes with the median error in theseflux measurements. For an image of uniform Gaussiannoise, Ψ ≈ > ≈
1, confirming our interpretation that this valueindicates no correlated background noise on an image.After correctly removing the effects of the area weightfor the aperture flux errors, this value is consistent forany size of aperture. The location and scale of the Gaus-sian noise was found to have no effect on this measure-ment. Figure 5 also shows three example images withcorrelated background noise and the UBI values thatthey produce, covering the range of Ψ that we detectedthroughout this analysis. These example images showthe correlation between a quantitatively larger Ψ anda qualitative increase in the appearance of atmosphericfringes.
Figure 5.
The Uniform Background Indicator Ψ equals ap-proximately 1 for a range of aperture sizes on images of Gaus-sian noise (top). This validates interpreting Ψ ≈ Having verified that the UBI is a valid indicator of cor-related background noise, we next measured the effectof removing atmospheric fringes on the UBI. A sam-ple of g-band, r-band and i-band images was created bydownloading one random image for each filter, readout-channel and field observed in the ZUDS survey from theweek of 2020-02-01 to 2020-02-08 with a limiting mag-nitude greater than 19. This sampling method ensureda representative sample of airmass and limiting mag-nitude for ZTF observations, while removing images of emoving ZTF Atmospheric Fringes ≥ .
13 and Ψ =1 .
16, indicating a small but measurable amount of cor-related background noise remaining after flat-fielding.The i-band images contain far more correlated back-ground noise and are notably bimodal, with one popula-tion averaging Ψ = 1 .
33 and a second wider distributionaveraging a larger Ψ = 1 .
91. This split is caused bythe location of the readout channel on the image plane.The inner 32 readout channels (16 ≤ rcid <
48) withtheir additional layer of AR coating are less susceptibleto atmospheric fringing. The outer 32 readout channels(0 ≤ rcid <
16, 48 ≤ rcid <
64) experience significantlymore fringing due to a lack of an additional later of ARcoating. Less than 20% of the images have a Ψ < . ≥ .
20. Thecleaned i-band images show indistinguishable behaviorbetween the inner and outer readout channels forming asingle distribution averaging Ψ = 1 .
15. This populationappears similar to the g-band and r-band populations,indicating a successful removal of atmospheric fringes.However the g-band and r-band images have a signifi-cantly longer tails with an 80th percentile of Ψ = 1 . .
32 respectively, compared to an 80th per-centile for cleaned i-band images of Ψ = 1 .
20. Thisindicates that there is correlated background noise oc-curring in g-band and r-band images that PCA analysiscould potentially model and remove. It is clear that theprocess of removing atmospheric fringes significantly re-duces correlated background noise from i-band images.It is reasonable to predict that UBI will increase withairmass, as the presence of atmospheric fringes is causedfrom the column of atmosphere through which imagesare taken. This is found to be the case in Figure 7,where collecting the images into airmass bins and calcu-lating the median UBI shows a trend toward larger UBIfor larger airmass in i-band images. There is a slightincrease in the g-band and r-band images as well, al-though the effect is relatively weak. The cleaned i-bandimages have comparable UBI values to the g-band andr-band images, again confirming the effectiveness of ourmethod.While the UBI is a useful tool to calculate a metric formeasuring the presence of correlated background noise,
Figure 6.
Our sample of g-band, r-band and i-band im-ages show distinctively different distributions in their Uni-form Background Indicator (UBI) as shown in both a his-togram (top) and cumulative distribution function (bottom).The g-band (green) and r-band (red) average Ψ ≈ .
15 in-dicating a small but measurable amount of correlated back-ground noise after flat fielding. The i-band is split betweentwo populations. The images taken on the inner 32 readoutchannels (light blue) are only moderately affected by atmo-spheric fringing, averaging Ψ = 1 .
33. However the outer32 readout channels (dark blue) are significantly affected bythese fringes, with an average of Ψ = 1 .
91 and less than 20%of the images with Ψ < .
72. The population of cleaned i-band images (yellow) is monomodal and has a median valuesimilar to the g-band and r-band of Ψ = 1 .
15. ZTF i-bandimages processed with our method show similar amounts ofcorrelated background noise as present in g-band and r-bandimages. a quantification of the improvement in photometric pre-cision from this method requires determining how themeasurement of a source is effected by the presence andremoval of atmospheric fringes. We will first determinethe photometric error caused by atmospheric fringes rel-ative to a reference catalog. Next we will measure theadditional photometric scatter when measuring 5-sigmafake sources injected into a single image. Last we willmeasure the recoverability of 0.5-sigma fake sources in-jected into 100 images before a median co-addition isapplied. For each of these experiments we will demon-strate how our method significantly improves the photo-metric precision of faint sources affected by atmosphericfringes. 4.2.
Photometric Error Due to Fringes
First we measure the photometric error caused by at-mospheric fringes. For each of the i-band images in oursample, we used SExtractor (Bertin & Arnouts 1996)to generate an instrumental photometric aperture cata-
Medford et al.
Figure 7.
The Uniform Background Indicator (UBI) corre-lates with airmass for i-band images with significant amountsof atmospheric fringes. Observing sources through the addi-tional column of atmosphere increases the exposure to stim-ulated emission of atmospheric lines that caused the emer-gence of fringes. Cleaning the i-band images removes thiscorrelation and produces a relationship with airmass indis-tinguishable from the g-band and r-band populations. log of astrophysical sources. We then cross-matched theoriginal and cleaned images with Pan-STARRS1 (PS1)(Chambers et al. 2016) i-band catalogs downloaded fromthe Vizier database using astroquery (Ginsburg et al.2019) to create cross-matched catalogs. A zeropoint wascalculated for each image using matching sources withPS1 i-band magnitudes less than 17. This zeropoint wasused to transform the instrumental photometric cata-logs into ZTF i-band magnitudes comparable with PS1i-band magnitudes, as well as to calculate a 5-sigma lim-iting magnitude for each image. After removing imageswith a limiting magnitude less than 21 to ensure reason-able measurement of faint sources, our final sample sizewas 738 images. For each pair of fringed and cleaned im-ages, the cross-matched catalogs were used to find thevariance in the difference between ZTF and PS1 magni-tudes for stars within a magnitude bin. The photometricerror caused by fringes is then calculated as: σ mag = std( m PS1 − m ZTF ) (2) σ fringe = (cid:113) σ , fringed − σ , cleaned (3)The photometric error due to fringing is shown inFigure 8, plotted separately for different faint magni-tude bins against the UBI of the fringed images beforecleaning. Images with larger UBI values have largeramounts of additional magnitude scatter relative to thePanSTARRS catalog than those images with smallerUBI values. Those images with Ψ ≤ . > . . . ≥ .
4, resulting in errors as large as 0.21magnitudes for 18.5 magnitude stars and up to 0.39 mag-nitudes for 19.5 magnitude stars. Images with Ψ ≥ . Figure 8.
Measurement of the photometric error on faintsources due to fringing on 738 high quality i-band images asshown against the Uniform Background Indicator (UBI) ofthe fringed image before cleaning. This error is calculated bycomparing the variance in the difference between ZTF andPS1 magnitudes before and after removing fringes. LargerUBI values correlate with larger amounts of photometric er-ror, getting as large as 0.46 magnitudes for 20.5 magnitudesources and 0.57 magnitudes for 21.5 magnitude sources atΨ = 2 .
0. Atmospheric fringes add a significant systematicerror to the photometry of faint sources that our method isable to remove.
Effects on Fake Sources: Single Epoch
Another way to measure the effect of atmosphericfringes on photometric precision of faint sources isthrough the injection and recovery of fake sources. Weselected an i-band image with Ψ = 1 .
78 as a repre-sentative image. The image’s zeropoint and 5-sigmalimiting magnitude were calculated using a cross-matchto PanSTARRS1 sources as described above. A PSF forthe image was derived using the psfex (Bertin 2011) and emoving ZTF Atmospheric Fringes galsim (Rowe et al. 2015) python package. 100 5-sigma sources were injected into the original image usingthis PSF model at random locations, including Poissonnoise. The image was then cleaned using the fringez package. Photometric catalogs using both aperture andPSF photometry were calculated on the original andcleaned images using
SExtractor . We also calculatedthe ideal aperture corrected flux of the injected fakesources by calculating the median ratio of aperture toPSF fluxes of high signal-to-noise astrophysical sourcesfrom the catalogs, for both fringed and cleaned images.We then repeated this process 50 times for a total of5000 fake sources injected to form our sample. Wealso duplicated this experiment with the
SExtractorBACKGROUND parameter set to
LOCAL and
GLOBAL to testthe effects of forcing the source identification algorithmto attempt to characterize local variations in the back-ground noise.Figure 9 shows the results of this experiment. Thedistributions show the fractional offset of the measuredflux to the injected flux scaled by the theoretical erroron a 5-sigma source. Plots are drawn in log-scale tohighlight the long tail of overestimated measurementsfor images that have not been cleaned, with a Gaussiandistribution drawn in gray as a reference.We first note some observations about the quality ofrecovered fake sources on the images which have notbeen cleaned. Setting the
SExtractor BACKGROUND pa-rameter to
LOCAL has an immediate improvement onmaking correct measurements for both aperture andPSF photometry, reducing the long tail of overestimat-ing the source’s brightness. This long tail is due tosources that fall onto the additive atmospheric fringes.
LOCAL computes the background flux with a rectangularannulus around the source that prevents attributing theadditional brightness in the aperture or PSF model tothe source but instead to the background. PSF photom-etry is exceptionally poor at correctly identifying thisbackground as a PSF model derived from all sourcesacross the image plane will be artificially broadened onan image with fringes. Aperture photometry also showsa long tail toward overestimating the brightness dueto excessively bright local backgrounds for sources onfringes. Switching to a
LOCAL background also reducesthe long tail in this case, showing results that are quiteclose to those in the cleaned catalog. It should be notedthat for all aperture photometry results there are miss-ing sources at the underestimated flux end of the distri-bution, as those sources are shifted to the right due tothe mis-estimation of the background. In the absence ofa method to remove fringes, performing aperture pho- tometry with a
LOCAL background is the best methodfor minimizing the effects of atmospheric fringes.The sources from the cleaned catalogs are more ac-curately and precisely measured under all conditions.There is a slight tail of overestimated fluxes when per-forming PSF photometry but it is far closer to a Gaus-sian distribution than without cleaning. For both
LOCAL and
GLOBAL backgrounds, the distribution of recoveredsources in the cleaned catalogs very nearly resembles aGaussian distribution. There exists only a few sourceswith measured fluxes exceeding the injected flux caus-ing a deficit of fainter sources. Aperture photometryremains the best way to evaluate sources even where at-mospheric fringes have been removed. However the cat-alogs are invariant to the selection of the
SExtractorBACKGROUND parameter. Failure to clean images con-taining atmospheric fringes results in a systematic over-estimation of the flux of faint sources. Applying ourmethod enables near-ideal recovery of 5-sigma sourcesfor a variety of measurement methods.4.4.
Effects on Fake Sources: Multi Epoch
It is particularly difficult to overcome the effects offringes when combining multiple images of the same fieldto recover sources fainter than a single image’s limitingmagnitude. Changing atmospheric conditions and vari-ous observational airmass will alter the strength of theatmospheric fringes, while dithering and sky motion willplace astrophysical sources onto slightly different pix-els for each exposures. Failure to remove atmosphericfringes will contribute significant excess flux to either aco-addition or median filtering of multiple images. Wedemonstrate here this effect quantitatively by injectingextremely faint sources into individual images and at-tempting to measure them in a co-addition.100 i-band images of the same field were zero-pointedusing the previously outlined method and each injectedwith 0.5 sigma sources at a common list of 100 co-ordinates in right ascension and declination. Thesesources should theoretically appear as 5-sigma sourcesafter combining the 100 i-band images as the signal-to-noise increases as the square root of the total ex-posure time. All images were then cleaned with the fringez package.
SCAMP (Bertin 2006) was run to findastrometric projection parameters for each of the im-ages such that they could be transformed onto a com-mon reference frame. All fringed and cleaned imageswere then combined into two separate co-additions us-ing
SWarp (Bertin et al. 2002), both generated with amedian combination filter and background subtraction.Aperture photometry catalogs were then generated onthe final co-additions using
SExtractor with the
LOCAL Medford et al.
Figure 9.
The measured flux of 5000 fake 5-sigma sources injected into an i-band image on the fringed images (orange)and after cleaning (red) using the
SExtractor GLOBAL background setting (top),
LOCAL background setting (bottom), aperturephotometry (left) and PSF photometry (right). On each sub-figure a Gaussian of 5000 sources is drawn (gray) as a visualguide. In all cases the cleaning method significantly increases the accuracy and precision of the recovered flux, particularly forPSF photometry where attempting photometry on images with fringes can often result in overestimating the brightness of thesource. Aperture photometry also overestimates the flux resulting in a deficit of lower flux detections than would be statisticallyexpected, although to a lesser degree. If a method for removing fringes is not applied, it is best to use the
SExtractor LOCAL background setting and a catalog of aperture photometry to best measure the true magnitude of faint sources. background setting. Sources at the locations of the in-jected signals were recovered and their signal-to-noisemeasured as the ratio of the aperture flux to the locallydetermined aperture flux error. We then repeated thisprocess 50 times for a total of 5000 fake sources dis-tributed over 50 fringed and 50 cleaned co-additions.Figure 10 shows the quantitative and qualitativephotometric improvements to these recovered sources.Sources observed on fringed co-additions peak at a signalto noise of 3, pushing them below the typical 5-sigmaobservable threshold. These sources also have a longtail of excessive flux stretching as high as 10 sigma dueto the additional flux caused by atmospheric fringes.Sources observed on cleaned co-additions are observedmuch closer to their theoretical distribution. Thesesources peak at exactly 5-sigma and the large majorityof sources are within the normal signal-to-noise rangeof 4 to 6. There still exists a tail of excessive fluxes,demonstrating that our method is failing to removeall excessive correlated background from the individ-ual exposures. However our method clearly producesan observed flux distribution that is much closer to aGaussian distribution. The largest difference between the two populations ap-pears in their yields. Of the 5000 sources injected intoeach of the fringed and cleaned images, 1161 sourceswere recovered from cleaned co-additions and only 156sources were recovered from fringed co-additions. Over95% of the 0.5 sources injected into the fringed imagesfailed to be recovered after co-addition. The order ofmagnitude increase in the number of sources recoveredby our method enables ZTF i-band surveys to recoverfaint sources that would otherwise have been extremelyunlikely to observe. We note that forced photometry atthe known locations of the fake sources may have in-creased the recovered yields for both populations butwould fail to be an accurate representation of the obser-vation process undertaken for unknown sources.Our analysis demonstrates the power of our methodto remove photometric error due to atmospheric fringesand enable the recovery of faint sources in both singleimages and co-additions that would otherwise have beenundetectable. Failure to implement a method for remov-ing atmospheric fringes greatly reduces the effectivenessof the i-band filter for observing any sources fainter than18th magnitude. Our method increases the photomet- emoving ZTF Atmospheric Fringes Figure 10.
The signal-to-noise distribution of 5000 0.5sigma sources injected into 100 images after a median co-addition (top) for images with atmospheric fringes (orange)and after cleaning (red). The photometric catalogs weregenerated with a
SExtractor LOCAL background and aper-ture photometry. 1161 sources were recovered in the cleanedimages and only 156 sources were recovered in the fringedimages, demonstrating the necessity to clean i-band imagesin any attempt to find faint sources after multi-epoch co-addition. Those sources that were recovered in the fringedimages were most likely to be detected at an artificially lowsignal-to-noise, with a long tail of higher signal-to-noise dueto falling on positive fringes. The cleaned images have a dis-tribution much closer to Gaussian, with a small deficit atthe low signal-to-noise end that also appears as a tail at thehigher end. The photometric improvement in multi-epochco-additions can be clearly seen in as a smoothly varyingbackground and the presence of faint sources after cleaninghas been applied (bottom). ric precision of the i-band to that of g-band and r-bandimages that do not suffer from atmospheric fringes. DISCUSSIONOver the past 30 years astronomy has moved from ascience driven by the studies of individual, often unique,objects to one in which astronomical surveys are unrav-eling new astrophysics through the statistical analysisof large samples of data. While the field of cosmologyhas driven much of this effort, it is now playing a vi-tal role in nearly all branches of astronomy. From themeasurement of H (Freedman 2017), to the mapping ofthe structure of the MW through astrometry (Gaia Col-laboration et al. 2018), to the detection of new planetsaround distant stars (Borucki et al. 2010), we are clearlyin the era of precision astronomy. ZTF, a forebearer offuture wide-field survey such as LSST on the Vera C.Rubin Observatory is no different. Many of its sciencegoals hinge on precise photometric measurements, chiefamong them cosmological measurements from Type Ia supernovae as this field is no longer dominated by sta-tistical uncertainty, but rather systematic uncertainties(Abbott et al. 2019).The application of our method has several benefitsbeyond the photometric quality of individual i-band im-ages. Image references are constructed through the com-bination of as many as 40 epochs taken at a single field.As shown in Section 4.4, removing fringes significantlyimproves the photometric precision on a multi-epoch co-addition. This will also greatly improve the quality ofthe ZTF alert stream. The alert stream packets aregenerated on difference images created from subtract-ing individual epochs from a multi-epoch co-addition(after appropriate scaling). Correctly removing fringesmakes visible the faint end of the luminosity functionthat would otherwise not be observable in the i-bandalert stream.Future work on removing atmospheric fringes usingPCA could improve upon our method in several ways.Training on CCDs instead of readout channels couldproduce better eigen-images by including more corre-lated pixels in each set of features. Also, different meth-ods of calculating the local flux error such as calculat-ing the root-mean-square error on a smaller (or larger)box around each pixel could produce a less noisy UBImeasurement. Lastly, investigating the eigenvalues forfringe bias images generated on different readout chan-nels from the same exposure could reveal correlationsthat could be used to perturb the fringe bias into a bet-ter fit for each fringed readout channel. Treating eachreadout channel as entirely independent, while conve-nient and a natural fit for the IPAC processing pipeline,may leave out valuable information that could improveour method.The eigen-images shown in Figure 2, as well as theeigen-images of many of the other fringe models, showsignificant smooth variations that are not being re-moved by the current flat fielding pipeline. This in-dicates that the fringe bias images include not only at-mospheric fringes, but residual global gradients. Futurework could be done on exploring the application of thisPCA method to supplement or even replace the currentflat fielding pipeline on not only the i-band images, butg-band and r-band images as well. The Uniform Back-ground Indicator can be used as a quantitative measure-ment to compare how well a PCA method, as comparedto more classical flat fielding methods, generates astro-nomical images with normal backgrounds.This work is based on observations obtained withthe Samuel Oschin Telescope 48-inch and the 60-inchTelescope at the Palomar Observatory as part of the2 Medford et al.