Figures of Merit for Testing Standard Models: Application to Dark Energy Experiments in Cosmology
aa r X i v : . [ a s t r o - ph . C O ] S e p Mon. Not. R. Astron. Soc. , 000–000 (2009) Printed 14 November 2018 (MN L A TEX style file v2.2)
Figures of Merit for Testing Standard Models:Application to Dark Energy Experiments in Cosmology
A. Amara ⋆ & T. D. Kitching † Department of Physics, ETH Zurich, Wolfgang-Pauli-Strasse 16, CH-8093Zurich, Switzerland SUPA, Institute for Astronomy, University of Edinburgh, Royal Observatory Edinburgh, Blackford Hill, EH9 3HJ
Accepted —. Received —; in original form —.
ABSTRACT
Given a standard model to test, an experiment can be designed to: (i) measure thestandard model parameters; (ii) extend the standard model; or (iii) look for evidenceof deviations from the standard model. To measure (or extend) the standard model,the Fisher matrix is widely used in cosmology to predict expected parameter errorsfor future surveys under Gaussian assumptions. In this article, we present a frame-work that can be used to design experiments such that it maximises the chance offinding a deviation from the standard model. Using a simple illustrative example,discussed in the appendix, we show that the optimal experimental configuration candepend dramatically on the optimisation approach chosen. We also show some simplecosmology calculations, where we study Baryonic Acoustic Oscillation and Supernovesurveys. In doing so, we also show how external data, such as the positions of theCMB peaks measured by WMAP, and theory priors can be included in the analysis.In the cosmological cases that we have studied (DETF Stage III), we find that thethree optimisation approaches yield similar results, which is reassuring and indicatesthat the choice of optimal experiment is fairly robust at this level. However, this maynot be the case as we move to more ambitious future surveys.
Key words:
Numerical Methods, Cosmology
In cosmology, the ΛCDM concordance model has becomeour standard model of the Universe. This model satisfiescurrent data and depends on three critical sectors: (i) DarkEnergy; (ii) Dark Matter; and (iii) Initial Conditions. Thesesectors are linked through our theory of gravity - generalrelativity. Although this model is well defined, the additionof each component has typically been done to explain theavailable data rather than arising from some fundamentaltheory of the cosmos. Hence, cosmology is currently in adata-driven era, with little known about the fundamentalnature of dark matter and dark energy. As a result, a signif-icant effort is underway in this very active field to buildexperiments to measure and extend our standard model.These include KIDS, Pan-STARRS , DES , LSST , JDEM ⋆ [email protected] † [email protected] http://pan-starrs.ifa.hawaii.edu http://jdem.gsfc.nasa.gov and Euclid , . In planning such future observations, the ap-proach to date has been to optimise the experimental andmethodological designs to minimise the errors on extendedparameters. In particular, the dark energy equation of state(the ratio of pressure to density of dark energy w ( z )) gar-ners the most attentions and is typically parameterised interms of a second order Taylor expansion in the scale factoror redshift z (e.g. w ( z ) = w + w a z/ (1+ z )). Experiments arethen designed to measure these equation of state parametersto the highest possible precision. The dark energy Figure ofMerit (FoM; Albrecht et al. 2006), which is proportional tothe area of the error ellipse in the w - w a plane is widely usedto gauge performance. Other possible metrics have also beensuggested, such as the addition of parameters to test for de-viations from Einstein gravity or the division of w ( z ) intoa large number of redshift slices that can then be used toconstruct principal components through a matrix inversion(Albrecht et al. 2009; Huterer & Starkman 2003). However,these two suffer from their own problems. For instance, theadditional modified gravity parameters may not be strongly http://sci.esa.int/euclid c (cid:13) Amara & Kitching motivated and the eigenfunction decomposition of w ( z ) cansuffer from instabilities (Kitching & Amara 2009).In this article, we present an alternative methodology to beapplied to experimental design when faced with a standardmodel and no guidance from theory. We show that an exper-iment can be designed such that the probability of breakingthe standard model (finding evidence against the model) canbe maximised.This article is organised as follows. In Section 2, we reviewthe alternative approaches to experimental design. We then,in Section 3, compare each approach using a simple explana-tory model, as well as a cosmological example that studiesthe performance of the ‘current’ and Stage III experimentsdiscussed in Albrecht et al. (2006). We summarise our con-clusions in Section 4. When planning an experiment with a standard model (a setof parameters) in mind, we can think of three possible ap-proaches that we can take. The first is to stay within thestandard model and to design an experiment that will mea-sure the parameters of this model to the highest possible pre-cision. The next is to extend the standard model (add extraparameters), and ideally this extension would be driven by acompelling theoretical framework with clear testable predic-tions. Finally, in the absence of any compelling theory, onecan take a more exploratory approach, where the driving aimis to design an experiment with the greatest chance of break-ing the standard model. Ideally, this approach would dependonly on well-founded knowledge, such as today’s data, theexpected error bars of future data and the standard modelthat is being tested.
Within a well-specified model, the Fisher matrix formalism(Tegmark et al. 1997) is a well-defined framework for esti-mating the errors that a given experiment will have on themeasurement of the parameters of the model. For an exper-iment where the parameters have an effect on the mean, theFisher matrix is defined as F ij = X C ∂C∂ Θ i ∂C∂ Θ j , (1)where C is some observable signal, ∆ C is the expectederror for an experiment and Θ is a vector containingthe parameters. A cosmology model may include Θ = { σ , Ω m , Ω b , Ω Λ , n s , h, etc } , where, for instance, the darkenergy equation of state is assumed to be a cosmological con-stant ( w ( z ) ≡ − Cov ), which is given by
Cov = F − . When seeking out new physics, we look for ways of going be-yond the standard model. Ideally this would be done throughthe guidance of theory. There are many examples of caseswhere theories have been put to the test by experimentsbased on verifiable predictions. One such example is neutrinomass. In the standard model of particle physics, neutrinoshave zero mass, but the assumption of zero mass is an ad hocchoice. A natural and physically motivated extension of thismodel was to add mass to neutrinos (through the lepton mix-ing matrix addendum). Neutrino mass has now been exper-imentally confirmed by a number of particle physics exper-iments (Ahmed et al. 2004; Eguchi et al. 2003; Ahn et al.2006), and cosmological experiments should be able to con-strain this mass to high accuracy (e.g. Refregier et al. 2010;Thomas et al. 2009; Kitching et al. 2008).Extra parameters, Ψ, can be added to the parameters of thestandard model, Θ. In this case, the Fisher matrix formalismcan once again be used to estimate the errors on all theparameter sets. Here, it becomes useful to decompose thematrix as F = (cid:18) F ΘΘ F ΘΨ F ΨΘ F ΨΨ (cid:19) , (2)where the matrix F ΘΘ contains the Fisher matrix elementsfor the parameters of the standard model, F ΨΨ contains theelements for the new model parameters and F ΘΨ containsthe cross terms.This approach has been widely adopted by the cosmologicalcommunity in dark energy studies. In this case, the extra pa-rameters are typically added in the form of equation of stateparameters (the ratio of pressure to density) of dark energy( w ). However, this is a specific way of thinking about darkenergy (as a dynamical fluid). Therefore, models that do nottreat dark energy as a fluid have to work in terms of an ‘effec-tive’ equation of state. A further complexity arises becausethe observed low redshift acceleration that motivates darkenergy could result from other physics, such as the break-down of Einstein gravity on cosmic scales. A move away fromEinstein gravity may not be well represented by the addi-tion of equation of state parameters and may require theaddition of new parameters that specifically allow for suchdeviations. As a result, these extra dark energy parametersdo not have a firm theoretical basis but are, in fact, an arbi-trary expansion of the equation of state (Kitching & Amara2009). c (cid:13) , 000–000 odel Breaking Here, we introduce a new approach to experimental plan-ning, where we explicitly design an experiment to maximisethe probability of finding a deviation from the standardmodel. This deviation is allowed to come from any part ofthe theory and should not depend on any particular theoret-ical extension of the standard model. The robustness of suchan approach can be achieved by relying on minimal inputs,namely: (i) current data; (ii) expected error bars of futuremeasurements; and (iii) the standard model that we wantto test.We begin by defining some basic parameters. Let X be adata vector containing today’s measurements (for instancea correlation function). These data points have associatederrors, σ X , which means that the measured data points arerandomly scattered about T, the data vector that would bemeasured with no measurement error or systematic, i.e. theunderlying values of the observable as measured with theperfect experiment . The expected error bars of a futureexperiment are σ Y , which would produce a data vector Y .Given today’s data, we can calculate the probability of thefuture data, P ( Y | X ), by marginalising over T, P ( Y | X ) = Z P ( Y | T ) P ( T | X ) dT, (3)where P ( T | X ) is the probability of T given today’s dataand P ( Y | T ) is the probability of the future data given T.The integral is performed over all possible T since we do notknow what T is a priori.For each realisation of the future data, there will be an asso-ciated best-fit that can be achieved with the standard model.We focus here on the χ . With the probability distribu-tion of future data given current data ( P ( Y | X )), which, forsimplicity, we will sometimes also denote using P ( Y ), wecan calculate the expectation value of the minimum χ byintegrating over all possible future data vectors: h χ i = Z χ ( Y ) P ( Y ) dY. (4)A high χ means that the standard model is not able togive a good fit to future data. Hence, an experiment designerwho wants to maximise his or her chances of breaking thestandard model should focus on an experiment configura-tion that maximises the expectation value of the minimum χ ; max[ h χ i ]. Strictly, we should use a quantity that isrobust to the number of data points (for instance the re-duced χ ). We avoid such problems in what follows by only As an example, if X is calculated from the mean of n inde-pendent data points and the errors are given by the variance( σ ( ¯ X ) = σ ( X ) /n ), then T would be the measure given as ngoes to infinity in the absence of systematics. We note that, inthis case, cosmic variance would come from the fact that due toa finite Universe the number of independent data points will belimited to a finite number. making comparisons between experiments with equal num-bers of data points. The χ and reduced χ are, therefore,simply scaled versions of each other. In this work, we havefocused on the expectation value of the minimum χ of thefuture data, with the understanding that a χ correspond-ing to a reduced χ significantly larger than one will requireadditional parameters beyond those available in the stan-dard model. However, it may be interesting to also considerthe higher order statistics of the minimum χ distribution.Along similar lines of thought, our FoM could also be re-cast in terms of the probability that a future experimentwill give a χ greater than some threshold value. For thework presented here, we use the simplest expression (givenin equation 4), but we are continuing to investigate furtherpossible expressions of this model breaking FoM.Here, we use the maximum likelihood fit to the data (mini-mum χ ). We have used this frequentist measure, as opposedto a Bayesian evidence criteria, because there are no objec-tive Bayesian measures in the case of assessing the qualityof a theoretical fit for a single model, given that a singlemodel Bayesian evidence must conclude (through a normal-isation of probabilities) that there is 100% evidence for thatmodel (see Taylor & Kitching, 2010 for further discussion).In general, this χ ( Y ) measure could be replaced with any‘goodness of fit’ criteria G ( Y ), where equation 4 optimisesfit. In Appendix A, we explore the impact of the choice of opti-misation metric on a simple illustrative example. We set upa system of three data points and ‘a standard model’ thatis a straight line with one degree of freedom - the slope ofthe line. What this shows is that the optimal configurationof a future experiment can vary drastically and can leadto exactly opposite optimisations in some cases dependingon whether model breaking or standard model extension isused.The simple model that we set up has a ‘pivot point,’ wherethe model makes an exact prediction, C ( x = 8) ≡
10. Tomeasure the standard model parameter (the slope), assum-ing that this model is correct, it is clear that there is nosensitivity at this point. Therefore, an optimisation will min-imise future error bars away from the pivot point. However,in the model breaking mode, it is optimal to place the small-est future error bars at the pivot point, since it is here thateven the slightest deviation from the standard model predic-tion would yield proof that the standard model is broken.Of course the model breaking paradigm here is a high-risk,high-gain approach. If T happens to have the same valueas that of the pivot point, then this approach would yieldno extra information. When extending the standard model,the optimal configuration is entirely dependent on the exactform of the extension. For instance, a clear difference is seen c (cid:13) , 000–000 Amara & Kitching between a standard model that is extended by adding a con-stant parameter and one that is expanded with a parabolicterm about the pivot point, thereby preserving the pivotpoint.
We now apply our approach to investigate the planning ofcosmology surveys. In this work, we focus on some sim-ple examples that show how this can be done, with amore complete investigation of future surveys to followin later work. In this example, we focus on supernovea(SNe) (Tegmark et al. 1998) and Baryon Accoustic Oscil-lation (BAO) (e.g. see Rassat et al. 2008, for discussion). Inaddition, we will show: (i) how external data, in this case theCMB peak separation, can be added; (ii) how priors comingfrom theory can be included; and (iii) a simple treatmentfor systematics errors.
Due to the computational limits of performing the integralshown in equation 4, the dimensionality of which scales withthe number of data points, we have decided to bin the lowredshift data (i.e. SNe and BAO) into four redshift bins (i.0 . < z < .
4; ii. 0 . < z < .
7; iii. 0 . < .
0; and iv.1 . < z < . χ values directly. This simplifies the com-parison between different survey configurations. For currentBAO data, we use the galaxy number counts presented inPercival et al. (2010). This work presented a BAO analy-sis of the Sloan Digital Sky Survey Data Release 7 sam-ple (DR7). This is composed of roughly 900,000 galaxiesover 9100 deg in the redshift range z = [0.0, 0.5]. We re-binned this data into our four redshift bins which leads tothe distribution shown in Table 1. For current SNe data,we use the Union data presented in Kowalski et al. (2008).This is a compilation of SNe data coming from a numberof measurements, including the Supernova Legacy Survey,the ESSENCE Survey and supernovae measurements fromthe Hubble Space Telescope (HST). Once again, as with theBAO data, we have re-binned this data to match the fourbins that we use in this paper (see Table 2). As we willdiscuss in Section 3.2.2, we have also included constraintscoming from current measurements of the CMB peak sep-aration presented in Komatsu et al. (2009), which uses theWMAP data.For future surveys, we have decided to focus on a configu-ration that illustrates the technique presented here, ratherthan to make concrete recommendations about specific mis-sion concepts. The reason for this is that the calculationsthat we present here include a number of simplifications,such as using only four redshfit bins. These, we feel, allowus to calculate trends and make some statements about the Figure 1.
Fractional errors on the observed quantities for ‘cur-rent’ (black) and stage III (red) experiments. For the BAO mea-surements, these are the errors on the transverse BAO scale fromBlake et al. For the SNe surveys, the observable is the flux lossof the SNe. relative merits of broad concept ideas. However, to draw de-tailed conclusions on specific mission configurations wouldtake further detailed work that we will address in follow uppublications on this topic. For the future surveys that weuse to illustrate our method we have relied on the Stage IIIsurveys described in Albrecht et al. (2006), although manyof the projects may have evolved since this document wasreleased. Once again, we re-bin the Stage III data into ourfour redshift bins (see Tables 1 and 2).For the BAO surveys, we simplify the analysis by only us-ing the tangential modes, which is pessimistic, and assumeno systematics, which is optimistic. Due to these reasons,the results below are illustrative, and we do not claim thatthe optimistic and pessimistic approaches cancel out eachother. We calculate the errors on BAO scale using the fittingfunction given in Blake et al. (2006), which has been im-plemented in iCosmo (Refregier et al. 2008; Kitching et al.2009). For the Supernova error calculations, we have usedthe Fisher matrix approach outlined in Tegmark et al.(1997) and Huterer & Turner (2001) and have assumed asystematic contributions outlined in Kim et al. (2004) andIshak et al. (2006). However, we will also show results with-out systematics in order to gauge their impact.
In this study, we focus on the potential of future BAO andSNe surveys. It is, however, straightforward to include otherdata sets. To do this, we must decide whether to only includecurrent measurements (for instance, in the case of the CMBto include WMAP data) or try and anticipate the joint im- c (cid:13) , 000–000 odel Breaking Area Number Density of Galaxies (n g ) [num/amin ]0.1 < z < < z < < z < < z < Table 1.
Parameters of the BAO surveys considered in this study. The current survey is chosen to be close to the BAO survey parametersfor the SDSS DRL7 (Percival et al. 2010). The future surveys have been chosen from the Stage III surveys of the Dark Energy TaskForce report (Albrecht et al. 2006). Number of Supernovae (n s )0.1 < z < < z < < z < < z < Table 2.
Parameters of the Supernovae surveys considered in this study. The current survey is chosen to be close to the Union supernovaesample (Kowalski et al. 2008). The future surveys have been chosen from the Stage III surveys of the Dark Energy Task Force report(Albrecht et al. 2006). pact of future measurement of that probe (for instance, toinclude predictions for Planck ). If the latter is desired, thenthe prescription for doing so follows the same logic as thatused for the BAO and SNe calculations and would increasethe data vectors ( F and X ) in equation 4. While conceptu-ally simple, adding external data in this way can quickly leadto computational challenges, since the dimensionality of theintegral scales the number of data points. The computationtime for convergent results can diverge quickly, even using asimple Monte-Carlo integration scheme. To solve potentialproblems, we would either need to develop a sophisticatedMonte-Carlo integration scheme with, for instance, impor-tance sampling that is tailor made for this problem or tryto reduce the number of data points by focusing on specificfeatures of the external data that we wish to consider. Forinstance, in the case of the CMB we can consider addingthe peak position and height information rather than imple-menting the full correlation data ( C ( ℓ )).If we only add existing external data, then the calculationis greatly simplified, since the dimensionality of the integralin equation 4 remains the same. Instead, the external datais simply used when calculating the minimum χ . In thework presented here, we have included the measured spac-ing of the acoustic oscillation peaks of the CMB, ℓ A , whichdepends on the ratio of angular diameter distance to thesound horizon at photon decoupling epoch ( z ∗ ), ℓ A = (1 + z ∗ ) πD A ( z ∗ ) r s ( z ∗ ) , (5)where D A is the angular diameter distance and r s is thesound horizon. This peak spacing has been measured to be ℓ A = 302 . ± .
86 for WMAP (Komatsu et al. 2009), whichgives an expression for z ∗ in equation 66. For the sound horizon calculation, we follow the calculations presented inAppendix A of Parkinson et al. (2007). We now turn our attention to priors coming from our the-ory and how these can bound our results. For example, ifwe impose no knowledge at all about what we expect, thenthe PDFs for each of the data points in equation 3 are in-dependent. A simple consequence of this is that the proba-bility distribution for future data in bins where no currentdata exists ( P ( F | X )) will be flat between −∞ and ∞ . In-puting this PDF into equation 4 would lead to a h χ i ofinfinity, which is not fully useful when comparing expectedperformances. One can view this result in two ways. Thefirst is that a data purist (i.e. someone who wishes not toadd any bounds from theory) would conclude that the bestsurveys are those that explore new regions where no mea-surements have yet been made. The alternative approach isto introduce some expectation from our knowledge of basiccosmological theory. Theory priors modify the PDFs of fu-ture data by imposing relationships between different datapoints. A simple addition is to impose a link between theangular diameter distance and the luminosity distance.For the configurations shown in Table 1, we immediatelysee that if we take no guidance from theory then we will bedriven towards WiggleZ and WFMOS (see Table 1), sincethese two surveys will provide BAO measurements at red-shifts that are currently not explored by current BAO exper-iments and, hence, have an expectation value of minimum χ of infinity. Once again, a data purist may argue that thesesurveys should, therefore, be our top priority. In contrast,another simple approach is to rely on the widely acceptedrelationship between angular diameter distance ( D A ) andluminosity distance ( D L ) given by c (cid:13) , 000–000 Amara & Kitching D L = (1 + z ) D A . (6)By explicitly adding this very weak prior from the theory,the probability of future data is modified (equation 3) to P ( Y | X ) = Z P ( Y B | D L ) P ( Y S | D L ) P ( D L | X B ) P ( D L | X S ) dD L , (7)where Y B and Y S are the data vectors for future surveysfor BAO and SNe (respectively) and X B and X S are thedata vectors for today’s surveys. This PDF, therefore, in-cludes a relationship between the SNe measurements andthe BAO measurements at any given redshift. For what wepresent later, this relationship between distances is the onlyinformation that we impose from theory. However, a natu-ral question is what would happen if the future data were toextend to redshifts that are not covered by either the BAOor the SNe data? A detailed exploration of this will be pre-sented in follow-up work. Nonetheless, here we give a briefdiscussion of the basic principles. Once again, priors fromtheory can be used to impose relationships between differ-ent data points, which in turn modify the PDF of the futuredata. In particular, the question raised here would look forrelationships between data points at different redshifts. Thiscan be done by introducing an integral relationship betweendistance (co-moving - D c ) and the Hubble function, H ( z ), D c = c Z z ′ dz ′ H ( z ′ ) , (8)where c is the speed of light. Without resorting to the Fried-mann equation, which links H(z) to density parameters ofthe matter-energy components of the Universe, we can placesimple constraints on the functional form of H(z) that can beused to compute the probability of future data. For instance,an assumption that H(z) is a positive definite function overcosmic time would bound the comoving distance at a red-shift of z i to be between the comoving distances at z i − and z i +1 , i.e. that of the redshifts on either side. Here the in-clusion of the CMB, with z ∼ D L and D A , must be included explicitly. This then allows usto decide explicitly what assumptions should be included. h χ i For each realisation of the the future data (Y) we calculatethe weighted average data, which is given by X c = σ X Y + σ Y Xσ X + σ Y , (9)where X c is the value of the combined data, Y and X arethe future and current data values, and σ are the associatederrors. The errors on the combined data are σ c = σ X σ Y σ X + σ Y . (10)The data vector X c can also contain external data for whichthere will not be corresponding future measurements. In thiscase, the data vector enteries that correspond to the externaldata have X c = X and σ c = σ x . With this combined datavector, we then calculate χ , χ = X ( X c − M ) σ c , (11)where the sum is over the entries of the data vector. In ourcase, this corresponds to a total of nine data points (BAOscale at four redshifts, SNe at four redshifts and the CMBpeak spacing). For a given choice of cosmology parameters,M is the value given by the model. For each integration step,we use a minimiser to find the parameters that lead to thesmallest χ value.The Stage III surveys will look for deviations from the stan-dard ΛCDM concordance model. We consider the standardcosmological model as one with Gaussian initial conditions following inflation, with scale-free perturbations ( n s =1),where spatial curvature is allowed and dark energy is un-derstood to come from the cosmological constant Λ (i.e. w = − : { Ω m , Ω Λ , h } . The model breaking approachdoes not rely a adding further parameters beyond these well-understood ones and will test how likely it is that future ex-periments, based on today’s data, would find any deviationfrom ΛCDM, including, for example, evidence for w = − Z fdV ≈ V h f i ± V r h f i − h f i N , (12)where the expectation values, denoted by the angular brack-ets, can be calculated by randomly sampling the function f at positions x i with h f i ≡ N N − X i =0 f ( x i ) . (13) See Amara & Refregier (2004); Desjacques & Seljak (2010);Pillepich et al. (2010) for examples of how non-Gaussian initialconditions impact observables at low redshifts We note that there is a weak dependence on Ω b through z ∗ ,but we have neglected this here since it has little impact on theresults and only complicates the calculation.c (cid:13) , 000–000 odel Breaking The volume of the parameter space is denoted as V. Thisis set by the bounds of the integral, which we have choosenin such a way as to ensure that the integrand is vanishinglysmall at this limit.
Performing survey optimisations for future experiments typ-ically involves a trade-off between different configurationsthat compete for resources. A classic example is a trade-offbetween the depth and area of a survey for a fixed expo-sure time (see Amara & R´efr´egier 2007, for an example ofthis for weak lensing surveys). Another, more difficult andoften controversial trade-off study, is to trade-off resourcesbetween different proposed probes. For instance, if due tolimited resources it is not possible to support both SNe andBAO missions envisioned for stage III. A natural questionmight be - should we invest in one over the other? Or shouldscaled-down versions of each mission be pursued? This isa complex issue for a number of reasons, but the modelbreaking figure of merit, along with other FoMs, can helpguide such decisions by quantifying the likelihood of find-ing a deviation from ‘the standard cosmological model’. Forthis reason, our first illustrative example focuses on a possi-ble trade-off study between SNe and BAO stage III surveys.We note again that a thorough treatment of such a trade-offis complicated. For instance, quantifying the impact of lim-ited resources is significantly more complicated than that oflimited observation times. We made a number of simplifyingassumption, so the results stated here are only to illustratethe method rather than to offer concrete recommendationsabout one experiment over another. In this spirit, we willshow results for the full Stage III surveys, as well as for thescaled down versions. We do not attempt to make a link be-tween the scaled-down versions for a fixed set of resources,since this is well beyond the scope of this work. For scalingdown the surveys, we have decided to fix the distributions inredshifts (i.e. the PDF of the number of SNe and galaxies asa function of redshift is fixed), and we vary an overall scaling.For BAO this corresponds to a change in survey area, andfor SNe this corresponds to a reduction in the total numberof SNe.In Figure 2, we show the expectation value of the minimum χ when we consider only some fraction of the area of theStage III surveys shown in Table 1. For instance, for a frac-tion of 0 . χ . Beyond this, however, we see alarge increase in h χ as the area of the BAO surveys is in-creased, leading to mean χ values that are greater than5 (i.e. a reduced χ greater than one) for all survey con-figurations with 100% of the DEFT stage III survey area.This is true with and without SNe systematics. In Figure3, we show similar results as a fraction of future SNe sur- h χ m i σ c (Ω Λ ) /σ III (Ω Λ ) FoM III / FoM c BAO III 5.5 10 2.2SNe III 3.5 (4.0) 6 (14) 1.1 (2.6)BAO & SNe III 7.0 (8.0) 10 (19) 2.2 (4.4)
Table 3.
Comparison between the model breaking approach( h χ i ), working within the standard model (here we show er-rors on Ω Λ in a model with only cosmological constant) andDETF FoM (which involved parameterising the equation of statein terms of w and w a ). The numbers in parentheses are when nosystematics are included for SNe, while the other numbers havethis systematic included. veys. We see linear rise in h χ i with the SNe fraction from1% to 100% of stage III experiments. Here, the rise is lessdramatic than in the BAO case, and this suggests that it ismore likely for discovery to come from the BAO experiment.This result can also be seen in Table 3, where we also showthe comparison with the other figures of merits discussed insections 2.1 and 2.2. The middle column shows the errors onthe standard model parameters, in this case the density of Λ,and on the right we show the FoM proposed by the DETF,which is proportional to the area of the error ellipse in the w - w a plane Albrecht et al. (2006). Reassuringly, all threemeasures show similar trends, which would suggest that thesimple optimisations done here are reasonably robust andthe overall information content is increased between exper-iments with lower FoM and ones with higher ones. This isdifferent from the tradeoff studied in appendix A, where theoverall error bars are fixed and the sensitivity in differentregions ( x values) leads to changes in the FoMs.Finally, we investigate a simple optimisation where we ex-plore the model breaking redshift sensitivity of the Stage IIIsurveys. We do this by boosting the performance of the sur-veys at a particular redshift by dividing the statistical errorsat that redshift by a factor of 2. This is not a physically mo-tivated optimisation. Instead, it can be thought of as simplyprobing where an improvement would be the most effective.The results are shown in Figure 4. The coloured bars showthe fractional increase in h χ m i for the calculations whereSNe systematics have been included. We see here that im-proving the SNe survey in the two lowest redshift bins causesa notable increase in the h χ i , while improving the SNeperformance in higher redshift bins has little effect, exceptin the no systematics case. This suggests that to go beyondstage III SNe experiments we should focus on improving er-rors at low redshifts first, unless we can demonstrate that thesystematics levels can be brought below those presented byKim et al. (2004) and Ishak et al. (2006). For the BAO ex-periments, we find a different result. Improving the errors inour lowest redshift bin has no effect on h χ i . However, wesee that if the errors in our final redshift bin (0 . < z < . h χ i . Thissuggests that a BAO experiment beyond stage III shouldaim to make measurements at high redshifts. c (cid:13) , 000–000 Amara & Kitching
Figure 2.
Expectation value of the minimum χ as a function ofthe areas of the stage III BAO surveys. The fraction correspondsto the fraction of the full survey areas (shown in Table 1) used.These are shown for three configurations of stage III SNe surveys,where only a fraction of the SNe in Table 1 are used. The solidcurves include SNe systematics while the dotted curves do not.The dashed line shows the χ that would correspond to a reduced χ of 1. Figure 3.
Expectation value of the minimum χ as a functionof the number if SNe of the stage III surveys. The fraction corre-sponds to the fraction of the full survey number (shown in Table1) used, where the PDF is fixed and only a global fraction is ap-plied. These are shown for three configurations of stage III BAOsurvey area where only a fraction of the areas in Table 1 are used.The solid curves include SNe systematics while the dotted curvesdo not. The dashed line shows the χ that would correspond toa reduced χ of 1. Figure 4.
The impact of boosting the performance in one of theredshift bins. This is done by reducing the statistical error in therelevant bin by a factor of 2. The y-axis shows the ratio of the ex-pectation value of the minimum χ of the boosted stage III surveyrelative to the standard stage III survey. The different colours cor-responding to which probe have been enhanced; the solid coloursare when SNe systematics are included, and the dashed lines showthe results when SNe systematics are eliminated. We have presented a framework in which experimental op-timisation can be placed. Given a standard model, one caneither (i) measure the standard model parameters to highprecision; (ii) attempt to extend the standard model; or (iii)attempt to find deviations from the standard model.When designing an experiment to measure or extend thestandard model, the Fisher matrix formalism can be used.We have introduced a framework that can be used to designan experiment to have the best chance of finding discrepan-cies with the standard model. This framework only dependson three sets of information (current data, future expectederror bars and the standard model). No external assump-tions are needed for the calculations, though we have alsoshown how priors from the theory can, if needed, be added.By using a simple illustrative example, we find that theoptimal future experiment configuration can depend verystrongly on the choice of optimisation metric. In our simplemodel, C = m ( x −
8) + 10, the data position x = 8 is apivot point since C ( x = 8) ≡
10. When designing an ex-periment to measure the standard model, it is optimal tohave small errors away from the pivot point. However, whendesigning an experiment to break the model, it is optimal tohave a small error at the pivot point since any measurementof C ( x = 8) = 10 would provide evidence that the standardmodel was incorrect. When extending the model, the opti-misation naturally depends on the exact parameterisationof the extension.In cosmology we have a standard model, ΛCDM. A largenumber of experiments have been designed to measure anad hoc extension of this model, parameterisations of thedark energy equation of state, to high accuracy. Our rec-ommendation here is that future cosmology missions shouldbe optimised by using the three approaches we have out- c (cid:13) , 000–000 odel Breaking lined above: (i) measure the standard ΛCDM parameters;(ii) measure extended parameters, specifically the equationof state parameters, the DETF FoM and the modified grav-ity parameter γ ; and (iii) calculate the expectation valuethat the experiment will find a deviation from ΛCDM. Wecalculate quantities in these three regimes for SNe, BAO(transverse modes) and the CMB peak positions by focus-ing on ‘current’ and the DETF stage III surveys. Should thethree quality quantifiers agree, then we can be reassured thatthe optimisation is somewhat robust. For instance, there hasbeen some concern that the DETF FoM is biased in favourof redshifts. However, in the calculations shown in this pa-per, we do not find evidence for this, with the results for theDETF FoM being consistent with the other figures that wehave shown. In the event that the three approaches lead toconflicting configurations, the the fact that these measureslook for distinctly different thing means that we should beable to make a choice based on a judgement of the prioritiesof a given experiment. ACKNOWLEDGMENTS
We would like to thank Alexandre Refregier for the usefuland insightful discussions that initiated this work. We alsothank Andy Taylor, Fergus Simpson and Anais Rassat foruseful comments. AA is supported by the Zwicky Fellowshipat ETH Zurich. TDK is supported by STFC rolling grantnumber RA0888.
APPENDIX A: ILLUSTRATIVE EXAMPLE
To illustrate the distinction between the three optimisationapproaches highlighted in this article, we will present a sim-ple worked example. We begin with a standard model wherethe signal C at x depends only on the parameter m (i.e.Θ = { m } ). Our standard model is that C = m ( x −
8) + 10 . (A1)For this simple example, we also assume that measurementscan only be made at x = { , , } , where today’s measure-ments have yielded X = { , , } with Gaussian errorsof variance σ Y = { , , } . This is shown in Fig. A1. We willassume that future experiments can be built to measure thesignal at the same x positions as today but that the errorson the measurements will be significantly smaller than thoseof today. Specifically, we will assume that the quadratic sumof the future errors, over all data points, is P x σ Y ( x ) = 2 . Figure A1.
The system being used to illustrate the availableoptimisation options. For this example, the black points are to-day’s data and the red lines are examples of our standard modelthat are consistent with today’s data. In the top right hand cor-ner, an example of the typical size of the error bars in the futureexperiment is shown.
A1 Measuring the Standard Model
To measure performance of a future experiment, we use theFisher matrix to estimate the errors on the parameter m forspecific configurations of the errors. This allows us to findthe optimal configuration of the errors for the purpose ofmeasuring m .Fig. A2 shows how the future errors at the x = 4 and x = 8points are optimised such that the error on m is minimised.It is clear that the optimal configuration is insensitive tothe error at x = 8. This is understandable since within thestandard model there is not sensitivity to m at x = 8, sothere is no gain in placing any measurement at this point.The optimal strategy to measure the standard model m isthen to place small future error bars at either x = 4 or x = 12. It is also interesting to note that since the value ofthe standard model at x = 8 is fixed, it is better to haveone small error on either x = 4 or x = 12 (with the otherbeing large) than to distribute the errors between these twopoints. A2 Extending the Standard Model
To extend the model, we first have to decide on a way of ex-tending the standard model. We must also decide whetherto optimise or minimise the errors on the extended param-eters - after marginalizing over m - or to minimize both thestandard and extended parameters simultaneously. c (cid:13) , 000–000 Amara & Kitching
Figure A2.
The results of an optimisation analysis designed tomeasure m (the only parameter of our standard model) to thehighest precision possible. The quadratic sum of the errors of thethree data points, ( σ x =4 + σ x =8 + σ x =12 ), has been set to 2 . σ x =4 (andby symmetry σ x =12 ). The fact that the lines are close to verticalshows that this optimisation is totally insensitive to the measure-ment precision at x=8. This can be understood since x = 8 is apivot in our standard model and therefore offers no informationwithin our standard model since it can only have a value of 10.For a fixed error at x = 8, we see a clear preference to mimise theerrors at either x = 4 or x = 12, which means that it is better tohave one small error bar than mimising both. For illustration, we assume that there are two equally validways of extending the standard model used here. The firstis the addition of a quadratic term, C = m ( x −
8) + 10 + p ( x − , (A2)and the second is the addition of a constant, C = m ( x −
8) + 10 + p . (A3)Again the Fisher matrix formalism is used to predict thefuture errors on the model parameters, ( m , p ) or ( m , p ),given a configuration for the future data error bars.The results are shown in Figs. A3 and A4. We show theerrors on m (marginalised over p i ) and on p i (marginalisedover m ). We could have constructed a figure of merit thatcombines the errors of m and p i , but this is somewhat su-perfluous in this illustrative example.In Fig. A3, we show how the errors on m and p from model1 are optimised. In this case, x = 8 is a pivot point of theextended model so the optimal strategy is to maximise fu-ture errors at x = 8 since the parameters are not sensitive todata at this point. The quadratic sum of the errors at x = 4 Figure A3.
Optimisation for the two parameters of extendedmodel 1, C = m ( x −
8) + 10 + p ( x − . The plots show theexpected marginalised errors on m and p as a function of possiblemeasurement errors at x = 4 and x = 8. As in Fig. A2, the errorsat x = 12 are set by fixing the quadratic sum of the errors to2 .
01. For this extended model, we see that we are pushed to aconfiguration with maximum errors at x = 8 for both parameters m and p . As with the standard model, this extended model hasa pivot point at x = 8 and so the measurment here does not bringuseful information. Unlike the example shown in Fig. A2, here theerrors at both x = 4 and
12 are important since they are bothneeded to distinguish between m and p (with only one data pointthe two parameters are degenerate). This is why maximising theerror at x = 8, and hence minimising the quadratic sum at x = 4and 12, is preferred. and
12 are then minimised. We note that in this exampleboth of these data points are needed to distinguish betweenthe parabolic and the linear term.In Fig. A4, we show how the errors on m and p from model2 are optimised. In this extended model, x = 8 is no longera pivot point of the model. In fact, a small future error barat x = 8 could measure p very accurately (for a given m )because the errors are not degenerate with m at this point.Hence, the optimisation places a small error bar at x = 8.Next, to accurately measure m, the optimisation tries tominimise the errors on one of the two remaining errors in asimilar way to what happen in Fig. A2. A3 Breaking the Standard Model
For the Fisher matrix calculations we have made the implicitassumption that future measurement errors are Gaussian(Tegmark et al. 1997). For the model breaking approach, wemake the same assumptions, namely that the probability ofT given today’s data is given by c (cid:13) , 000–000 odel Breaking Figure A4.
Similar to Fig. A3, this shows the optimisation forthe two parameters of extended model 2, C = m ( x −
8) + 10 + p .For the extended parameter p , we are pushed towards a config-uration with minimal errors at x = 8. Because this point is not apivot point of the model, it can be used to directly measure p .For m we see that maximum precision is reached by minimisingthe errors at x = 4 and 8 (or by symetry at x = 8 and 12). Thisis because the data point at x = 8 gives the best measure of p ,which is degenerate with m . Once p is measured, only one extradata point is needed in this model. Hence, either x = 4 or 12should be minimised. P ( T | X ) = exp (cid:18) − ( T − X ) σ X (cid:19) , (A4)where today’s data vector is once again, X = { , , } ,and the probability of the future data given T is P ( Y | T ) = exp (cid:18) − ( Y − T ) σ Y (cid:19) . (A5)The future χ is given simply by χ = X i σ Y i ( C i − Y i ) , (A6)which for the illustrative standard model used here (equa-tion A1) is a minimum for m = P i σ − i ( x i − − Y i ) P i σ − i ( x i − , (A7)where the sums are over x = 4, 8, 12. These allow us, forthe simple model being considered here, to solve equation 4analytically. Figure A5.
The expectation value of the future χ . This ex-pectation value must be maximised to have the best chance ofbreaking our standard model. The colour scheme for this plot hasbeen chosen such that the best configuration (max( h χ i )) ispurple (dark), which is consistent with Fig. A2 to A4 where theoptimal strategies are also purple (dark). We see that using thiscriterion that the optimal configuration is one that minimises theerrors at x = 8. This can be understood since any deviation from y ≡
10 at this point cannot be explained within our standardmodel. Given today’s data and no guidance from theory, a highprecision measurement here is, therefore, most likely to break thestandard model.
Fig. A5 shows the result of the model breaking optimisa-tion for this illustrative example. To have the best chance ofbreaking this standard model, one should place a very smallerror bar at x = 8. This is understood since x = 8 has a verystringent prediction that C ( x = 8) ≡ any deviation fromthis prediction would be proof that the standard model wasincorrect. REFERENCES
Ahmed S. N., et al 2004, Physical Review Letters, 92,181301Ahn M. H., et al 2006, Phys. Rev. D, 74, 072003Albrecht A., Amendola L., Bernstein G., Clowe D., Eisen-stein D., Guzzo L., Hirata C., Huterer D., Kirshner R.,Kolb E., Nichol R., 2009, ArXiv e-printsAlbrecht A., Bernstein G., Cahn R., Freedman W. L., He-witt J., Hu W., Huth J., Kamionkowski M., Kolb E. W.,Knox L., Mather J. C., Staggs S., Suntzeff N. B., 2006,ArXiv Astrophysics e-printsAmara A., Refregier A., 2004, MNRAS, 351, 375Amara A., R´efr´egier A., 2007, MNRAS, 381, 1018Blake C., Parkinson D., Bassett B., Glazebrook K., KunzM., Nichol R. C., 2006, MNRAS, 365, 255Desjacques V., Seljak U., 2010, Classical and QuantumGravity, 27, 124011 c (cid:13) , 000–000 Amara & Kitching
Eguchi K., et al 2003, Physical Review Letters, 90, 021802Huterer D., Starkman G., 2003, Physical Review Letters,90, 031301Huterer D., Turner M. S., 2001, Phys. Rev. D, 64, 123527Ishak M., Upadhye A., Spergel D. N., 2006, Phys. Rev. D,74, 043513Kim A. G., Linder E. V., Miquel R., Mostek N., 2004, MN-RAS, 347, 909Kitching T., Amara A., Rassat A., Refregier A., 2009,ArXiv e-printsKitching T. D., Amara A., 2009, MNRAS, 398, 2134Kitching T. D., Heavens A. F., Verde L., Serra P., Mel-chiorri A., 2008, Phys. Rev. D, 77, 103008Komatsu E., Dunkley J., Nolta M. R., Bennett C. L., GoldB., Hinshaw G., Jarosik N., Larson D., Limon M., Page L.,Spergel D. N., Halpern M., Hill R. S., Kogut A., MeyerS. S., Tucker G. S., Weiland J. L., Wollack E., WrightE. L., 2009, ApJS, 180, 330Kowalski M., et al 2008, ApJ, 686, 749Parkinson D., Blake C., Kunz M., Bassett B. A., NicholR. C., Glazebrook K., 2007, MNRAS, 377, 185Percival W. J., et al 2010, MNRAS, 401, 2148Pillepich A., Porciani C., Hahn O., 2010, MNRAS, 402, 191Press W. H., Teukolsky S. A., Vetterling W. T., FlanneryB. P., 2007, Numerical Recipes 3rd Edition: The Art ofScientific Computing. Cambridge University Press, NewYork, NY, USARassat A., Amara A., Amendola L., Castander F. J., Kitch-ing T., Kunz M., Refregier A., Wang Y., Weller J., 2008,ArXiv e-printsRefregier A., Amara A., Kitching T., Rassat A., 2008,ArXiv e-printsRefregier A., Amara A., Kitching T. D., Rassat A.,Scaramella R., Weller J., Euclid Imaging Consortium f. t.,2010, ArXiv e-printsTegmark M., Eisenstein D. J., Hu W., Kron R., 1998, ArXivAstrophysics e-printsTegmark M., Taylor A. N., Heavens A. F., 1997, ApJ, 480,22Thomas S. A., Abdalla F. B., Lahav O., 2009, ArXiv e-prints c (cid:13)000