Discussion of "Search for the Wreckage of Air France Flight AF 447"
aa r X i v : . [ s t a t . M E ] M a y Statistical Science (cid:13)
Institute of Mathematical Statistics, 2014
Discussion
A. H. Welsh
This collection of papers gathering and promot-ing highly successful applications of Statistics is agood antidote for anyone feeling somewhat defen-sive about Statistics. The focus on the successful useof Bayesian methods has produced a powerful andstimulating set of stories; the Editors and Authorsare all to be congratulated on their successful effortsto bring out the stories behind these analyses. Thepapers are relatively short (as was required by theEditors) and a good measure of their success is thatthey both stand alone and motivate the reader tofollow up and read the original papers.The article on the search for the wreckage of flightAF 447 (Stone et al.) is fascinating. The descrip-tion of the careful and detailed thinking about whatmight have happened, the evaluation and inclusionof relevant empirical evidence to quantify the possi-ble scenarios and the final success of the analysis inassigning substantial posterior probability to wherethe wreckage was ultimately found are all inspir-ing. Like many inspiring articles, it challenges us tothink about both the difficult issues of the partic-ular problem considered and general issues aboutthe overall approach. I think a Bayesian analysis ishighly appropriate for this problem, but it is not soeasy to explain why and it is clear that, as always,the analysis itself has to be done extremely well.One motivation for doing a Bayesian analysis forthis problem (and one that is commonly articulated)is that the event in question is unique so it is notmeaningful to think about replications. This is notreally convincing because hypothetical replicationsare hypothetical whether they are conceived of foran event that is extremely rare (and in the extremehappens once) or for events that occur frequently.Moreover, it turns out later that nine past crashes
A. H. Welsh is E. J. Hannan Professor of Statistics,The Australian National University, Canberra ACT0200, Australia e-mail: [email protected].
This is an electronic reprint of the original articlepublished by the Institute of Mathematical Statistics in
Statistical Science , 2014, Vol. 29, No. 1, 101–102. Thisreprint differs from the original in pagination andtypographic detail. were deemed similar enough to be used to provideinformation for constructing the prior, making it dif-ficult to argue that the event really is unique.Another widely used motivation for Bayesiananalysis is that it propagates the uncertainty cor-rectly. This is true and important, but it is also truethat it propagates only the uncertainties that we de-cide to include in the model. We make choices overwhat uncertainties to include and we also make rela-tively arbitrary choices which we subsequently treatas fixed. For example, were the uncertainties in theweights for the different scenarios or the chosen α propagated through to the conclusion? As a prac-tical matter, I do not believe we can or should tryto propagate all uncertainty, simply that we shouldnot get too carried away and forget about aspects wehave treated as certain. This highlights the fact thatthe Bayesian approach is a tool that is extremelyuseful for combining the quantitative informationwe choose to use and are able to express in terms ofdistributions but which, like any tool, needs to beused well to be effective; the tool on its own doesnot solve the problem but needs to be applied byhighly skilled people.The four unsuccessful searches that preceded thefinal, successful search highlight some of the issues.They too used assumptions and information to se-lect the search location. Presumably they did notuse a Bayesian analysis? If they did not (and it isnot really possible with the benefit of hindsight togo back and redo this fairly), differences betweenthe particular techniques used may be outweighedby differences in the information and beliefs thatfed into the analysis. For example, the fourth searchbased on possible drift concentrated in a small rect-angle relatively far from the actual crash site. Woulda Bayesian analysis based on the information used tocome up with that search rectangle have produceddifferent results? It is difficult to be sure from themaps but it looks like a passive acoustic search ac-tually covered the crash site but that the wreckagewas not discovered. We can interpret this as mea-surement error or as using an incorrect prior. Thesearchers tried to find the sonar beacons, not real-izing that these had failed and were not operating. A. H. WELSH
The successful search both allowed for this possi-bility (at least by not ruling out that area as hav-ing been previously searched) and, because so muchtime had elapsed that the beacons could not havebeen expected to still work, adopted different tech-nology in the search. Had they adopted the beliefthat the area had been searched so the wreckagecould not be there and built this into the prior, itwould not have been found. Thus, it was crucial toadopt the correct beliefs to end up with the right re-sult. The point is that the tool had to be used welland the credit is due to the users rather than simplythe tool.In their very interesting paper on managing Balticsalmon, Kuikka et al. make the point that Bayesianmethods make it possible to combine “relevant datafrom many sources.” The paper explicitly acknowl-edges the role of politics in salmon management andthe need to combine empirical data with “data” thatis too difficult or expensive to ever be collected. Theword relevant is critical here since irrelevant datamay at best just increase uncertainty and at worstlead to seriously wrong answers. The choice of whatis relevant or not depends ultimately on the userand is not an automatic property of the approach.Kuikka et al. also make the point that biologicallyrealistic models for salmon involve too many param-eters to fit without using informative priors. This ismentioned again in Carroll’s intriguing paper on di-etary consumption; Bayesian computations can beused to fit models that frequentist methods cannotfit. Running a Bayesian computation will producenumbers but, as in any computation, we need toconvince ourselves that the numbers are meaningfulbefore we use and interpret them. In particular, it isimportant to understand clearly whether the modelis identifiable or not, whether the model is incor-rect in some important way (so the computationalissues reflect lack of fit) and the extent to whichthe prior is driving the analysis. The fact that thesequestions are not easy to answer with complicatedmodels and high-dimensional parameter spaces doesnot lessen the importance of trying. Identifiability isimportant because it is resolved by using informa-tive priors which regularize the likelihood and en-able the model to be fitted; even vague priors canbe informative in this context. There is no problemwith using informative priors but we need to knowwhen the priors are informative, particularly whenthey are so informative that the posterior is essen-tially the prior. Conceptually, this may not be sodifferent from the frequentist approach of imposing nonestimable constraints on the parameters. A dif-ferent kind of identifiability issue arises in Bayesianhistory matching (Vernon et al.) because it is pos-sible that different scenarios or models can lead tothe same observable data, particularly when this isa single slice in time. Here, finding matching simula-tions seems only part of the really difficult scientificproblem being considered.Another reason a model may be difficult to fit isthat it does not describe the data. Forcing it to “fit,”for example, by switching to a Bayesian analysis,may not be the best response. It is difficult to checkcomplicated models, particularly hierarchical mod-els with latent variables, measurement error, missingdata, etc., but using an incorrect model may be aconcern when the model proves difficult to fit.A challenging issue acknowledged in Carroll is theissue of using survey weights in a Bayesian analy-sis. We can think about this as a way of estimatingthe likelihood by the pseudo-likelihood and then us-ing this estimated likelihood in a regular Bayesiananalysis. This does involve a combination of design-based and model-based approaches which requiredifferent conditioning but, somewhat like approx-imate Bayesian computation (ABC), it might beviewed as a pragmatic approach to solving difficultproblems. It is not clear what the Bayesian costsand benefits are; in frequentist analysis, Chamberset al. (2012) show that pseudo-likelihood estimationis less efficient than maximum likelihood estimationso there is some loss of information. Constructingthe likelihood requires including all the design vari-ables in the model. Aside from the fact that, incontrast to the survey weights, the design variablesare not usually available to secondary analysts, thestudy from which the data are taken (NHANES)uses a complicated design (with several nested levelsof cluster sampling) which it would not be straight-forward to incorporate into the model. Moreover,making the model more complicated may increasethe computational difficulties of fitting the model.The use of pseudo-likelihood in Bayesian analysisdefinitely needs research into its meaning and con-sequences before we can consider it with equanim-ity. REFERENCE
Chambers, R. L. , Steel, D. G. , Wang, S. and
Welsh, A. (2012).
Maximum Likelihood Estimation for Sample Sur-veys . Monographs on Statistics and Applied Probability125