R J Douglas

Pair-difference chi-squared statistics for Key Comparisons

Pair-difference chi-squared statistics are useful for analysing metrological consistency within a Key Comparison. We show how they relate to classical chi-squareds and how they can be used with full rigour, for any comparison of a scalar measurand, to compare the observed dispersion of results with the dispersion that would be expected on the basis of the claimed uncertainties. In several limits, the distributions of these pair-difference statistics are exact chi-squareds. For other cases, the Monte Carlo method can evaluate the distributions even in the presence of non-Gaussian uncertainty distributions, including the Student distribution to be construed when a participant has reported a degrees of freedom. Monte Carlo methods also treat inter-laboratory covariances in a transparent manner appropriate for metrology. Pair-difference chi-squared statistics are independent of the choice of a Key Comparison Reference Value (KCRV) and so may expedite the process of analysis, consensus building and publication for Key Comparisons. They are appropriate for judging pair metrology. We discuss them as a necessary, but not sufficient, test for a Key Comparison reported in the conventional way using a KCRV. The deficiencies of using solely the classical chi-squared test are discussed. A good remedy is available with pair-difference chi-squared statistics.

Chi-squared statistics for KCRV candidates

We examine chi-squared statistics that are appropriate for analysing the adequacy of different key comparison reference value (KCRV) candidates in accounting for the observed dispersion of results of a key comparison, about the candidate estimator and within the stated uncertainty claims. We extend the analysis to cover cases where the uncertainty budgets incorporate low degrees of freedom or have significant correlations. In this context, we discuss when it is important to view the KCRV as a method and not merely as a number. To use these statistics for the usual chi-squared tests of consistency, the required distributions (that can depart from the exact chi-squared distribution) can readily be evaluated by Monte Carlo simulation, for any KCRV algorithm that uses only the peer results of the comparison. Similarly, the effects of non-Gaussian distributions (such as the Student distribution implied when a participant has reported finite degrees of freedom) can be evaluated with the requisite precision.

A useful reflection

We discuss a simple, practical transformation between the probability density function (PDF) of the measurand and the associated PDF of the process for producing measurement results, both of which are used in metrology. This transformation is a reflection through the reported value and captures the essential reversal of skewness of these two perspectives for one-dimensional cases (i.e. any scalar quantity with no free parameters). It is useful for reconciling the skewness of the measurand and measurement-results perspectives, identifying cases where the choice of perspective should not matter and helping to treat the exceptional cases, involving PDF asymmetry, where it can matter.

Outlier rejection for the weighted-mean KCRV

We propose new criteria to help select a subset of peer results for determining a weighted-mean Key Comparison Reference Value (KCRV) for describing Key Comparisons supporting the CIPM Mutual Recognition Arrangement (MRA). Often the results are all well behaved: their scatter is in the range expected from the Gaussian probability distributions associated with their stated standard uncertainties. In this case, the KCRV would usually be the inverse-variance weighted mean of all eligible results, the maximum likelihood estimate of independent Gaussian probability distributions. If most—but not all—of the results are well behaved, the comparison may be better described in terms of a KCRV computed with ‘outliers’ excluded from the weighted mean. An outlier’s effects are quantified in terms of the logarithmic slope and logarithmic curvature of its associated probability density function in the vicinity of the consensus value. The outer tails of the probability distributions are generally not known with confidence, so two new criteria are suggested to help justify the identification of outliers: one based on uncertainty in the logarithmic slope, and the other based on uncertainty about whether the logarithmic curvature is negative definite.

Evaluation of uncertainty in grating pitch measurement by optical diffraction using Monte Carlo methods

Measurement of grating pitch by optical diffraction is one of the few methods currently available for establishing traceability to the definition of the meter on the nanoscale; therefore, understanding all aspects of the measurement is imperative for accurate dissemination of the SI meter. A method for evaluating the component of measurement uncertainty associated with coherent scattering in the diffractometer instrument is presented. The model equation for grating pitch calibration by optical diffraction is an example where Monte Carlo (MC) methods can vastly simplify evaluation of measurement uncertainty. This paper includes discussion of the practical aspects of implementing MC methods for evaluation of measurement uncertainty in grating pitch calibration by diffraction. Downloadable open-source software is demonstrated.

Measurement science and the linking of CIPM and regional key comparisons

A generalized formalism is presented for linking a Comite International de Poids et Mesures (CIPM) key comparison (KC) with similar regional KCs. The mechanism proposed provides a straightforward and practical means for reporting on the equivalence of the larger ensemble of national metrology institutes resulting from the linking of a regional metrology organization KC to the CIPM KC of the same measurand. Redundancy in the linking path which results when there is more than one linking lab is exploited for testing the statistical consistency of the linking relative to the claimed uncertainties. The proposed linking procedure is described and illustrated with a real-world practical example employing extended chi-squared-like statistics. A procedure for linking in the presence of artefact drift or other parametrized corrections is outlined.

Analysing redundancy in a line scale comparison using Monte Carlo methods

Supplement 1 to the ISO Guide to the Expression of Uncertainty in Measurement (GUM) outlines the application of Monte Carlo (MC) simulation to uncertainty propagation through the mathematical model of the measurement. The MC approach uses the information contained in the probability density functions associated with each of the influence parameters of the measurement to determine the probability density function of the output quantity. This paper discusses an approach to uncertainty evaluation for calibration of line scales amenable for application of MC methods. The model for the measurement uncertainty in line scale calibration is interpreted from the perspective of the line-scale user, namely a distance measurement between any two intervals of the line scale. For this purpose, the introduction of a consensus invariant assigned in an alternative interpretation of the data can also expedite international comparison analysis. The proposed method is illustrated using a practical real-world example from an international comparison of calibration of line scales.

Monte Carlo Modeling of Randomness

The evaluation of expressions of measurement uncertainty can be enhanced by Monte Carlo simulation techniques. For any claimed uncertainties that have been expressed (either explicitly or implicitly) as probability density functions, Monte Carlo techniques can propagate uncertainties through any measurement equation or algorithm – including statistical aggregates. These techniques are introduced in this chapter, and applied to examples drawn from measurement comparisons. Examples illustrate the power of the techniques in extending uncertainty analysis to cases where the applicability of the usual methods is in doubt, such as consistency testing using chi-squared-like statistical aggregates.

Avoiding a complex reference value

We address a difficulty in appreciating that the pair-difference chi-squared-like statistic (Steele et al 2002 Metrologia 39 269–77, Douglas and Steele 2006 Metrologia 43 89–97) is truly unmediated (i.e. independent of any reference value), namely the incorrect thought that this aggregating statistic might merely be hiding a reference value that has the same chi-squared. We show that the equation for this reference value commonly has no real solution. Regarding pair-difference statistics as unmediated statistics conveniently avoids any complex reference value, and offers a stark simplicity that exactly captures the simplest way in which measurements are really used in practical chains of metrological inference.

Precision comparisons and pooled presentations of probability

Inter-laboratory measurement comparisons can benefit from explicit graphical presentation of their probability density functions (PDFs). It is now as easy to create and manipulate these graphs as it is to use traditional error bars. These graphs improve clarity and generality when discussing uncertainties and their associated assumptions.