Geoff Norman

Measuring change over time: assessing the usefulness of evaluative instruments.

Reliability, the ratio of the variance attributable to true differences among subjects to the total variance, is an important attribute of psychometric measures. However, it is possible for instruments to be reliable, but unresponsive to change; conversely, they may show poor reliability but excellent responsiveness. This is especially true for instruments in which items are tailored to the individual respondent. Therefore, we suggest a new index of responsiveness to assess the usefulness of instruments designed to measure change over time. This statistic, which relates the minimal clinically important difference to the variability in stable subjects, has direct sample size implications. Responsiveness should join reliability and validity as necessary requirements for instruments designed primarily to measure change over time.

Likert scales, levels of measurement and the ''laws'' of statistics

Reviewers of research reports frequently criticize the choice of statistical methods. While some of these criticisms are well-founded, frequently the use of various parametric methods such as analysis of variance, regression, correlation are faulted because: (a) the sample size is too small, (b) the data may not be normally distributed, or (c) The data are from Likert scales, which are ordinal, so parametric statistics cannot be used. In this paper, I dissect these arguments, and show that many studies, dating back to the 1930s consistently show that parametric statistics are robust with respect to violations of these assumptions. Hence, challenges like those above are unfounded, and parametric methods can be utilized without concern for “getting the wrong answer”.

Non-analytical models of clinical reasoning: the role of experience

Objective  This paper aims to summarise the evidence supporting the role of experience‐based, non‐analytic reasoning (NAR) or pattern recognition as a central feature of expert medical diagnosis.

The minimal relationship between simulation fidelity and transfer of learning

Medical Education 2012

Problem-solving skills, solving problems and problem-based learning.

Summary. This paper reviews the empirical evidence in support of the three concepts in the title. To the extent that a skill should be a general strategy, applicable in a variety of situations, and independent of the specific knowledge of the situation, there is little evidence that problem‐solving skills, as described and measured in medical education, possess these characteristics. Instead there is an accumulation of evidence that expert problem‐solving in medicine is dependent on (1) a wealth of prior specific experiences which can be used in routine solution of problems by pattern recognition processes, and (2) elaborated conceptual knowledge applicable to the occasional problematic situation.

Pitfalls in the pursuit of objectivity: issues of reliability

Summary. Objectivity has been one of the hallmarks in the assessment of clinical competence in recent decades. A consistent shift can be noticed in which subjective measures are being replaced by objective measurement methods. In the transition from subjective to objective methods trade‐offs are involved, both in the effort expended and in the range of behaviours assessed. The issue of the presumed superiority of objective measures is addressed in two successive papers.

How medical students learn spatial anatomy

Despite its importance in clinical tasks, spatial learning is poorly understood. We did a randomised, single-blind study on 146 medical students to assess the effectiveness of student-controlled multiple views on the spatial learning process. High spatial ability of the student, and self-directed examination of an object from multiple different perspectives improves spatial learning.

Clinical problem‐solving by medical students: a cross‐sectional and longitudinal analysis

The evolution of clinical reasoning in medical students was studied. A cross‐sectional sample consisted of randomly‐selected medical students from three classes. Additionally, twenty‐two students were observed at yearly intervals from the preclerkship period to the first post‐graduate year. Subjects were observed in a clinical examination of a simulated patient, and their thought processes were abstracted from a ‘stimulated recall’ of the videotaped encounter.

RCT = results confounded and trivial: the perils of grand educational experiments.

In a recent issue of Medical Education, Torgerson made an impassioned plea for greater use of randomized trial methods in medical education. It seems that her claim is based on two premises: 1 ‘The randomised trial is generally acknowledged as the gold standard of evaluative research’, so we should all be doing it; and 2 Any argument against doing more randomised clinical trials (RCTs) amounts to a display of ignorance of RCT methods on the part of the nay-sayer. In buttressing these arguments, she makes some astonishing claims about the strengths and weaknesses of the method. One which is central to the difference between her perspective and my own is embodied in the phrase:

Pitfalls in the pursuit of objectivity: issues of validity, efficiency and acceptability.

Summary. In a previous article the distinction is made between objectivity and objectification. Objectivity is considered a generic goal of measurement, marked by freedom of subjective influences in general, whereas the latter term is used to describe strategies to reduce measurement error. A survey of several studies indicated that objectified methods are not intrinsically more reliable than subjective measures.