Timothy J. Wood

Validation of a Decision Regret Scale

Background. As patients become more involved in health care decisions, there may be greater opportunity for decision regret. The authors could not find a validated, reliable tool for measuring regret after health care decisions. Methods. A5- item scale was administered to 4 patient groups making different health care decisions. Convergent validity was deter- mined by examining the scales correlation with satisfaction measures, decisional conflict, and health outcome measures. Results. The scale showed good internal consistency (Cronbachs = 0.81 to 0.92). It correlated strongly with decision satisfaction (r = -0.40 to -0.60), decisional conflict (r = 0.31 to 0.52), and overall rated quality of life (r = -0.25 to - 0.27). Groups differing on feelings about a decision also differed on rated regret: F(2, 190) = 31.1, P < 0.001. Regret was greater among those who changed their decisions than those who did not, t(175) = 16.11, P < 0.001. Conclusions. The scale is a useful indicator of health care decision regret at a given point in time.

Measuring Acceptability of Clinical Decision Rules: Validation of the Ottawa Acceptability of Decision Rules Instrument (OADRI) in Four Countries

Background. Clinical decision rules can benefit clinicians, patients, and health systems, but they involve considerable up-front development costs and must be acceptable to the target audience. No existing instrument measures the acceptability of a rule. The current study validated such an instrument. Methods. The authors administered the Ottawa Acceptability of Decision Rules Instrument (OADRI) via postal survey to emergency physicians from 4 regions (Australasia, Canada, United Kingdom, and United States), in the context of 2 recently developed rules, the Canadian C-Spine Rule (C-Spine) and the Canadian CT Head Rule (CT-Head). Construct validity of the 12-item instrument was evaluated by hypothesis testing. Results. As predicted by a priori hypotheses, OADRI scores were 1) higher among rule users than nonusers, 2) higher among those using the rule ‘‘all of the time’’ v. ‘‘most of the time’’ v. ‘‘some of the time,’’ and 3) higher among rule nonusers who would consider using a rule v. those who would not. We also examined explicit reasons given by respondents who said they would not use these rules. Items in the OADRI accounted for 85.5% (C- Spine) and 90.2% (CT-Head) of the reasons given for not considering a rule acceptable. Conclusions. The OADRI is a simple, 12-item instrument that evaluates rule acceptability among clinicians. Potential uses include comparing multiple ‘‘protorules’’ during development, examining acceptability of a rule to a new audience prior to implementation, indicating barriers to rule use addressable by knowledge translation interventions, and potentially serving as a proxy measure for future rule use.

Diversion of attention in everyday concept learning: Identification in the service of use

Many people tend to believe that natural categories have perfectly predictive defining features. They do not easily accept the family resemblance view that the features characteristic of a category are not individually sufficient to predict the category. However, common category-learning tasks do not produce thissimpler-than-it-is belief. If there is no simple classification principle in a task, the participants know that fact and can report it. We argue that most category-learning tasks in which family resemblance categories are used fail to produce the everyday simpler-than-it-is belief because they encourage analysis of identification criteria during training. To simulate the learning occurring in many natural circumstances, we developed a procedure in which participants’ analytic activity is diverted from the way in which the stimuli are identified to the use to which the stimuli will be put. Finally, we discuss the prevalence of thisdiverted analysis in everyday categorization.

Self-monitoring and its relationship to medical knowledge

In the domain of self-assessment, researchers have begun to draw distinctions between summative self-assessment activities (i.e., making an overall judgment of one’s ability in a particular domain) and self-monitoring processes (i.e., an “in the moment” awareness of whether one has the necessary knowledge or skills to address a specific problem with which one is faced). Indeed, previous research has shown that, when responding to both short answer and multiple choice questions, individuals are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test. These studies, however, were conducted in the context of low-stakes tests of general “trivia”. The purpose of the present study was to further this line of research by investigating the relationship between self-monitoring and performance in the context of a high stakes test assessing medical knowledge. Using a recent administration of the Medical Council of Canada Qualifying Examination Part I, we examined three measures intended to capture self-monitoring: (1) the time taken to respond to each question, (2) the number of questions a candidate flagged as needing to be considered further, and (3) the likelihood of changing one’s initial answer. Differences in these measures as a function of the accuracy of the candidate’s response were treated as indices of each candidate’s ability to judge his or her likelihood of responding correctly. The three self-monitoring indices were compared for candidates at three different levels of overall performance on the exam. Relative to correct responses, when examinees initially responded incorrectly, they spent more time answering the question, were more likely to flag the question for future consideration, and were more likely to change their answer before committing to a final answer. These measures of self-monitoring were modulated by candidate performance in that high performing examinees showed greater differences on these indices relative to poor performing examinees. Furthermore, reliability analyses suggest that these difference measures hold promise for reliably differentiating self-monitoring at the level of individuals, at least within a given content area. The results suggest that examinees were self-monitoring their knowledge and skills on a question by question basis and altering their behavior appropriately in the moment. High performing individuals showed stronger evidence of accurate self-monitoring than did low performing individuals and the reliability of these measures suggests that they have the potential to differentiate between individuals. How these findings relate to performance in actual clinical settings remains to be seen.

The effect of reused questions on repeat examinees

Abstract Reusing questions on an examination is a concern because test administrators do not want to unfairly aid examinees by exposing them to questions they have seen on previous examinations. The purpose of this study was to investigate the effect that prior exposure of questions has on the performance of repeat examinees. Two recent administrations of an examination repeated a block of 36 multiple-choice questions. Scores for 130 repeat examinees were analyzed. Examinee ability estimates on reused questions increased but estimates on non-reused questions also increased by an equal amount. Subsequent analyses compared the match between options chosen on the first and second attempts. There is a tendency to choose the same option but it does not appear to be due to examinees remembering questions. Repeat examinees do not appear to be advantaged by seeing reused questions.

Testing the validity of a scenario-based questionnaire to assess the ethical sensitivity of undergraduate medical students

Background: Although medical educators acknowledge the importance of ethics in medical training, there are few validated instruments to assess ethical decision-making. One instrument is the Ethics in Health Care Questionnaire – version 2 (EHCQ-2). The instrument consists of 12 scenarios, each posing an ethical problem in health care, and asking for a decision and rationale. The responses are subjectively scored in four domains: response, issue identification, issue sophistication, and values. Goals: This study was intended to examine the inter-rater and inter-case reliability of the AHCQ-2 and validity against a national licensing examination of the EHCQ-2 in an international sample. Methods: A total of 20 final year McMaster students and 45 final year Glasgow students participated in the study. All questionnaires were scored by multiple raters. Generalizability theory was used to examine inter-rater, inter-case and overall test reliability. Validity was assessed by comparing EHCQ-2 scores with scores on the Canadian written licensing examination, both total score and score for the ethics subsection. Results: For both samples, reliability was quite low. Except for the first task, which is multiple choice, inter-rater reliability was 0.08–0.54, and inter-case reliability was 0.14–0.61. Overall test reliability was 0.12–0.54. Correlation between EHCQ-2 task scores and the licensing examination scores ranged from 0.07 to 0.40; there was no evidence that the correlation was higher with the ethics subsection. Conclusions: The reliability and validity of the measure remains quite low, consistent with other measures of ethical decision-making. However, this does not limit the utility of the instrument as a tool to generate discussion on ethical issues in medicine.

How clinical features are presented matters to weaker diagnosticians

Medical Education 2010: 44: 775–785

Two models of raters in a structured oral examination: does it make a difference?

Oral examinations have become more standardized over recent years. Traditionally a small number of raters were used for this type of examination. Past studies suggested that more raters should improve reliability. We compared the results of a multi-station structured oral examination using two different rater models, those based in a station, (station-specific raters), and those who follow a candidate throughout the entire examination, (candidate-specific raters).Two station-specific and two candidate-specific raters simultaneously evaluated internal medicine residents’ performance at each station. No significant differences were found in examination scores. Reliability was higher for the candidate-specific raters. Inter-rater reliability, internal consistency and a study of station inter-correlations suggested that a halo effect may be present for candidates examined by candidate-specific raters. This study suggests that although the model of candidate-specific raters was more reliable than the model of station-specific raters for the overall examination, the presence of a halo effect may influence individual examination outcomes.

Why do physicians volunteer to be OSCE examiners

Recruitment of physician examiners for an objective structured clinical examination (OSCE) can be difficult. The following study will explore reasons why physicians volunteer their time to be OSCE examiners. A questionnaire was collected from 110 examiners including a fourth year formative student OSCE (SO) (nu2009=u200949), formative internal medicine OSCE (IM) (nu2009=u200921) and the Medical Council of Canada Qualifying Exam Part II (MCCQE II) (nu2009=u200940). A 5-point Likert scale was used. Statements with high mean ratings overall included: enjoy being an examiner (4.05), gain insights into learners’ skills and knowledge (4.27), and examine out of a sense of duty (4.10). The MCC participants produced higher ratings (pu2009<u20090.05). Overall, OSCE examiners volunteer their time because they enjoy the experience, feel a sense of duty and gain insight into learners’ skills and knowledge. The MCC examiners appear to value the experience more. The ability to provide feedback and the provision of CME credits were not significant factors for increasing examiner satisfaction.

Assessing change in clinical teaching skills: are we up for the challenge?

Background: The faculty development community has been challenged to more rigorously assess program impact and move beyond traditional outcomes of knowledge tests and self ratings. Purpose: The purpose was to (a) assess our ability to measure supervisors feedback skills as demonstrated in a clinical setting and (b) compare the results with traditional outcome measures of faculty development interventions. Methods: A pre–post study design was used. Resident and expert ratings of supervisors demonstrated feedback skills were compared with traditional outcomes, including a knowledge test and participant self-evaluation. Results: Pre–post knowledge increased significantly (pre = 61%, post = 85%; p < .001) as did participants self-evaluation scores (pre = 4.13, post = 4.79; p < .001). Participants self-evaluations were moderately to poorly correlated with resident (pre r = .20, post r = .08) and expert ratings (pre r = .43, post r = −.52). Residents and experts would need to evaluate 110 and 200 participants, respectively, to reach significance. Conclusions: It is possible to measure feedback skills in a clinical setting. Although traditional outcome measures show a significant effect, demonstrating change in teaching behaviors used in practice will require larger scale studies than typically undertaken currently.