Daniel J. Klass

The accuracy of standardized patient presentation.

Summary. The accuracy of standardized patient clinical problem presentation was evaluated by videotape rating of a random sample of 839 student‐patient encounters, representing 88 patients, 27 cases and two university test sites. Patient‐student encounters were sampled from a collaborative inter‐university final‐year clinical examination of fourth‐year medical students which was conducted at the University of Manitoba and Southern Illinois University in 1987 and 1988. The accuracy, replicability and portability of standardized patient cases were evaluated. The average accuracy of patient presentation was 90.2% in 1987 and 93.4% in 1988. Perfect accuracy scores were obtained by 15 patients; however, 11 patients had average scores below 80% with the accuracy of presentation in some encounters being as low as 30%. There were significant differences in the accuracy score achieved by patients trained together for the same case in 6 of 35 possible comparisons. There was also a systematic trend for patients trained at Southern Illinois to be more accurate in their presentation than patients trained at the University of Manitoba. These differences were significant in 5 of the 15 cases used in the examination.

Sources of unreliability and bias in standardized‐patient rating

In tests of clinical competence, standardized patients (SPs) can be used to present the clinical problem and rate actions taken by the examinee in the patient encounter. Both these aspects of the “test”; have the potential to contribute to unreliability and bias in measurement. In 1987, two universities collaborated to develop and execute the same SP test to clinical clerks in their respective institutions. This provided us with the opportunity to evaluate rating bias attributable to test site and three sources of rating unreliability within the same population of raters: those attributable to inconsistencies within the same rater (within‐rater reliability), those attributable to inconsistencies between two raters trained in the same test site (between‐raters reliability—same site), and those attributable to inconsistencies between two raters trained in different test sites (between‐raters reliability—different sites). A stratified random sample of 537 of the 2,560 examinee‐patient encounters that occurre...

Influence of Physicians' Management and Communication Ability on Patients' Persistence With Antihypertensive Medication

BACKGROUND Less than 75% of people prescribed antihypertensive medication are still using treatment after 6 months. Physicians determine treatment, educate patients, manage side effects, and influence patient knowledge and motivation. Although physician communication ability likely influences persistence, little is known about the importance of medical management skills, even though these abilities can be enhanced through educational and practice interventions. The purpose of this study was to determine whether a physicians medical management and communication ability influence persistence with antihypertensive treatment. METHODS This was a population-based study of 13,205 hypertensive patients who started antihypertensive medication prescribed by a cohort of 645 physicians entering practice in Quebec, Canada, between 1993 and 2007. Medical Council of Canada licensing examination scores were used to assess medical management and communication ability. Population-based prescription and medical services databases were used to assess starting therapy, treatment changes, comorbidity, and persistence with antihypertensive treatment in the first 6 months. RESULTS Within 6 months after starting treatment, 2926 patients (22.2%) had discontinued all antihypertensive medication. The risk of nonpersistence was reduced for patients who were treated by physicians with better medical management (odds ratio per 2-SD increase in score, 0.74; 95% confidence interval, 0.63-0.87) and communication (0.88; 0.78-1.00) ability and with early therapy changes (odds ratio, 0.45; 95% confidence interval, 0.37-0.54), more follow-up visits, and nondiuretics as the initial choice of therapy. Medical management ability was responsible for preventing 15.8% (95% confidence interval, 7.5%-23.3%) of nonpersistence. CONCLUSION Better clinical decision-making and data collection skills and early modifications in therapy improve persistence with antihypertensive therapy.

Standardized patientsʼ accuracy in recording examineesʼ behaviors using checklists

No abstract available.

An expert-judgment approach to setting standards for a standardized-patient examination.

No abstract available.

Assessing post-encounter note documentation by examinees in a field test of a nationally administered standardized patient test.

The large-scale standardized patient (SP) test in this study assessed the clinical skills of fourth-year medical students in a series of clinical encounters targeting history taking, physical examination, communication, and interpersonal skills. Yearly large-scale field tests have been undertaken over the past seven years in preparation for national administration. The study reported here was conducted in 1998. Students are oriented to the test prior to completing up to 12 15-minute SP encounters (cases). Following each encounter, the SP records history elicited, counseling provided, or physical examination performed using an objective checklist developed by expert clinicians. The checklists may be thought of as a process measure, serving as a reflection of actual behaviors demonstrated by the candidate. Interpersonal skills are assessed using the Patient Perception Questionnaire (PPQ), a six-item instrument with a fivepoint Likert rating scale (uniform for every case). Following each encounter, students are given seven minutes to write a free-response Post-Encounter Note (PEN) (either a list of significant positive and negative history and physical findings or a written chart note documenting findings and counseling). The PEN is specifically tailored to reflect each case. There is no limit to the number of findings students may write. Patient management (diagnosis or therapeutic plans) and interpretation of diagnostic tests are not assessed in these PENs. The PENs potentially reflect a candidate’s ability to determine the most significant findings elicited from the encounter and to accurately record them. While numerous studies have examined the use of checklists with respect to fairness, security, and accuracy, there is limited research investigating the psychometric properties of PENs. Previous studies have examined appropriate methods for scoring the PEN. Soliciting global judgments from experts seems appealing because scores are derived from the expertise of practicing physicians, but global ratings can be unreliable unless the scoring task is highly structured and extensive standardized training is provided. From a national testing perspective, recruiting physicians to score the PENs for thousands of candidates may not be feasible. As a result, many researchers have favored the use of analytic keys to score PENs. A significant advantage of using such scoring keys is the fact that non-physicians can be trained to score the PENs with an accuracy level comparable to that of physicians. Research examining the usefulness of PENs with an SP test has suggested that these scores contribute valuable information to the assessment of clinical skills by providing unique information different from that derived from checklist scores. However, other research indicates that the chart audit scores should not replace the checklist entirely, since the information written by candidates in a simulated medical record may not provide a complete picture of events during an SP encounter. The inclusion of the PEN in an SP test is appealing. First, it is thought that PENs are relatively immune to within-site and crosssite effects. Also, they do not depend on the accurate recording of checklists by SPs. Additionally, threats to security are minimized because the PEN is a free-response instrument and does not reveal checklist content or other exam material. However, before the PEN can be used in large-scale testing, it is important to determine whether the PEN is a reflection of the checklist or whether the PEN contributes unique information about a student’s ability to synthesize and record medical information. The purpose of this study was, therefore, to investigate the relationship between entries recorded in the PEN and actions captured on the checklist. It is hoped that the results of this study will help determine how to best incorporate PEN information into a composite score.

Modeling the Effects of a Test Security Breach on a Large-scale Standardized Patient Examination with a Sample of International Medical Graduates

2 The validity of these score-based interpretations can be weakened by several test-related phenomena, including breaches to the security of the environment. The impacts of various forms of test security breaches need to be clearly ad- dressed to determine the extent to which a priori knowledge of materials might provide an undue advantage to subgroups of ex- aminees. This evidence also ensures that misinterpretation of scores is minimized on the part of the user. This task is especially crucial with performance-based tests such as standardized patient (SP) ex- aminations, given the typically limited nature of case banks, the long exposure of items/cases, and the high costs associated with developing these types of assessments. 3

Assessing the factor structure of a nationally administered standardized patient examination.

No abstract available.

Differential item functioning in checklist items from a standardized-patient-based examination.

No abstract available.

Regression-Based Weighting of Items on Standardized Patient Checklists

The use of checklists for scoring standardized patient evaluations reduces the rating task to recording whether a defined behaviour was or was not displayed. Although this may enhance objectivity, checklist-based scoring has the potential limitation that it may fail to account for the complexity of the judgment process used by experts. Minimally, it is clear that experts may consider the behaviours represented by some checklist items to be more important than others. The research described in this paper examines the potential to increase the correspondence between checklist scores and clinician ratings by weighting checklist items using regression-derived item weights. Results show that the expected increase in correspondence between checklist scores and clinician ratings occurred for all cases in which the correlation between the unweighted checklist scores and the ratings was less than. 92. Cross-validation of the results with an independent set of ratings is provided as are generaliz-ability analyses of the ratings, weighted, and unweighted scores.