James R. Boex

Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7

Review date: 1966 to April 2003. Background and context: There is a basis for the assumption that feedback can be used to enhance physicians’ performance. Nevertheless, the findings of empirical studies of the impact of feedback on clinical performance have been equivocal. Objectives: To summarize evidence related to the impact of assessment and feedback on physicians’ clinical performance. Search strategy: The authors searched the literature from 1966 to 2003 using MEDLINE, HealthSTAR, the Science Citation Index and eight other electronic databases. A total of 3702 citations were identified. Inclusion and exclusion criteria: Empirical studies were selected involving the baseline measurement of physicians’ performance and follow-up measurement after they received summaries of their performance. Data extraction: Data were extracted on research design, sample, dependent and independent variables using a written protocol. Data synthesis: A group of 220 studies involving primary data collection was identified. However, only 41 met all selection criteria and evaluated the independent effect of feedback on physician performance. Of these, 32 (74%) demonstrated a positive impact. Feedback was more likely to be effective when provided by an authoritative source over an extended period of time. Another subset of 132 studies examined the effect of feedback combined with other interventions such as educational programmes, practice guidelines and reminders. Of these, 106 studies (77%) demonstrated a positive impact. Two additional subsets of 29 feedback studies involving resident physicians in training and 18 studies examining proxy measures of physician performance across clinical sites or groups of patients were reviewed. The majority of these two subsets also reported that feedback had positive effects on performance. Headline results: Feedback can change physicians’ clinical performance when provided systematically over multiple years by an authoritative, credible source. Conclusions: The effects of formal assessment and feedback on physician performance are influenced by the source and duration of feedback. Other factors, such as physicians’ active involvement in the process, the amount of information reported, the timing and amount of feedback, and other concurrent interventions, such as education, guidelines, reminder systems and incentives, also appear to be important. However, the independent contributions of these interventions have not been well documented in controlled studies. It is recommended that the designers of future theoretical as well as practical studies of feedback separate the effects of feedback from other concurrent interventions.

Measuring Professionalism: A Review of Studies with Instruments Reported in the Literature between 1982 and 2002

Purpose To describe the measurement properties of instruments reported in the literature that faculty might use to measure professionalism in medical students and residents. Method The authors reviewed studies published between 1982 and 2002 that had been located using Medline and four other databases. A national panel of 12 experts in measurement and research in medical education extracted data from research reports using a structured critique form. Results A total of 134 empirical studies related to the concept of professionalism were identified. The content of 114 involved specific elements of professionalism, such as ethics, humanism, and multiculturalism, or associated phenomena in the educational environment such as abuse and cheating. Few studies addressed professionalism as a comprehensive construct (11 studies) or as a distinct facet of clinical competence (nine studies). The purpose of 109 studies was research or program evaluation, rather than summative or formative assessment. Sixty five used self-administered instruments with no independent observation of the participants’ professional behavior. Evidence of reliability was reported in 62 studies. Although content validity was reported in 86 studies, only 34 provided strong evidence. Evidence of concurrent or predictive validity was provided in 43 and 16 studies, respectively. Conclusions There are few well-documented studies of instruments that can be used to measure professionalism in formative or summative evaluation. When evaluating the tools described in published research it is essential for faculty to look critically for evidence related to the three fundamental measurement properties of content validity, reliability, and practicality.

The Ambulatory Long-Block: An Accreditation Council for Graduate Medical Education (ACGME) Educational Innovations Project (EIP)

IntroductionHistorical bias toward service-oriented inpatient graduate medical education experiences has hindered both resident education and care of patients in the ambulatory setting.AimDescribe and evaluate a residency redesign intended to improve the ambulatory experience for residents and patients.SettingCategorical Internal Medicine resident ambulatory practice at the University of Cincinnati Academic Health Center.Program DescriptionWe created a year-long continuous ambulatory group-practice experience separated from traditional inpatient responsibilities called the long block as an Accreditation Council for Graduate Medical Education Educational Innovations Project. The practice adopted the Chronic Care Model and residents received extensive instruction in quality improvement and interprofessional teams.Program EvaluationThe long block was associated with significant increases in resident and patient satisfaction as well as improvement in multiple quality process and outcome measures. Continuity and no-show rates also improved.DiscussionAn ambulatory long block can be associated with improvements in resident and patient satisfaction, quality measures, and no-show rates. Future research should be done to determine effects of the long block on education and patient care in the long term, and elucidate which aspects of the long block most contribute to improvement.

Measuring the costs of primary care education in the ambulatory setting.

In 1995, the authors obtained cost, operations, and educational activity data from 98 ambulatory care sites across the United States in which primary care teaching was occurring and compared those data with the corresponding data from 84 ambulatory care sites where no teaching was going on. The teaching sites in the sample were found to have 24–36% higher operating costs than the non-teaching sites. This overall difference in costs is approximately the same difference in costs earlier estimated for university teaching hospitals compared with non-teaching hospitals. These costs are shared by all involved in the ambulatory education process: sponsors, sites, and faculty. In a related finding, the authors discovered that 30–50% of all ambulatory care sites thought not to be involved in education are in fact teaching at a high level of involvement. Further research into not only the costs but the value of education in the clinical setting is encouraged. The authors also hope that the publication of this report will encourage accrediting bodies and professional organizations to improve the information available about ambulatory care training in general.

Factors Contributing to the Variability of Direct Costs for Graduate Medical Education in Teaching Hospitals.

Medicares support of graduate medical education includes funds allocated to the direct costs of graduate medical education: housestaff stipends and benefits, faculty costs, and related educational costs such as classroom space. As reimbursed through the mechanism called the direct graduate medical education (DGME) pass-through, these direct costs have been reported to vary widely from one teaching hospital to another, with little explanation for this variation being available. Based on a national survey of 69 teaching hospitals–principally affiliated community teaching hospitals–the author suggests that a major cause for the variation in these costs might be found in their faculty-expenses component. It is further suggested that economies of scale may provide some clue as to the variability of these costs. The author also reports lower DGME costs for the survey sample than for the national sample, and suggests that the fact that community teaching hospital faculties include a significant volunteer component may account for some of these savings.

Understanding the costs of ambulatory care training.

While patient care has been shifting to the ambulatory setting, the education of health care professionals has remained essentially hospital-based. One factor discouraging the movement of training into community-based ambulatory settings is the lack of understanding of what the costs of such training are and how these costs might be offset. The authors describe a model for ambulatory care training that makes it easier to generalize about to quantify its educational costs. Since ambulatory care training does not exist in a vacuum separate from inpatient education, the model is compatible with the way hospital-based education costs are derived. Thus, the models elements can be integrated with comparable hospital-based training cost elements in a straightforward way to allow a total-costing approach. The model is built around two major sets of variables affecting cost. The first comprises three types of costs--direct, indirect, and infrastructure--and the second consists of factors related to the training site and factors related to the educational activities of the training. The model is constructed to show the various major ways these two sets of variables can influence training costs. With direct Medicare funding for some ambulatory-setting-based education pending, and with other regulatory and market dynamics already in play, it is important that educators, managers, and policymakers understand how costs, the characteristics of the training, and the characteristics of the setting interact. This model should assist them. Without generalizable cost estimates, realistic reimbursement policies and financial incentives cannot be formulated, either in the broad public policy context or in simple direct negotiations between sites and sponsors.

Medical care use by treated and untreated substance abusing Medicaid patients

Medicaid reimbursement costs for county residents at least 18 years old who used a treatment service (n = 1043) and residents who were Medicaid enrollees with a substance abuse diagnosis but who did not receive treatment (n = 2125) were compared. Untreated patients were more likely to be male (47% vs. 39%), white (56% vs. 45%), and older (39.7 yrs. +/- 13 SD vs. 35.5 yrs +/- 10 SD). The average monthly Medicaid costs (

Understanding the value added to clinical care by educational activities. Value of Education Research Group.

257) for the untreated were higher in the year prior to identification than were costs (

Multisource Feedback in the Ambulatory Setting

207) for the treated. The monthly costs in the six months following identification were

Academic Health Centers and Public Health Departments: Partnership Matters

761 for the untreated and