Douglas Sinclair

Challenges in multisource feedback: intended and unintended outcomes

Context  Multisource feedback (MSF) is a type of formative assessment intended to guide learning and performance change. However, in earlier research, some doctors questioned its validity and did not use it for improvement, raising questions about its consequential validity (i.e. its ability to produce intended outcomes related to learning and change). The purpose of this qualitative study was to increase understanding of the consequential validity of MSF by exploring how doctors used their feedback and the conditions influencing this use.

Learning in practice: experiences and perceptions of high-scoring physicians.

Purpose To increase understanding of informal learning in practice (e.g., consulting with colleagues, reading journals) through exploring the experiences and perceptions of physicians perceived to be performing well. Objectives were to find out how physicians learned in practice and maintained their competence, and how they learned about the communication skills domain specifically. Method Of 142 family physicians participating in a formal multisource feedback (360-degree) formative assessment, 25 receiving high scores were invited to participate in interviews conducted in 2003 at Dalhousie University Faculty of Medicine. Twelve responded. Interviews were 1.5 hours each, recorded, transcribed, and analyzed by the research team using accepted qualitative procedures. Results While formal learning appeared important to most, informal learning, especially through patients and colleagues, appeared to be fundamental. The physicians appeared to learn intentionally from practice and work experiences, and reflection appeared integral to learning and monitoring the impact of learning. Two findings were surprising: participants’ conceptions of competence and perceptions that communication skills were innate rather than learned. Conclusions These physicians’ ways of intentional learning from practice concur with current models of informal learning. However, informal learning is largely unrecognized by formal institutions. Additionally, the physicians did not in general share notions of professional competence held by educators and others in authority. These findings suggest the need to make implicit content and learning processes more explicit. Additional research areas include exploring whether physicians across the range of performance levels demonstrate similar processes of reflective learning.

Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study.

Purpose. To describe responses of family physicians, their medical colleagues, and coworker raters to a multisource feedback assessment process. Method. Data collection tools included multisource feedback self-assessment and medical colleague, coworker, and patient rating forms; and program evaluation physician and rater questionnaires. Results. The pilot study included 142 physicians and their raters, with 113 (80%) physicians completing evaluations. Positive correlations were found between familiarity scores and medical colleague and coworker mean ratings. Peer medical colleagues were significantly more familiar with physicians than were consultants. Consultants were unable to rate items most frequently. Physicians disagreed with colleague feedback more frequently. Agreement was positively correlated with scores. Conclusions. Familiarity, ability to observe physicians appropriately to rate them, and physicians’ responses to feedback are factors to consider when multisource feedback is used.

The Canadian Triage and Acuity Scale for Children: A Prospective Multicenter Evaluation

STUDY OBJECTIVE The aims of the study are to measure both the interrater agreement of nurses using the Canadian Triage and Acuity Scale in children and the validity of the scale as measured by the correlation between triage level and proxy markers of severity. METHODS This was a prospective multicenter study of the reliability and construct validity of the Canadian Triage and Acuity Scale in 9 tertiary care pediatric emergency departments (EDs) across Canada during 2009 to 2010. Participants were a sample of children initially triaged as Canadian Triage and Acuity Scale level 2 (emergency) to level 5 (nonurgent). Participants were recruited immediately after their initial triage to undergo a second triage assessment by the research nurse. Both triages were performed blinded to the other. The primary outcome measures were the interrater agreement between the 2 nurses and the association between triage level and hospitalization. Secondary outcome measures were the association between triage level and health resource use and length of stay in the ED. RESULTS A total of 1,564 patients were approached and 1,464 consented. The overall interrater agreement was good, as demonstrated by a quadratic weighted κ score of 0.74 (95% confidence interval 0.71 to 0.76). Hospitalization proportions were 30%, 8.3%, 2.3%, and 2.2% for patients triaged at levels 2, 3, 4, and 5, respectively. There was also a strong association between triage levels and use of health care resources and length of stay. CONCLUSION The Canadian Triage and Acuity Scale demonstrates a good interrater agreement between nurses across multiple pediatric EDs and is a valid triage tool, as demonstrated by its good association with markers of severity.

Emergency Department Observation Unit: Can It Be Funded Through Reduced Inpatient Admission?

STUDY OBJECTIVE We sought to test the assumption that an emergency department observation unit can be funded through the reallocation of resources made available through the units impact in reducing inpatient admissions and facilitating bed closures. METHODS We conducted our study in a tertiary care center ED with 46,000 visits annually. For a 3-month period, all patients admitted to the hospital through the ED were screened by an emergency physician for suitability for admission to an observation unit. Any patient in the hospital for 3 days or less who did not undergo surgery or other inpatient procedure, and who was admitted through the ED, was considered a candidate for the observation unit. RESULTS Of 1,840 admissions, 147 patients met the admission criteria. Only 48 (32.2%) could have been treated in an observation unit, and these patients were not admitted to any single unit in high frequency. The potential savings from inpatient bed closures would only have amounted to 1.68 full-time equivalents-not enough to staff a 4-bed observation unit, which would require 5 full-time equivalents. CONCLUSION Because of the diffuse and inconsistent effect such a unit had on inpatient bed use, funding for an ED observation unit at our institution could not be justified on the basis of the closure of inpatient beds and transfer of resources.

Data collection on patients in emergency departments in Canada.

OBJECTIVE Relatively little is known about the ability of Canadian emergency departments (EDs) and the federal, provincial and territorial governments to quantify ED activity. The objectives of this study were to determine the use of electronic patient data in Canadian EDs, the accessibility of provincial data on ED visits, and to identify the data elements and current methods of ED information system (EDIS) data collection nationally. METHODS Surveys were conducted of the following 3 groups: 1) all ED directors of Canadian hospitals located in communities of >10,000 people, 2) all electronic EDIS vendors, and 3) representatives from the ministries of health from 13 provincial and territorial jurisdictions who had knowledge of ED data collection. RESULTS Of the 243 ED directors contacted, 158 completed the survey (65% response rate) and 39% of those reported using an electronic EDIS. All 11 EDIS vendor representatives responded. Most of the vendors provide a similar package of basic EDIS options, with add-on features. All 13 provincial or territorial government representatives completed the survey. Nine (69%) provinces and territories collect ED data, however the source of this information varies. Five provinces and territories collect triage data, and 3 have a comprehensive, jurisdiction-wide, population-based ED database. Thirty-nine percent of EDs in larger Canadian communities track patients using electronic methods. A variety of EDIS vendor options are available and used in Canada. CONCLUSION The wide variation in methods and in data collected presents serious barriers to meaningful comparison of ED services across the country. It is little wonder that the majority of information regarding ED overcrowding in Canada is anecdotal, when the collection of this critical health information is so variable. There is an urgent need to place the collection of ED information on the provincial and national agenda and to ensure that the collection of this information consistent, comprehensive and mandatory.

Comparison of Canadian versus United States emergency department visits for chronic obstructive pulmonary disease exacerbation

INTRODUCTION Despite the frequency of emergency department (ED) visits for chronic obstructive pulmonary disease (COPD) exacerbation, little is known about practice variation in EDs. OBJECTIVES To examine the differences between Canadian and United States (US) COPD patients, and the ED management they receive. METHODS A prospective multicentre cohort study was conducted involving 29 EDs in the US and Canada. Using a standard protocol, consecutive ED patients with COPD exacerbations were interviewed, their charts reviewed and a two-week telephone follow-up completed. Comparisons between Canadian and US patients, as well as their treatment and outcomes, were made. Predictors of antibiotic use were determined by multivariate logistic regression. RESULTS Of 584 patients who had physician-diagnosed COPD, 397 (68%) were enrolled. Of these, 63 patients (16%) were from Canada. Canadians were older (73 years versus 69 years; P=0.002), more often white (97% versus 65%; P<0.001), less educated (P=0.003) and more commonly insured (P<0.001) than the US patients. US patients more commonly used the ED for their usual COPD medications (17% versus 3%; P=0.005). Although Canadian patients had fewer pack-years of smoking (45 pack-years versus 53 pack-years; P=0.001), current COPD medications and comorbidities were similar. At ED presentation, Canadian patients were more often hypoxic and symptomatic. ED treatment with inhaled beta-agonists (approximately 90%) and systemic corticosteroids (approximately 65%) were similar; Canadians received more antibiotics (46% versus 25%; P<0.001) and other treatments (29% versus 11%; P=0.002). Admission rates were similar in both countries (approximately 65%), although Canadian patients remained in the ED longer than the US patients (10 h versus 5 h, respectively; P<0.001). CONCLUSIONS Overall, patients with acute COPD in Canada and the US appear to have similar history, ED treatment and outcomes; however, Canadian patients are older and receive more aggressive treatment in the ED. In both countries, the prolonged length of stay and high admission rate contribute to the ED overcrowding crisis facing EDs.

How Do Physicians Assess Their Family Physician Colleagues' Performance? Creating a Rubric to Inform Assessment and Feedback.

Introduction: The Colleges of Physicians and Surgeons of Alberta and Nova Scotia (CPSNS) use a standardized multisource feedback program, the Physician Achievement Review (PAR/NSPAR), to provide physicians with performance assessment data via questionnaires from medical colleagues, coworkers, and patients on 5 practice domains: consultation communication, patient interaction, professional self‐management, clinical competence, and psychosocial management of patients. Physicians receive a confidential report; the intent is practice improvement. However, research indicates that feedback from medical colleagues appears to be less understood than that from coworkers or patients, due to a lack of specificity and concerns regarding feedback credibility. The purpose of this study was to determine how physicians make decisions about performance ratings for family physician (FP) colleagues in the 5 practice domains. Methods: This was an exploratory qualitative study using focus groups—one with 11 family physicians and one with 12 specialists—who had served as NSPAR “medical colleague” reviewers. We analyzed focus group transcripts using content analysis. Results: Family and specialist physicians provided examples of behaviors indicative of both high‐ and low‐scoring performance for items within the 5 practice domains. From these, an assessment rubric was created to inform both external reviewers and the physicians being reviewed of performance expectations. Reviewers reported using varied sources of information to make assessments, including shared patients, medical records, referral letters, feedback from others, and self‐reference. Discussion: The CPSNS has used the assessment rubric to create an online resource to inform medical colleague assessment and enhance the usefulness of their NSPAR scores. Further research will be required to determine its impact.

Strategies for managing a busy emergency department.

In a time of increased patient loads and emergency department (ED) exit block, the need for strategies to manage patient flow in the ED has become increasingly important. In March 2002 we contacted all 1282 members of the Canadian Association of Emergency Physicians and asked them to delineate strategies for enhancing ED patient flow and ED productivity without increasing stress levels, reducing care standards or compromising patient safety. Thirty physicians responded. Their suggested flow management strategies, which ranged from clinical decision-making to communication to choreography of time, space and personnel, are summarized here.

Presentation of Evidence in Continuing Medical Education Programs: A Mixed Methods Study

Introduction Clinical trial data can be presented in ways that exaggerate treatment effectiveness. Physicians consider therapy more effective, and may be more likely to make inappropriate practice changes, when data are presented in relative terms such as relative risk reduction rather than in absolute terms such as absolute risk reduction and number needed to treat. Our purpose was to determine (1) how frequently continuing medical education (CME) speakers present research data in relative terms compared to absolute terms; (2) how knowledgeable CME speakers and learners are about these terms; and (3) how CME learners want these terms presented. Methods Analysis of videotapes and PowerPoint slides of 26 CME presentations, questionnaire survey of CME speakers and learners, and focus groups with learners. Results Speakers presented data more frequently in relative than absolute terms, but most frequently in general terms such as frequencies, percentages, graphs, and P‐values with no data. Of 1367 PowerPoint slides, 269 presented research data, and of these, 225 (84%) presented data in general terms, 50 (19%) in relative terms and 19 (7%) in absolute terms. CME speakers understood relative and absolute terms better than learners. Approximately 25–35% of speakers and 45–65% of learners could not correctly calculate relative risk reduction, absolute risk reduction, and number needed to treat. Learners wished to have these terms presented in CME programs in a consistent and easily understood format and requested a brief review of them at the beginning of CME programs. Discussion Presentation of research data in most CME programs is inadequate to allow learners to make fully informed therapeutic decisions. Speakers and learners need professional development to improve their presentation and understanding of research data.