Deborah A. Miller

A Value-Added Methodology for Assessing the Effectiveness of Psychiatric Residency Training

Educators often find it difficult to identify variables that accurately assess the knowledge that residents have gained due to their training. Recently in professional education circles, attention has shifted from process to outcome variables, with more emphasis being placed on student gains rather than on program resources. A methodology used extensively to assess undergraduate education is the value-added approach. This methodology is often demonstrated by a change in scores from two administrations of a test instrument that measures students’ specific knowledge or aptitudes. Residency training programs that offer standardized examinations to their residents are in a good position to utilize this evaluation technique. This article will demonstrate how the value-added technique can be used to demonstrate cognitive growth in psychiatric residents by using the Psychiatry Resident In-Training Examination.

Associate Residency Training Directors in Psychiatry: Demographics, Professional Activities, and Job Satisfaction

ObjectiveThe purpose of this study was to characterize associate training director (ATD) positions in psychiatry.MethodAn on-line survey was e-mailed in 2009 to all ATDs identified through the American Association of Directors of Psychiatric Residency Training (AADPRT). Survey questions elicited information regarding demographics, professional activities, job satisfaction, and goals.ResultsOf 170 ATDs surveyed,73 (42.9%) completed the survey. Most respondents (71.3%) had been in their positions for 3 years or less. Many ATDs indicated that they were involved in virtually all aspects of residency training; 75% of respondents agreed that they were happy with their experience. However, specific concerns included inadequate time and compensation for the ATD role in addition to a lack of mentorship and unclear job expectations.ConclusionsThoughtful attention to the construction of the ATD role may improve job satisfaction.

What Components Should Be Evaluated in a Psychiatry Residency

In this pilot study, the authors examined faculty responses to a proposed new evaluation form for residents. Because past research on priorities for psychiatric residency curricula have depended on direct surveys, previous findings may represent auricular ideals instead of what faculty actually evaluate. The goal of the study was to draw inferences about priorities in psychiatric education by studying what components (knowledge, skills, and attitudes) of a psychiatry residency that faculty are willing to spend their time evaluating. The responses were grouped by agreement about whether the item 1) could be evaluated on their site, 2) could be evaluated during psychotherapy supervision, 3) should be combined with other items, and 4) should be eliminated. The results showed much agreement about which items to include and little agreement about which ones to exclude. Fundamental psychiatric skills and attitudes were rated as most important, and there was widespread interest in a diversity of knowledge, skills, and attitudes that psychiatric residents should possess. The authors found this method, which could be easily applied to other programs, useful in setting auricular priorities.

Psychiatry clerkship sites : factors that make a difference in student performance.

Using correlation ratios and analysis of variance, several potential variability factors in student clerkship performance were examined. Comparisons are made between clerkship clinical sites for scores on written work, clinical work, and examination scores, and for factors such as the amount of patient and faculty contact. A small amount of grade variance is due to clerkship site. Correlations between scores on clinical skills and examination scores were low, consistent with the idea that grades of clinical performance and performance on examinations measure different dimensions of student ability. The relevance for clerkship planning and grading equity is discussed.

Psychiatry residency accreditation and measuring educational outcomes.

During the last decade there has been a shift within accreditation circles toward evaluating educational outcomes as measures 01 institutional effectiveness in addition to educational inputs and process, which historically have been emphasized. The U.S. Department 01 Education has incorporated outcome assessment into the regulations required 01 any accrediting body it funds or recognizes. Although these regulations do not directly apply to accreditation 01 psychiatry residencies, their impact may be feit indirectly. The authors review some 01 the educationalliterature on outcome evaluation and suggest ways psychiatry residencies might be prepared to measure outcomes within their current procedures should the Residency Review Committee move in this direction.

Testing Psychiatric Diagnostic Recognition Skills with Video Vignettes

While the use of video-tape in evaluating student performance in the history and physical examination process has proven to be an extremely effective tool, there has been little use of video-tape to test student diagnostic recognition skills. We developed an assessment approach to measure second year medical students’ abilities to recognize common psychiatric diagnoses. We selected 12 cases from preexisting video vignettes that had been developed by the American Psychiatric Association to specifically illustrate common psychiatric problems or diagnoses. The 12 vignettes ranged from 20–30 seconds in length. Students were asked three questions for each vignette for a total of 36 items using the large menu or R type format. The test was administered in a large auditorium with 202 students. Total testing time was 30 minutes. The overall test was reliable with case variability, not unanticipated, being the greatest source of error. Although this was the first time that this form of testing was used, 75.1% of the students agreed that they would like to see more tests given in this format. Testing over 200 students’ diagnostic recognition skills simultaneously with 12 short video vignettes proved to be an efficient and reliable testing approach and is recommended for expanded application.

Self-directed learning in a psychopathology course.

Self-directed learning has been identified as an essential element of professional development. In this approach, students receive in advance a complete set of objectives and didactic knowledge (cognitive) learning materials, are provided opportunities to develop skills with actual or simulated clinical experiences, and adapt a broad and flexible array of educational media to their individual learning styles. In the development of a self-directed psychopa-thology course for second-year medical students, the authors incorporated four modifications to the traditional lecture and small group (faculty-directed) approach: 1) independent (self-directed) learning with no prescribed study times or sequence to the material, 2) use of student and facult y-directed interviews of patients, 3) greater freedom for selecting teaching methods in small groups, and 4) elimination of formal lectures. The authors describe the educational results with such an approach.