Mark C. Holter

Fidelity Criteria: Development, Measurement, and Validation

Fidelity may be defined as the extent to which delivery of an intervention adheres to the protocol or program model originally developed. Fidelity measurement has increasing significance for evaluation, treatment effectiveness research, and service administration. Yet few published studies using fidelity criteria provide details on the construction of a valid fidelity index. The purpose of this review article is to outline steps in the development, measurement, and validation of fidelity criteria, providing examples from health and education literatures. We further identify important issues in conducting each step. Finally, we raise questions about the dynamic nature of fidelity criteria, appropriate validation and statistical analysis methods, the inclusion of structure and process criteria in fidelity assessment, and the role of program theory in deciding on the balance between adaptation versus exact replication of model programs. Further attention to the use and refinement of fidelity criteria is important to evaluation practice.

Best practices: a program to support shared decision making in an outpatient psychiatric medication clinic.

This column presents preliminary findings of an intervention to support shared decision making in psychopharmacology consultation. The waiting area in an urban psychiatric medication clinic was transformed into a peer-run Decision Support Center featuring a user-friendly, Internet-based software program with which clients could create a one-page computer-generated report for use in the medication consultation. The Decision Support Center was used 662 times by 189 unique users from a young-adult and general adult case management team from October 2006 to September 2007. All clients had severe mental disorders. Only ten clients refused to use the intervention at some point during the pilot study. Focus groups with medical staff (N=4), clients (N=16), case managers (N=14), and peer-specialist staff (N=3) reported that the intervention helped to create efficiencies in the consultation and empower clients to become more involved in treatment-related decision making. A randomized controlled trial is currently in process.

Barriers to Evidence-Based Practice Implementation: Results of a Qualitative Study

This study reports on a qualitative study of barriers to EBP implementation in one state that sought to implement supported employment and integrated dual diagnosis treatment. The study found that the most significant obstacles emanated from the behavior of supervisors, front-line staff and other professionals in the agency. A lack of synergy profoundly impeded implementation.

Critical Ingredients of Consumer Run Services: Results of a National Survey

Fidelity criteria are increasingly used in program monitoring and evaluation, but are difficult to derive for emerging models (i.e., those not based on theory or a research demonstration project). We describe steps used to develop and operationalize fidelity criteria for consumer-run (CR) mental health services: articulating and operationalizing criteria based on published literature, then revising and validating the criteria through expert judgments using a modified Delphi method. Respondents rated highest those structural and process components emphasizing the value of consumerism: consumer control, consumer choices and opportunities for decision-making, voluntary participation (and the absence of coercion), and respect for members by staff.

Mental Health and Mental Illness: Out of the Closet?

The extensive changes in mental health services over the past 25 years negate the possibility of an overall research summary. Instead, we identify six pivotal topics to explore in depth: (1) the paradigm shift to a biopsychosocial mental illness etiology, (2) the organization and financing of mental health care, (3) community‐based programs for adults with serious mental illness, (4) the role of families and consumers, (5) services for children and adolescents with serious emotional or mental disturbances, and (6) the interface with criminal justice. The article concludes with a discussion of future issues in mental health services research and the role of social work researchers.

Outcomes of an illness self-management group using wellness recovery action planning.

OBJECTIVE The aim of this preliminary study was to examine the impact of participation in an illness self-management recovery program (Wellness Recovery Action Planning-WRAP) on the ability of individuals with severe mental illnesses to achieve key recovery related outcomes. METHODS A total of 30 participants from three mental health centers were followed immediately before and after engaging in a 12-week WRAP program. RESULTS Three paired sample t-tests were conducted to determine the effectiveness of WRAP on hope, recovery orientation, and level of symptoms. A significant positive time effect was found for hope and recovery orientation. Participants showed improvement in symptoms, but the change was slightly below statistical significance. CONCLUSIONS These preliminary results offer promising evidence that the use of WRAP has a positive effect on self-reported hope and recovery-related attitudes, thereby providing an effective complement to current mental health treatment.

Relevance of spirituality for people with mental illness attending consumer-centered services.

Spirituality has been cited in the literature as having a positive effect on mental health outcomes. This paper explores the relationship of spirituality to demographic, psychiatric illness history and psychological constructs for people with mental illness (N=1835) involved in consumer-centered services (CCS-Clubhouses and Consumer run drop-in centers). Descriptive statistics indicate that spirituality is important for at least two thirds of the members in the study. Members primarily indicated participation in public spiritual activities (i.e., church, bible study groups), followed by private activities (prayer, reading the bible, and meditation) (both of which were centered on belief in the transcendent). A logistic regression analysis was done to explore variables related to spirituality (i.e., demographics, psychiatric illness history, and psychological constructs). Results suggest that age, gender, having psychotic symptoms, having depressive symptoms, and having a higher global quality of life, hope and sense of community were all significant correlates of spirituality.

A Coding System to Measure Elements of Shared Decision Making During Psychiatric Visits

OBJECTIVE Shared decision making is widely recognized to facilitate effective health care. The purpose of this study was to assess the applicability and usefulness of a scale to measure the presence and extent of shared decision making in clinical decisions in psychiatric practice. METHODS A coding scheme assessing shared decision making in general medical settings was adapted to mental health settings, and a manual for using the scheme was created. Trained raters used the adapted scale to analyze 170 audio-recordings of medication check-up visits with either psychiatrists or nurse practitioners. The scale assessed the level of shared decision making based on the presence of nine specific elements. Interrater reliability was examined, and the frequency with which elements of shared decision making were observed was documented. The association between visit length and extent of shared decision making was also examined. RESULTS Interrater reliability among three raters on a subset of 20 recordings ranged from 67% to 100% agreement for the presence of each of the nine elements of shared decision making and 100% for the agreement between provider and consumer on decisions made. Of the 170 sessions, 128 (75%) included a clinical decision. Just over half of the decisions (53%) met minimum criteria for shared decision making. Shared decision making was not related to visit length after the analysis controlled for the complexity of the decision. CONCLUSIONS The rating scale appears to reliably assess shared decision making in psychiatric practice and could be helpful for future research, training, and implementation efforts.

TANF Workers’ Responses to Battered Women and the Impact of Brief Worker Training What Survivors Report

Battered women (n = 159) report on their experiences with their Temporary Assistance to Needy Families (TANF) case workers. Workers most often ask about physical harm, feelings of fear, and police involvement. They least often create a safety plan, give information about work exemptions, and ask whether the partner had a gun. Women’s major reasons for not talking about abuse are that the worker did not ask and a fear of negative outcomes. Workerswho attendeda 1-day training are more likely than untrained workers to discuss the women’s fear and physical harm, to help develop a safety plan, and to be viewed as generally helpful.

Characteristics of Users of Consumer-Run Drop-In Centers Versus Clubhouses

Clubhouses and consumer-run drop-in centers (CRDIs) are two of the most widely implemented models of consumer-centered services for persons with serious mental illness. Differences in structure and goals suggest that they may be useful to different types of consumers. Information on what types of consumers use which programs would be useful in service planning. This study analyzes data from the authors’ NIMH-funded research on 31 geographically matched pairs of clubhouses and CRDIs involving more than 1,800 consumers to address the following question: are there significant differences in the characteristics and outcomes of members of clubhouses versus CRDIs? Results from multilevel analyses indicated that clubhouse members were more likely to be female, to receive SSI/SSDI, to report having a diagnosis of schizophrenia, and to live in dependent care; and they reported both a greater number of lifetime hospitalizations and current receipt of higher intensity traditional MH services. Controlling for differences in demographic characteristics, psychiatric history, and mental health service receipt, clubhouse members also reported higher quality of life and were more likely to report being in recovery. CRDI consumers were more likely to have substance abuse histories. Possible reasons for the differences are discussed. The results suggest that CRDIs are a viable alternative to more traditional mental health services for individuals who might not otherwise receive mental health services.