Eric K. H. Chan

Validity and validation in social, behavioral, and health sciences

Section I: Opening Section.- 1 Setting the Stage for and Purposes of Validity and Validation in Social, Behavioral, and Health Sciences - Bruno D. Zumbo & Eric K. H. Chan.- 2 Review of Testing and Test Review Standards and Guidelines In Several International Jurisdictions - Eric K. H. Chan.- SECTION II: Quality of Life, Wellbeing, and Life Satisfaction.- 3 Reporting of Measurement Validity in Articles Published in Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement - Bruno D. Zumbo, Eric K. H. Chan, Michelle Y. Chen, Wen Zhang, Ira Darmawanti, & Olievia P. Mulyana.- 4 A Research Synthesis of Validation Practices Used to Evaluate the Satisfaction with Life Scale (SWLS) - Mary L. Chinni & Anita M. Hubley.- 5 Validation Practices in Counseling: Major Journals, Mattering Instruments, and the Kuder Occupational Interest Survey (KOIS) - Eric K. H. Chan, David W. Munro, Alexander H. S. Huang, Bruno D. Zumbo, Roya Vojdanijahromi, & Neelam Ark.- SECTION III: Psychology and Education.- 6 What Counts as Evidence? An Empirical Review of Validity Studies in Educational and Psychological Measurement - Benjamin R. Shear & Bruno D. Zumbo.- 7 Validity evidence presented in the Journal of Educational Psychology across two temporal periods - Rebecca J. Collie & Bruno D. Zumbo.- 8 A review of validity evidence presented in the Journal of Sport and Exercise Psychology (2002-2012): Misconceptions and recommendations for validation research - Katie E. Gunnell, Benjamin J. I. Schellenberg, Philip M. Wilson, Peter R. E. Crocker, Diane E. Mack, & Bruno D. Zumbo.- 9 The Edinburgh Postnatal Depression Scale (EPDS): A Review of the Reported Validity Evidence - Hillary L. McBride, Rachel M. Wiens, Marvin J. McDonald, Daniel W. Cox, & Eric K. H. Chan.- 10 Validity theory and validity evidence for scores derived from the Behavioural Regulation in Exercise Questionnaire - Katie E. Gunnell, Philip M. Wilson, Bruno D. Zumbo, Peter R. E. Crocker, Diane E. Mack, & Benjamin J. I. Schellenberg.-11 Synthesis of Validation Practices in Two Assessment Journals: Psychological Assessment and the European Journal of Psychological Assessment Anita M. Hubley, Sophie Ma Zhu, Ayumi Sasaki, & Anne M. Gadermann.- SECTION IV: Health and Medicine.- 12 Reporting of Measurement Validity in Articles Published in Quality of Life Research - Eric K. H. Chan, Bruno D. Zumbo, Michelle Y. Chen, and Wen Zhang, Ira Darmawanti & Olievia, P. Mulyana.- 13 Validity Evidence for a Perceived Social Support Measure in a Population Health Context - Daniel W. Cox & Jess J. Owen.- 14 Medical Outcomes Study Short Form-36 (SF-36) and the World Health Organization Quality of Life (WHOQoL) Assessment: Reporting of Psychometric Validity Evidence - Eric K. H. Chan, Bruno D. Zumbo, Wen Zhang, and Michelle Y. Chen, Ira Darmawanti & Olievia, P. Mulyana.- 15 Reporting of Validity Evidence in the Field of Health Care: A Focus on Papers Published in Value in Health - Eric K. H. Chan, and Bruno D. Zumbo, Ira Darmawanti, & Olievia, P. Mulyana.- 16 Validation Practices of the Objective Structured Clinical Examination (OSCE) - Tavinder K. Ark, Neelam Ark, & Bruno D. Zumbo.- 17 (Mis)Alignment of Medical Education Validation Research with Contemporary Validity Theory: The Mini-CEX as an Example - Debra Sandilands & Bruno D. Zumbo.- SECTION V: Conclusions.- 18 Validation Practices in the Social, Behavioral, and Health Sciences: A Synthesis of Syntheses - Juliette Lyons-Thomas, Yan Liu, & Bruno D. Zumbo.- 19 Reflections on Validation Practices in the Social, Behavioral, and Health Sciences - Bruno D. Zumbo & Eric K.H. Chan.

Standards and Guidelines for Validation Practices: Development and Evaluation of Measurement Instruments

The objectives of this chapter are to provide an overview of standards and guidelines for validation practices in developing and evaluating measurement instruments, as well as to examine the extent to which these standards and guidelines are in line with the contemporary theories of validity. Standards and guidelines such as the AERA, APA, and NCME’s Standards for Educational and Psychological Testing, Food and Drug Administration (FDA) guidance for industry (Patient-Reported Outcomes Measures: Use in Medical Product Development to Support Labeling Claims), Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) checklist, Society for Industrial and Organizational Psychology’s (SIOP) Principles for the Validation and Use of Personnel Selection Procedures, and European Federation of Psychologists’ Association’s (EFPA) test evaluation model are reviewed. These standards and guidelines cover different sources of validity and they do not appear to reflect the issues, foci, and theoretical orientations seen in contemporary views of validity (e.g., Kane, Messick, Zumbo).

Modern perspectives of measurement validation emphasize justification of inferences based on patient-reported outcome scores: Seventh paper in a series on patient reported outcomes

BACKGROUND Obtaining the patients view about the outcome of care is an essential component of patient-centered care. Many patient-reported outcome (PRO) instruments for different purposes have been developed since the 1960s. Measurement validation is fundamental in the development, evaluation, and use of PRO instruments. OBJECTIVES This paper provides a review of modern perspectives of measurement validation in relation to the followings three questions as applied to PROs: (1) What evidence is needed to warrant comparisons between groups and individuals? (2) What evidence is needed to warrant comparisons over time? and (3) What are the value implications, including personal and societal consequences, of using PRO scores? DISCUSSION Measurement validation is an ongoing process that involves the accumulation of evidence regarding the justification of inferences, actions, and decisions based on measurement scores. These include inferences pertaining to comparisons between groups and comparisons over time as well as consideration of value implications of using PRO scores. Personal and societal consequences must be examined as part of a comprehensive approach to measurement validation. The answers to these three questions are fundamental to the the validity of different types of inferences, actions, and decisions made on PRO scores in health research, health care administration, and clinical practice.

Reflections on Validation Practices in the Social, Behavioral, and Health Sciences

This chapter presents our reflections on the validation practices in the social, behavioral and health sciences. Based on the 15 syntheses included in this volume we make both narrow and broad recommendations. First, we discuss our observations about construct validation and relations with other variables as validity evidence. Second, we observed that validation studies are not guided by any theoretical orientation or validity perspectives and that validation practices have a feel of being opportunistic or somewhat haphazard. The Standards and other descriptions of current validity theories appear to lack practical guidance. Our recommendation is that validation studies need to have an explicit “validation plan” and the plan needs to be guided by some conceptual or theoretical orientation. We close this chapter with some reflections on what are consistently under-represented in validation practices: a concern for response processes and consequences. A framework that guides validation practice highlighting consequences is described.

Evaluation of a Group Intervention for Veterans Who Experienced Military- Related Trauma

Abstract Military-related trauma and veteran status have been linked with posttraumatic stress symptoms, depressive symptoms, and other personal and interpersonal difficulties. While many treatment evaluations for people with posttraumatic stress exist, few veteran populations or group formats have been evaluated. This report presents an evaluation of the Veterans Transition Program (VTP)–a group-based treatment for veterans who experienced a military-related trauma that is negatively impacting their lives. Fifty-six veterans attended the VTP; all attended every session and completed pre- and post-tests assessing posttraumatic stress and depressive symptoms. Significant pre- to post-test improvement was found on all scales. These findings demonstrate the potential benefit of the VTP and encourage further research.

Longitudinal Change in Response Processes: A Response Shift Perspective

This chapter focuses on the examination of response shift in patient-reported outcomes (PROs) research, with particular attention to measurement validity and response processes. Response shift occurs when changes in PROs over time are the result of changes in how people interpret and respond to PRO measurement items at different points in time. Conceptual foundations of response shift are discussed, followed by a review of statistical methods for examining response shift. The chapter concludes with a discussion of opportunities and challenges for response shift research. We specifically consider that, although response shift is inherently about response processes and measurement validation, most of the methods and applications of response shift are descriptive in nature with an eye towards detecting and controlling for response shift. There is significant opportunity for theoretical and methodological development in focusing on understanding the mechanisms (mediators, moderators and other causes) by which response shift occurs when measuring PROs.

Validation Practices in Counseling: Major Journals, Mattering Instruments, and the Kuder Occupational Interest Survey (KOIS)

Validity of measurement is an important concept in counseling. We present the results of three studies synthesizing the practices of validation in counseling, with an eye towards comparing the validation practices against the Standards for Educational and Psychological Testing (AERA, APA, NCME, Standards for educational and psychological testing. American Educational Research Association, Washington, DC, 1999). In Study 1, validation papers published in four major counseling journals in North America in 2009–2011 were synthesized. In Study 2, papers addressing the validation practices on the construct of mattering were summarized. In Study 3, the validity evidence of the Kuder Occupational Interest Survey (KOIS) was reviewed. Together, the three studies provide a broad lens into the validation practices in counseling. The results showed that validation practices were not in line with the current, modern view of validity. Implications for validation research, practice, and pedagogy/curriculum development in the counseling profession are discussed.

Reporting of Validity Evidence in the Field of Health Care: A Focus on Papers Published in Value in Health

Validity is a fundamental issue in the development and evaluation of psychometric instruments in health care. With an aim towards informing validation practice, we investigated the reporting of validity evidence in the field of health care, focusing on a journal that, by design and scope, is meant to influence health outcomes researchers, technology developers, regulators, policy makers, and health economists: Value in Health, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) official journal. A systematic search of papers since the journal’s inception (January 1998) to December 2010 was conducted using the official journal website. Only empirical psychometric validation papers were included in the analysis. The coding was conducted by two of the authors independently. A total of 68 articles met our inclusion criteria. The number and percentage of reports of the broad categories of the sources of validity evidence included internal consistency reliability (69.1 %), construct (50.0 %), discriminant (33.8 %), convergent (33.8 %), content (25.0 %), criterion (20.6 %), response processes (4.4 %), and consequences (2.9 %). Researchers conducting validation studies are not relying on only one source of validity evidence at the exclusion of all others and hence representing a broad perspective on psychometric validity. However, some sources of validity evidence such as response processes and consequences are rarely reported. Readers and practitioners should not uncritically accept the often-stated conclusion that “the instrument is valid”.

Setting the Stage for Validity and Validation in Social, Behavioral, and Health Sciences: Trends in Validation Practices

This chapter sets the stage for the book Validity and Validation in Social, Behavioral, and Health Sciences by examining trends in reporting practices. The book is a collection of inter-related chapters synthesizing the validation practices in the broad areas of social, behavioral, and health sciences with an eye towards improving the practice of measurement validation. The chapters also addressed whether recent work in validity theories (e.g. Kane MT, Validation. In: Brennan RL (ed) Educational measurement, 4th edn. American Council on Education/Praeger, Westport, pp 17–64, 2006; Messick S, Validity. In: Linn RL (ed) Educational measurement, 3rd edn. American Council on Education and Macmillan, New York, pp 13–103, 1989) or the Standards for Educational and Psychological Testing (AERA, APA, NCME, Standards for educational and psychological testing. American Educational Research Association, Washington, DC, 1999) were cited as informing the validation practice. In this opening chapter, Zumbo and Chan provide a brief sketch of the evolving concepts of validity theories and practices of validation as well as a description of an empirical database study of trends in validation practices since the 1960s.

The Edinburgh Postnatal Depression Scale (EPDS): A Review of the Reported Validity Evidence

The Edinburgh Postnatal Depression Scale (EPDS; Cox et al. Br J Psychiatr 150:782–786, 1987) is used internationally for the detection of postpartum depression in women. We reviewed the validity evidence of the EPDS and examined the extent to which the reported evidence is in line with the modern view of validity. Of the 57 articles included, 30 (52.6 %) included internal consistency, 29 (51 %) discussed predictive validity, 27 (47.4 %) reported on concurrent validity, 16 (28.1 %) studies addressed the internal structure of the EPDS, and 15 (26.3 %) included convergent validity. Fewer articles reviewed response processes (1.8 %), six (10.5 %) discussed content validity of the scale, five (8.8 %) addressed face validity of the EPDS, three studies (5.3 %) discussed discriminant validity, and two studies (3.5 %) included consequences. Certain sources of validity evidence such as response processes and consequences need to be accumulated to strengthen the score inferences of the EPDS.