Isabel Benítez

Validity evidence based on response processes

BACKGROUND Validity evidence based on response processes was first introduced explicitly as a source of validity evidence in the latest edition of Standards for Educational and Psychological Testing. In this paper, we present the theory, the relationship with other sources of validity evidence, and the methods available for validation studies aimed at obtaining validity evidence about response processes. METHOD A comprehensive review of the literature along with theoretical and practical proposals. RESULTS The articles provides arguments for determining when validity evidence based on response processes is critical for supporting the use of the test for a particular purpose, and examples of how to perform a validation study to obtain such validity evidence. CONCLUSIONS There are methods for obtaining validity evidence based on response processes. Special attention should be paid to validation studies using the cognitive interview method given its features and possibilities. Future research problems pose how to combine data from different methods -qualitative and quantitative-, to develop complete validity arguments that support the use of the test for a particular purpose.

Analysis of Nonequivalent Assessments across Different Linguistic Groups Using a Mixed Methods Approach: Understanding the Causes of Differential Item Functioning by Cognitive Interviewing.

Differential item functioning (DIF) can undermine the validity of cross-lingual comparisons. While a lot of efficient statistics for detecting DIF are available, few general findings have been found to explain DIF results. The objective of the article was to study DIF sources by using a mixed method design. The design involves a quantitative phase in which DIF was analyzed followed by a qualitative phase conducting cognitive interviews. To illustrate the proposal, polytomous DIF was analyzed in the scales from the PISA (Programme for International Student Assessment) Student Questionnaire (Organisation for Economic Co-operation and Development). Evidence obtained allowed DIF to be connected with differences in the interpretation patterns of participants from the different linguistic groups. Finally, benefits of mixed methods design for analyzing equivalence in cross-lingual assessments are discussed.

Development and Validation of the Spanish Hazard Perception Test

Objective: The aim of the current study is to develop and obtain valid evidence for a hazard perception test suitable for the Spanish driving population. To obtain valid evidence to support the use of the test, the effect of hazardous and quasi-hazardous situations on the participants’ hazard prediction is analyzed and the pattern of results for drivers with different driving experience—that is, learner, novice, and expert drivers and reoffender vs. nonoffender drivers—is compared. Potentially hazardous situations are those that develop without involving any real hazard (i.e., the driver did not actually have to decelerate or make any evasive maneuver to avoid a potential collision). The current study analyzed repeat offender drivers attending compulsory reeducation programs as a result of reaching the maximum number of penalty points on their driving license due to repeated violations of traffic laws. Method: A new video-based hazard perception test was developed, using a total of 20 hazardous situation videos plus 8 quasi-hazardous situation videos. They were selected from 167 recordings of natural hazards in real Spanish driving settings. Results: The test showed adequate psychometric properties and evidence of validity, distinguishing between different types of drivers. Psychometric results confirm a final version of the hazard perception test composed of 11 video clips of hazards and 6 video clips of quasi-hazards, for which an overall Cronbachs alpha coefficient of.77 was obtained. A lack of ability to detect quasi-hazards and distinguish them from hazardous situations was also found for learner, novice, and reoffender drivers. Learner drivers obtained lower average scores than novice and experienced drivers with the hazardous situation videos, and learner drivers obtained lower average scores than experienced drivers with the quasi-hazardous situation videos, suggesting that the ability to correctly identify hazardous traffic situations may develop early by accumulating initial driving experience. However, the ability to correctly identify quasi-hazardous situations may develop later with the accumulation of further driving experience. Developing this ability is also difficult for reoffender drivers. Conclusion: The test has adequate psychometric properties and is useful in distinguishing between learner, novice, and expert drivers. In addition, it is useful in that it analyzes the performance of both safe and unsafe drivers (reoffenders who have already lost their driving license).

EASY-DIF: Software for Analyzing Differential Item Functioning Using the Mantel- Haenszel and Standardization Procedures

Practitioners are increasingly becoming interested in Differential Item Functioning (DIF) forimproving the validity of test and scale interpretations. Among the statistical procedures availableto assess DIF in dichotomous and polytomous items, the Mantel-Haenszel chi-square (1959) andother standardization proceduresmay beparticularly attractive to practitioners.Some of the statis-tics associated with Mantel-Haenszel procedure can be performed using specific software such asEZ-DIF (Waller, 1998), DIFAS (Penfield, 2005), and MH-DIF (Fidalgo, 1994). However, usingthese programs requires being familiar with the statistics of DIF procedures in order to understandoutput. In addition, key characteristics of the Mantel-Haenszel procedures, such as the matchingstrategy (thick or thin), purification of the matching criteria, and so on, are not currently availablein all statistical software packages. As a result, EASY-DIF was developed to provide easy-to-usesoftware for performing the most common and useful MH and standardization procedures, witha view to guiding practitioners through the analyses and helping them interpret the output.EASY-DIFanalyzesuniformandnonuniformDIFforthetotalsampleandseparatelyforlow-performing and high-performing groups (Clauser, Mazor, & Hambleton, 1994). Users canexplore possible cancellation and amplification DIF effects by establishing different cut scoresfor each group. Up to six different matching strategies based on total score distributions can beimplemented: thinmatching,equal interval, percentage of totalsample, percentage of focalsam-ple, censored matching, and minimum cell frequency (Donoghue & Allen, 1993). In addition,users can use a procedure purification of matching criteria through the selection of a validsubtest.For dichotomous items, EASY-DIF computes the Mantel-Haenszel chi-square (Holland T Mantel & Haenszel, 1959), the Mantel-Haenszel common odds ratio (CamilliSMantel&Haenszel,1959),theMH-DeltaDIF,thestandarderrorofMH-DeltaComputer Program Exchange

Using Mixed Methods to Interpret Differential Item Functioning

ABSTRACT Analysis of differential item functioning (DIF) is often used to determine if cross-lingual assessments are equivalent across languages. However, evidence on the causes of cross-lingual DIF is still evasive. Expert appraisal is a qualitative method useful for obtaining detailed information about problematic elements in the different linguistic versions of items. In this article we propose and explore a mixed methods approach that integrates quantitative results from DIF analysis and qualitative findings from expert appraisal to discover reasons why items exhibit DIF across languages. First, polytomous DIF was analyzed in responses to the U.S. and Spanish version of scales from the PISA Student Questionnaire by Differential Step Functioning and Ordinal Logistic Regression. Items flagged by both methods were selected to be studied to interpret DIF causes. Secondly, experts were asked about non-comparable elements in items. Experts provided qualitative evidence on problematic issues (different interpretation patterns or response processes), that may have been the cause of the DIF. The integration of results from both methods was aimed at relating type of DIF to expert appraisal findings.

Linking extreme response style to response processes: a cross-cultural mixed methods approach

The aim of this study is to contribute to a better understanding of extreme response style in cross-cultural research by integrating quantitative and qualitative evidence in a mixed methods design. In the quantitative phase, indexes of extreme response style, derived from quality of life measures from different international studies, were compared between Spain and the Netherlands. Results indicated that extreme responding was more common among Spanish than among Dutch in endorsement of items, but that the opposite was found for frequency scales including never as a response anchor. In the qualitative phase, cognitive interviews were conducted with 25 participants in each country. The integration of quantitative results and qualitative findings suggests that country differences in extreme response style may stem from various sources, including the more independent evaluation of each item by Dutch, the stronger connotations of never for Spanish and stronger emotions triggered by specific topics such as work satisfaction that was more strongly associated with insecurity for Spanish. It is concluded that the integration of quantitative and qualitative evidence can help to understand cross-cultural similarities and differences in extreme response style.

Beyond diagnosis: the Core Sets for persons with schizophrenia based on the World Health Organization’s International Classification of Functioning, Disability, and Health

Abstract Purpose: Based on the International Classification of Functioning, Disability and Health (ICF), this paper presents the results of the process to develop the Comprehensive and Brief Core Sets for schizophrenia that allow to comprehensively describe functioning in persons with schizophrenia. Methods: Twenty health professionals from diverse backgrounds participated in a formal and iterative decision-making process during an international consensus conference to develop these Core Sets. The conference was carried out based on evidence gathered from four preparatory studies (systematic literature review, qualitative study, expert survey, and empirical study). The first step of this decision-making and consensus process comprised of discussions and voting in working groups and plenary sessions to develop the comprehensive version. The categories of the Comprehensive ICF Core Set for schizophrenia served as the basis for the second step –a ranking and cutoff procedure to decide on the brief version. Results: Of the 184 candidate categories identified in the preparatory studies, 97 categories were included in the Comprehensive Core Set for schizophrenia. A total of 25 categories were selected to constitute the Brief Core Set. Conclusions: The formal decision-making and consensus process integrating evidence from four preparatory studies and expert opinion led to the first version of the Core Sets for schizophrenia. Comprehensive and Brief Core Sets for schizophrenia may provide a common language among different health professionals and researchers, and a basic international standard of what to measure, report, and assess the functioning of persons with schizophrenia. Implications for rehabilitation Schizophrenia is a chronic mental disorder that has a tremendous impact on functioning and daily life of persons living with the disorder. The International Classification of Functioning, Disability and Health (ICF) offers an internationally recognized standard for describing the functioning status of these individuals. The Core Sets for schizophrenia have potential use in supporting rehabilitation practice such as for planning mental health services and other interventions or defining rehabilitation goals, and documenting patient care. The Core Sets for schizophrenia may also be used to promote interdisciplinary coordination and facilitate communication between members of a multidisciplinary rehabilitation team. Rehabilitation research is another potential area of application of the Core Sets for schizophrenia. This is valuable, since rehabilitation research provides crucial evidence for optimizing rehabilitation practice.

Analysis of Quality of Proxy Questions in Health Surveys by Behavior Coding

The aim of this study is to show how to analyze the quality of questions for proxy informants by means of behavior coding. Proxy questions can undermine survey data quality because of the fact that proxies respond to questions on behalf of other people. Behavior coding can improve questions by analyzing interviewer–respondent interactions. Twenty-nine proxies participated in the pretesting of a disability questionnaire. The questionnaire includes 11 questions related to daily-life limitations as a result of health problems. Interviewer–proxy interactions were coded and analyzed by means of Sequence Viewer program. The percentages, from a methodological perspective, of ideal “question-and-answer” sequences varied from 28% to 76% throughout the 11 questions analyzed. The results obtained pointed out the necessity of reviewing some of the proxy questions analyzed. Behavior coding can improve the quality of proxy questions in health surveys when proxy informants are surveyed.

Evaluating Structural Equivalence in Psychological Questionnaires Using Weighted Multidimensional Scaling

Cross-cultural scientists evaluate constructs across different cultural and linguistic groups. However, to make valid comparisons it is necessary to assure the equivalence of measurement instruments across populations. In this paper we use multidimensional scaling (MDS) to evaluate the structural equivalence of a measure of assertiveness developed originally for Mexican culture across Mexican and Spanish samples. 316 students from the Autonomous University of Yucatan (Mexico) and 316 students from the University of Granada (Spain) responded to the questionnaire. Separate dimensions were needed to account for the structure of the assertiveness questionnaire across the Mexican and Spanish samples. However, the distinction between dimensions was different in both samples. The results suggest that culture determines which assertive behaviors are considered appropriate. The usefulness of MDS for evaluating the equivalence of psychological questionnaires is discussed.

Advances in environmental psychology regarding the promotion of wellbeing and quality of life / Avances de la psicología ambiental ante la promoción del bienestar y la calidad de vida

Abstract In recent years, environmental psychology has evolved rapidly, integrating concepts and approaches from a range of different disciplines. Person-environment interaction involves so many factors that it requires solid theoretical foundations to understand the variables associated with this interaction. The present special issue offers a reflection of this reality, since it compiles research that aims to innovate from a theoretical point of view but which also introduces new methodological concepts and approaches. The influence of different psychosocial variables on behaviours and environmental commitment is evaluated at a substantive level. Traditionally tackled demographic variables are studied, but a more integral approach is also proposed, which incorporates complex concepts that influence the lives of people, such as culture, quality of life, satisfaction, life expectations, and the emotions present when making decisions about how to behave pro-environmentally.