Ryan J. Kettler

Base rates of social skills acquisition/performance deficits, strengths, and problem behaviors: an analysis of the Social Skills Improvement System--Rating Scales.

Base rate information is important in clinical assessment because one cannot know how unusual or typical a phenomenon is without first knowing its base rate in the population. This study empirically determined the base rates of social skills acquisition and performance deficits, social skills strengths, and problem behaviors using a nationally representative sample of children and adolescent ages 3-18 years. Using the national standardization sample of the Social Skills Improvement System--Rating Scales (N = 4,550) across 3 informants (teacher, parent, and student) and across 3 broad age groupings (3-5 years, 5-12 years, and 13-18 years), these base rates were computed. Results showed that the base rates for social skills acquisition deficits and problem behaviors are extremely low in the general population. Base rates for social skills performance deficits and social skills strengths were considerably higher, with students in the 5- to 12-year-old age group reporting fewer performance deficits and more social skills strengths than older children (13-18 years). Teachers and parents reported more performance deficits and fewer social skills strengths across all age groups than students in the 5- to 12-year-old age group. These results are discussed in terms of the utility of base rate information in clinical decision making.

Effects of Using Modified Items to Test Students with Persistent Academic Difficulties

This study investigated the effects of using modified items in achievement tests to enhance accessibility. An experiment determined whether tests composed of modified items would reduce the performance gap between students eligible for an alternate assessment based on modified achievement standards (AA-MAS) and students not eligible, and the impact on student proficiency levels. Three groups of eighth-grade students (N = 755) from four states took original and modified versions of reading and mathematics tests. Findings indicate modified item conditions were significantly easier for all students and modifications would result in more AA-MAS eligible students meeting proficiency status. Study limitations and follow-up research on item modifications and the performance of students with disabilities are discussed.

Modifying Achievement Test Items: A Theory-Guided and Data-Based Approach for Better Measurement of What Students With Disabilities Know

Federal regulations allow up to 2% of the student population of a state to achieve proficiency for adequate yearly progress by taking an alternate assessment based on modified academic achievement standards (AA-MAS). Such tests are likely to be easier, but as long as a test is considered a valid measure of grade level content, it is allowable as an AA-MAS (U.S. Department of Education, 2007b). In this article, we examine procedures for developing, modifying, and evaluating items and tests using an evolving modification paradigm, as well as a classic reliability and validity framework. Theoretical influences, such as principles of universal design, cognitive load theory, and item development research, are discussed. The Test Accessibility and Modification Inventory, a tool that provides systematic and comprehensive guidance to help educators modify grade-level tests, is introduced. Cognitive lab methods and experimental field tests are then described, along with examples and key findings from each, relevant to AA-MASs. The article concludes with a discussion of precautions, lessons learned, and questions generated about the methods used to improve both access and test score validity for the students who are eligible for this new alternate assessment.

Screening Young Children’s Risk for Mental Health Problems: A Review of Four Measures

Accurate identification of young children at risk for mental health problems is a key step in establishing early childhood preventive intervention programs. Without psychometrically valid identification procedures, children in need of early intervention may not be identified and may not receive appropriate care. This article provides a review of universal screening measures to help teachers, school psychologists, and other mental health professionals make informed decisions about selecting tools to assist in identifying preschool-age children at risk for mental health problems. A review of the literature on broadband measures designed to screen the social and emotional functioning of children age 3 to 5 years old yielded three published measures that met predetermined inclusionary criteria. An additional unpublished measure was also included. Selected measures were reviewed according to the Standards for Educational and Psychological Testing. Special attention was given to predictive validity indices, including sensitivity, specificity, positive predictive value, and negative predictive value. The results of the review provided information on the measures that were most psychometrically sound and cost-effective for screening young children for mental health problems. Recommendations are made for routine screening, so young children can be referred for prevention programming before their problems worsen and negatively affect their development.

Modified Multiple-Choice Items for Alternate Assessments: Reliability, Difficulty, and Differential Boost.

Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N = 755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal. Mean item difficulties within the Rasch model were shown to decrease more for students who would be eligible for the AA-MAS than for non-eligible groups, revealing evidence of differential boost. Exploratory analyses indicated that shortening the question stem may be a highly effective modification, and that adding graphics to reading items may be a poor modification.

Testing Accommodations: Theory and Research to Inform Practice.

This article presents a framework for practitioners in education and psychology to select accommodations based on student profiles and testing demands. A brief history of testing accommodations policy in the US and a definition of terms provide context for the discussion. A review of theory and empirical findings related to testing accommodations follow. The key assumption is that testing accommodations are used to obtain test scores from which valid inferences can be drawn. The article introduces a three-step process that involves identifying access skills, selecting accommodations, and considering their impact on target skills or knowledge. The reader is also introduced to tools that are commercially available to help with these decisions.

Assessing Opportunity-to-Learn for Students With Disabilities in General and Special Education Classes

Current legislation encourages schools to educate students with disabilities (SWDs) in general education settings to the greatest extent appropriate. However, it is unclear whether inclusion in general education settings provides SWDs a sufficient opportunity to learn the academic content assessed by accountability measures. This initial study was designed to (a) describe the extent to which general and special educators provide their eighth-grade classes with an opportunity to learn the state-specific standards and to (b) examine the extent to which SWDs experience a differentiated opportunity to learn compared with their overall class. Across three states, we trained 38 general and special educators to use an online teacher log to report on various opportunity-to-learn (OTL) indices for 46 mathematics and reading classes and 89 nested SWDs. Based on an average logging period of 151 days, the results indicated that this sample’s SWDs included in general education classes experienced less time on standards, more non-instructional time, and less content coverage than their overall class. Limitations and implications for policy and future research are discussed.

Effects of Testing Accommodations on Math and Reading Scores: An Experimental Analysis of the Performance of Students with and without Disabilities.

This study examined the effects of testing accommodations on the mathematics and reading test scores of a sample of 119 fourth graders and 78 eighth graders. The sample included 49 fourth-grade students diagnosed with a disability and 39 eighth-grade students diagnosed with a disability. The study used a 2 (Disability Status) x 2 (Testing Condition) ¥ 2 (Grade) ¥ 2 (Test Content Area) ¥ 2 (Order) mixed design. All students were tested under two conditions (i.e., accommodated or nonaccommodated) on equivalent forms of research editions of mathematics and reading tests from an achievement test used in many statewide assessment systems. Testing conditions were randomized to control for potential order effects. Results indicate that fourth-grade students with disabilities (SWD) benefited from testing accommodations more than students without disabilities (SWOD); this differential benefit was greater on the reading tests (effect size for SWD = .42, effect size for SWOD = .13) than on the mathematics tests (effect size for SWD = .46, effect size for SWOD = .27). Furthermore, a higher percentage of SWDs improved at least one proficiency level than SWODs. Both SWDs and SWODs in eighth grade gained slight benefits from the testing accommodations. More eighth-grade SWDs improved at least one proficiency level on the reading tests, but more SWODs improved at least one proficiency level on the math tests. The article discusses implications of these findings, limitations of the study, and directions for future research.

Testing a multi-stage screening system: Predicting performance on Australia's national achievement test using teachers' ratings of academic and social behaviors

This study addresses the predictive validity of results from a screening system of academic enablers, with a sample of Australian elementary school students, when the criterion variable is end-of-year achievement. The investigation included (a) comparing the predictive validity of a brief criterion-referenced nomination system with more comprehensive behavior ratings, and (b) determining the optimal combination of scores to gather from this system for identifying academic difficulties. Students (n = 360) were rated using both the Performance Screening Guides (PSGs; Elliott & Gresham, 2007) and the Rating Scales (Gresham & Elliott, 2008) of the Social Skills Improvement System. The PSGs were highly sensitive (0.95) predictors of below minimum standard performance on the national test. The Rating Scales, including the brief academic competence scale in particular, added predictive value in most domains. The discussion addresses implications of these findings for school psychologists and other educators interested in the role of social behaviors as academic enablers and predictors of academic achievement.

Identification of preschool children at risk for emotional and behavioral disorders: Development and validation of a universal screening system ☆

Three studies were designed to measure various indicators of the reliability and validity of the Preschool Behavior Screening System (PBSS) as a screening instrument for emotional and behavioral disorder risk status of children of preschool age. Study 1 and Study 2 served to establish evidence for content validity. Study 3 investigated one type of reliability evidence (i.e., internal consistency) and four types of validity evidence (i.e., internal structure validity, validity evidence based on external relations, concurrent validity, and cross-informant agreement).Teachers and parents of children (N=113) enrolled in preschool and childcare programs completed Phase 1 and Phase 2 of the PBSS and the Behavior Assessment System for Children (BASC-2). Cross-informant agreement between parent and teacher ratings on the PBSS was generally moderate, and internal consistency was high. PBSS Phase 1 and PBSS Phase 2 were correlated in the expected directions, and sensitivity and negative predictive value were high. Results suggest that the PBSS may yield psychometrically sound scores for helping states meet child-find requirements. Directions for future development of the PBSS are provided.