Youngsik Park

Generalizability theory analysis of balance error scoring system reliability in healthy young adults

CONTEXT An assessment of postural control is commonly included in the clinical concussion evaluation. Previous investigators have demonstrated learning effects that may mask concussion-induced balance decrements. OBJECTIVE To establish the test-retest reliability of the Balance Error Scoring System (BESS) and to provide recommendations that account for known learning effects. DESIGN Test-retest generalizability study. SETTING Balance research laboratory. PATIENTS OR OTHER PARTICIPANTS Young adults (n = 48) free from injuries and illnesses known to affect balance. INTERVENTION(S) Each participant completed 5 BESS trials on each of the assessment dates, which were separated by 50 days. MAIN OUTCOME MEASURE(S) Total score of the BESS was used in a generalizability theory analysis to estimate the overall reliability of the BESS and that of each facet. A decision study was completed to estimate the number of days and trials needed to establish clinical reliability. RESULTS The overall reliability of the BESS was G = 0.64. The test-retest reliability was improved when male (0.92) and female (0.91) participants were examined independently. Clinically acceptable reliability (greater than 0.80) was established when 3 BESS trials were administered in a single day or 2 trials were administered at different time points. CONCLUSIONS Learning effects have been noted in individuals with no previous exposure to the BESS. Our findings indicate that clinicians should consider interpreting the mean score from 3 BESS administrations on a given occasion for both normative data comparison and pretest and posttest design. The multiple assessment technique yields clinically reliable scores and provides the sports medicine practitioner with accurate data for clinical decision making.

PE Metrics: Background, Testing Theory, and Methods

New testing theories, concepts, and psychometric methods (e.g., item response theory, test equating, and item bank) developed during the past several decades have many advantages over previous theories and methods. In spite of their introduction to the field, they have not been fully accepted by physical educators. Further, the manner in which many assessments are developed and used in physical education has limitations, including isolated test development, weak or poor psychometric quality control, lack of evaluation frameworks, and failure to measure change or growth. To eliminate these shortcomings and meet the needs of standard-based assessment, a major national effort was undertaken to develop an item or assessment bank, called “PE Metrics,” for assessing the national content standards for physical education. After providing a brief introduction to the background of PE Metrics, this article will describe the nature of the testing theory, psychometric methods, and how they were used in the construction of PE Metrics. Constraints of developing such a system are acknowledged, and future directions in physical education assessments are outlined.

Development of PE metrics elementary assessments for national physical education standard 1

This article describes how assessments in PE Metrics were developed following six steps: (a) determining test blueprint, (b) writing assessment tasks and scoring rubrics, (c) establishing content validity, (d) piloting assessments, (e) conducting item analysis, and (f) modifying the assessments based on analysis and expert opinion. A task force, composed of researchers, measurement and evaluation experts, teacher educators, K–12 physical education teachers, and education administrators, was formulated. The task force then determined a test blueprint for Grades K, 2, and 5 and developed corresponding assessments to assess the standards. The content validity evidence was established by a panel of experts examining the degree to which the content of the assessments matched the content of the national standards, specifically Standard 1. A total of 30 assessments (Kindergarten = 8, Grade 2 = 11, and Grade 5 = 11) were developed. They were piloted to a total of 773 students (Kindergarten = 232, Grade 2 = 268, and Grade 5 = 273). Descriptive statistics (e.g., M, SD, frequency) were computed for each assessment. More than 50% of the means were between 2.2 and 2.8 (on a 4-point scoring rubric). Assessment responses were well distributed; only 2.2% had an SD of 0, which indicates that assessments were well developed. With some editorial changes, the assessments were ready for the final calibration of PE Metrics construction.

Related Critical Psychometric Issues and Their Resolutions during Development of PE Metrics.

In addition to validity and reliability evidence, other psychometric qualities of the PE Metrics assessments needed to be examined. This article describes how those critical psychometric issues were addressed during the PE Metrics assessment bank construction. Specifically, issues included (a) number of items or assessments needed, (b) training protocol for required intra- and inter-rater objectivity, and (c) the development of a score scale. First, using a subsample of data from the PE Metrics study, in which students were assessed using four assessments, the impact of the number of assessments was examined. It was found that at least two assessments are needed when applying PE Metrics for the purpose of high stakes testing. Single individual assessments can still be used in teaching practice, but the results must be interpreted with caution. Second, with the training protocol developed for PE Metrics, satisfactory intra-rater objectivity can be achieved. When two or more raters are involved in rating, however, an additional monitoring protocol should be employed so that inter-rater objectivity can be monitored and controlled. Third, to help allow for a consistent interpretation and reporting of PE Metrics results, a score scale was developed. Other related issues, such as test fairness and setting performance standards, were discussed, and future directions concerning PE Metrics maintenance and continuing development were outlined.