H. G. Osburn

A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization.

This article compares the accuracy of several formulas for the standard error of the mean uncorrected correlation in meta-analytic and validity generalization studies. The effect of computing the mean correlation by weighting the correlation in each study by its sample size is also studied. On the basis of formal analysis and simulation studies, it is concluded that the common formula for the sampling variance of the mean correlation, V r =V/ r /K where K is the number of studies in the meta-analysis, gives reasonably accurate results. This formula gives accurate results even when sample sizes and ps are unequal and regardless of whether or not the statistical artifacts vary from study to study. It is also shown that using sample-size weighting may result in underestimation of the standard error of the mean uncorrected correlation when there are outlier sample sizes

A Method for Maximizing Split-Half Reliability Coefficients

An efficient algorithm (MSPLIT) for maximizing split-half reliability coefficients is described. Coefficients derived by the algorithm were found to be generally larger than odd-even split-half coefficients and KR-20 coefficients and were nearly as large as the largest of the coefficients from among every possible split-half arrangement.

Accuracy of Corrections for Restriction in Range Due to Explicit Selection in Heteroscedastic and Nonlinear Distributions.

Corrections for restriction in range due to explicit selection assume linearity of regression and homoscedastic array variances. Violation of one or both of these assumptions can lead to serious errors in estimating the unrestricted population correlation. A Monte Carlo study was conducted to examine the effects of some common forms of violation of these assumptions. The results suggested that the corrected correlation provides a fairly good estimate of the population correlation for moderately restricted samples but becomes progressively less accurate with increasing degrees of restriction. In addition, corrected estimates were generally more accurate than corresponding uncorrected estimates of the population correlation.

Validity of Isometric Strength Tests for Predicting the Capacity to Crack, Open, and Close Industrial Valves

Cracking, opening, and closing valves are physically demanding tasks required of chemical plant process operators. This study determined if isometric strength tests predicted the capacity to: 1) crack valves; and 2) fully open or close them. The study involved three interrelated steps: 1) compete task analyses to define the torque required to crack valves and the total amount of work required to open or close industrial valves; 2) develop valve turning simulation tests; and 3) complete laboratory studies to define the level of isometric strength demanded for valve turning performance. A total of 405 valves at two major chemical plants were measured to find the torque required to crack, open, and close industrial valves. These data were used to develop two job simulation tests, one measured valve cracking capacity, and the second the endurance needed to open or close a valve. An electronic torque wrench measured valve cracking capacity in eight different ways, and a valve turning ergometer measured the subjects endurance to work for 15 minutes at a power output of 1,413.5 foot-pounds/minute. The sum of isometric grip, arm lift, and torso lift strength tests (ɛIS) measured strength. The isometric strength and endurance work valve tests were administered to 26 men and 25 women. The isometric strength and valve cracking tests were administered to a second sample of 118 men and 66 women. The correlations between ɛIS and work test performance were 0.65 and 0.83 for valve cracking and valve endurance tests respectively. Logistic regression models defined the strength level needed to crack valves and completely close valves.

A Note on Design of Test Experiments

THE theory of generslizability distinguishes between experiments in which items are matched and experiments in which items are unmatched (Cronbach, Rajaratnam and Gleser, 1963). These writers have shown that the unmatched item design leads to different reliability formulas for estimating the generalizability of a test. This paper shows that the unmatched item experiment has certain advantages over the usual matched item experiment depending upon the objectives of the study. Items are said to be matched if for every item in the test there is an zpr for every person in the sample of examinees. Items are said to be unmatched when the items are selected randomIy and independently for each person. Most studies using psychological tests have used the matched item design, and it might seem that unmatched item designs would be something to avoid. This is definitely not the case. It turns out that unmatched item designs have important applications in test experiments as will be shown below. Suppose tha t we are sampling from a population of persons, P, and a population of items, I , and we wish to generalize over both populations. Consider the following experiments.

A Computer Program for Maximizing and Cross-Validating Split-Half Reliability Coefficients

A FORTRAN program for maximizing and cross-validating split-half reliability coefficients is described. Externally computed arrays of item means and covariances are to be supplied as input for each of two samples of persons. The user may select a number of subsets from the complete set of items for analysis in a single run.

The Effect of Item Stratification on Errors of Measurement.

THIS paper shows that, in the case of matched item tests, the reduction in errors of measurement for tests constructed by stratified sampling as compared with tests constructed by random sampling from an infinite population of items is a simple function of the variance of the difference between pairs of strata true scores. For unmatched item tests the reduction in errors of measurement due to stratification is a function of the variance (across strata) of the strata mean true scores plus the variance of the difference between pairs of strata true scores. These results predict that, in the case of matched item tests the largest reductions in errors of measurement will result from stratification on item content rather than item difficulty while for unmatched item tests just the opposite is true.

Computer-Aided Item Sampling for Achievement Testing: A Description of a Computer Program Implementing the Universe Defined Test Concept.

A universe defined test is a test constructed and administered in such a way that an examinee’s score on the test provides an unbiased estimate of his score on some explicitly defined universe of item content (Osburn, 1968). Two general requirements for test construction are implied in the above definition. The first is that all items which could possibly appear in the test should be specified in advance; this implies that for a specified content area it is possible to clearly define a content population of items. Secondly, the items in a particular test should be selected by random sampling or stratified random sampling from the universe (or population) of items defining the content area. The idea of random sampling from a universe of test items is not new. Psychometric theorists have used this idea extensively in building reliability and other testing models. The point is that few test constructors have seriously attempted to implement the model in actual practice. Osburn (1968), however, has defined a procedure for implementing in practice the universe defined test concept. The focus of the present paper is not an elaboration of the implications of the universe defined test for test construction

Validity of Isometric Strength Tests for Predicting Performance in Physically Demanding Tasks

Job analysis has shown that many tasks in the energy industry are physically demanding. This study examined the validity of an isometric strength test battery for predicting performance in work sample tests that simulated physically demanding tasks in coal mining and oil production. Since two of the work sample tests involved an endurance component, metabolically determined VO(2) (L/Min) was used to measure arm endurance. All tests were administered to 25 male and 25 female physically fit subjects. Correlations between the isometric strength tests and the work sample tests ranged from .67 to .93. In addition, the isometric strength tests were more highly correlated with the endurance work sample tests than metabolically measured arm endurance. Slopes and intercepts of the male and female regression lines were homogeneous. These results support the validity of isometric strength tests for predicting performance in physically demanding jobs.

Bias in validity generalization variance estimates: a reply to Hoben Thomas

In a recent article Thomas (1988) derived the expected value of the true validity variance estimate used in validity generalization studies. Based on computations of the expected values for certain scenarios, Thomas made a number of critical assertions regarding the variance estimate. This article shows that Thomass arguments regarding deficiencies in the variance estimate used in validity generalization studies are misleading