Kim May

Symptom exaggeration by college adults in attention-deficit hyperactivity disorder and learning disorder assessments.

To test the hypothesis that sub-optimal effort detected by one popular symptom validity measure, the Word Memory Test (WMT), should be interpreted as symptom exaggeration, the authors examined attention-deficit hyperactivity disorder (ADHD) and learning disorder (LD) assessment data collected from healthy adult patients over the past four years at one mid-size Southeastern college. They conducted six tests of this hypothesis, drawing upon extant research. Rates of apparent symptom exaggeration comparable to those found in medicolegal settings (e.g., personal injury cases), particularly in the context of ADHD evaluations, were found. WMT scores were positively correlated with intellectual and neurocognitive test scores, and negatively correlated with self-report symptom inventory scores. Measures of negative response bias embedded in one common self-report measure of psychopathology (the Personality Assessment Inventory) were not correlated with WMT performance. Unattended WMT administrations led to somewhat higher failure rates than were found when the examiners were present in the room during all phases of the tests administration. In light of considerable secondary gain motives in this population, the authors conclude that poor effort as evidenced by low WMT scores implies symptom exaggeration and not other factors in these assessments. The routine inclusion of empirically supported symptom validity measures in these evaluations is recommended, and future research directions are suggested.

A Monte Carlo Evaluation of Tests for Comparing Dependent Correlations

Abstract The authors conducted a Monte Carlo simulation of 8 statistical tests for comparing dependent zero-order correlations. In particular, they evaluated the Type I error rates and power of a number of test statistics for sample sizes (Ns) of 20, 50, 100, and 300 under 3 different population distributions (normal, uniform, and exponential). For the Type I error rate analyses, the authors evaluated 3 different magnitudes of the predictor-criterion correlations (py,x1 = py,x2=.1, .4, and .7). For the power analyses, they examined 3 different effect sizes or magnitudes of discrepancy between py,x2 and py,x2 (values of .1, .3, and .6). They conducted all of the simulations at 3 different levels of predictor intercorrelation (px1,x2 = .1, .3, and .6). The results indicated that both Type I error rate and power depend not only on sample size and population distribution, but also on (a) the predictor intercorrelation and (b) the effect size (for power) or the magnitude of the predictor-criterion correlations (for Type I error rate). When the authors considered Type I error rate and power simultaneously, the findings suggested that O. J. Dunn and V. A. Clarks (1969) z and E. J. Williamss (1959) t have the best overall statistical properties. The findings extend and refine previous simulation research and as such, should have greater utility for applied researchers.

DF analysis of NLSY IQ/achievement data: Nonshared environmental influences

Abstract DeFries and Fulker (1985) proposed DF analysis to measure genetic and shared environmental variance in kinship data. We use an adaption of DF analysis that can simultaneously account for genetic, shared environmental, and nonshared environmental influences within the same model. We fit this model to achievement measures from 5- to 12-year-old children from the National Longitudinal Survey of Youth (NLSY). The NLSY is a large national sample containing information to link kinship pairs at multiple levels, including cousins, half-siblings, and twins. One thousand forty-four pairs were identified by a kinship-linking algorithm. From five specific measures of intellectual ability we estimated median heritability of h 2 = .50 and shared environmental variance of c 2 = .16. We then tested for the presence of several specific nonshared influences. As predicted, differences between two related children in the number of books owned were related to differences in reading recognition scores, and trips to the museum were related to a measure of mathematical ability. A general measure of the home environment accounted for nonshared environmental variance in several specific measures of intelligence and in a general measure of cognitive ability.

A FORTRAN 77 Program for Comparing Dependent Correlations.

Suppose that a researcher is interested in determining whether the correlation between job satisfaction and salary is larger than between job satisfaction and supervisor ratings within the same sample. This would constitute a test of dependent correlations with one element in common (ρ12 = ρ23). May and Hittner (1997) and Hittner, May, and Silver (2003) performed Monte Carlo simulations on a number of statistical tests for examining this hypothesis and found that Williams’s (1959) t, Dunn and Clark’s (1969) z, Steiger’s (1980) modification of Dunn and Clark’s z using average correlations, a similar modification of Dunn and Clark’s z using average Fisher zs, and Meng, Rosenthal, and Rubin’s (1992) z were all fairly equal in terms of controlling Type I error and power. As a second example, consider the case in which a researcher is interested in determining whether the correlation between job satisfaction and salary is smaller at Time 1 than 1 year later (e.g., after a 6% raise is given across the board). This example calls for a test of dependent correlations with zero elements in common (ρ12 = ρ34). Silver, Hittner, and May (2004) performed a Monte Carlo simulation on four different procedures for testing this hypothesis and found that Dunn and Clark’s (1969) z, Steiger’s (1980) modification of Dunn and Clark’s z using average correlations, and the modification of Dunn and Clark’s z using average Fisher zs were all fairly equal in terms of controlling Type I error and power. Unfortunately, these hypothesis tests are computationally tedious and cannot readily be computed by most statistical packages. Although Hittner and May (1998) created a SAS program for testing dependent correlations, their program only tests the null hypothesis of ρ12 = ρ23. Moreover it is possible to perform tests of dependent correlations using structural equation modeling methods (Cheung & Chan, 2004), however, many researchers may not be familiar with these techniques. Hence, DEPCOR provides a user-friendly means for obtaining the procedures advocated by Hittner et al. (2003) and Silver et al. (2004) for testing dependent correlations.

Relations of Machiavellian Behavior with Sales Performance of Stockbrokers

The hypothesis of a relationship between Machiavellian behavior and sales performance of Christie and Geis was tested with a sample of 110 stockbrokers. Scores on a measure called the Machiavellian Behavior scale were positively and significantly correlated with two self-reported measures of sales performance of the stockbrokers. Present results together with those of two earlier studies supported the hypothesis that salespeople with a Machiavellian orientation are likely to be more successful. Analysis of the data also indicated predictive validity and acceptable internal consistency of the Machiavellian Behavior scale. Limitations of the present study and a need for further research are discussed.

Testing Dependent Correlations with Nonoverlapping Variables: A Monte Carlo Simulation.

The authors conducted a Monte Carlo simulation of 4 test statistics for comparing dependent correlations with no variables in common. Empirical Type I error rates and power estimates were determined for K. Pearson and L. N. G. Filons (1898) z, O. J. Dunn and V. A. Clarks (1969) z, J. H. Steigers (1980) original modification of Dunn and Clarks z, and Steigers modification of Dunn and Clarks z using a backtransformed average z procedure for sample sizes of 10, 20, 50, and 100 under 3 different population distributions. For the Type I error rate analyses, the authors evaluated 3 different magnitudes of the predictor-criterion correlations (ρ12 = ρ34 = .10, .30, and .70). Likewise, for the power analyses, 3 different magnitudes of discrepancy or effect sizes between correlations with no variables in common (ρ12 and ρ34) were examined (values of .10, .40, and .60). All of the analyses were conducted at 3 different levels of predictor intercorrelation. Results indicated that the choice as to which test statistic is optimal, in terms of power and Type I error rate, depends not only on sample size and population distribution but also on (a) the predictor intercorrelations and (b) the effect size (for power) or the magnitude of the predictor-criterion correlations (for Type I error rate). For the conditions examined in the present study, Pearson and Filons z had inflated Type I error rates when the predictor-criterion correlations were low to moderate. Dunn and Clarks z and the 2 Steiger procedures had similar, but conservative, Type I error rates when the predictor-criterion correlations were low. Moreover, the power estimates were similar among the 3 procedures.

Measuring Change Conventionally and Adaptively.

The ordinary difference or gain score is known generally to be unreliable. However, it is not widely known that when a difference score is used to measure change, the difficulty of the pretest can bias the amount of gain observed in groups that differ in initial achievement. This type of bias we call scale distortion. Using item response theory, one may compute gain scores based on differences in estimated Os (the latent trait being measured). The principal focus of the present inquiry was the degree to which scale distortion in the ordinary difference score could be removed by using differences based on estimated Os in either conventional or adaptive testing situations. Conventional tests that use these 0-based gains are shown to remove much of the scale distortion inherent in ordinary difference scores. Adaptive tests using estimated Os also offer substantial improvement in reflecting actual changes in latent proficiency.

ON THE RELATION BETWEEN POWER AND RELIABILITY OF DIFFERENCE SCORES

The potential problems which may arise from the use of difference (a.k.a., gain) scores in the measurement of change are well documented, including (a) difference scores are often negatively correlated with initial ability, and (b) they often tend to be unreliable. Conversely, they make excellent dependent variables in a true experiment since they tend to reduce variability due to individual differences among persons. This brief didactic paper presents a conclusion similar to Nicewander and Price, using a perhaps more straightforward argument based on difference scores. We argue that the same reason difference scores provide powerful significance tests, namely, reduction of “true score” variance, is also the reason they tend to be unreliable. Further, we make the point that reducing true score variance will increase the power of a significance test (since it will reduce the denominator or “error term” of the observed statistic) but will decrease reliability (since it is the numerator and a component of the denominator of the reliability coefficient).

Reliability and Validity of Gain Scores in Three Possible Scenarios

In 1996 Williams, et al. presented results on the reliability and validity of difference (gain) scores for the case where the pretest variance is less than the posttest variance. We extend this work to the remaining two possible cases—wherein the two variances are approximately equal and wherein the pretest variance exceeds the posttest variance. Plausible applied scenarios are presented for these two cases. Using these scenarios and varying the pretest-posttest reliabilities, validities, and inter-correlation, the resulting reliabilities and validities for the gain score are delineated. Our results provide the applied researcher with additional insights into the psychometric properties of gain scores in various potential situations.

A Note on Statistics for Comparing Dependent Correlations

The four statistics examined are available for use in comparing two dependent correlation coefficients (correlations between two predictors and a common criterion from a single sample wherein the predictors themselves may be correlated). There has been much past discussion in the literature of the properties and appropriate situations for these statistics. Two somewhat counterintuitive results are given here; both are examined using variables of interest to applied clinical researchers. It is shown by example that there are situations for which the observed value of these statistics increases (and thus power increases) as the predictor intercorrelation increases. Further, it is shown that there are situations for which the observed statistics increase as the magnitude (but not the difference) of the two correlations being compared increases, holding the predictor intercorrelation constant.