Hoi K. Suen

Analyzing Quantitative Behavioral Observation Data

Contents: Preface. Introduction. Behavior Sampling and Recording. Subject Sampling and Assignment. Time Sampling. Reliability: Conventional Methods. Reliability: The Generalizability Approach. Validity. Time Series Analysis: Introduction. Time Series Analysis: ARIMA. Time Series Analysis: Other Methods. Summary. Appendices: Introductory Statistics. Probabilities of the Systematic Error in Single-Subject Time Sampling for Selected Intervals. Maximum and Minimum Kappa for Given Proportion Agreement Value.

A post hoc correction procedure for systematic errors in time-sampling duration estimates

Numerous previous studies have shown that partial-interval sampling in direct observation systematically overestimates duration and underestimates frequency. Whole-interval sampling systematically underestimates both duration and frequency. This paper presents a post hoc method through which the systematic errors in duration estimates in partial-interval sampling and whole-interval sampling can be minimized.

Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment

The percentage agreement index has been and continues to be a popular measure of interobserver reliability in applied behavior analysis and child development, as well as in other fields in which behavioral observation techniques are used. An algebraic method and a linear programming method were used to assess chance-corrected reliabilities for a sample of past observations in which the percentage agreement index was used. The results indicated that, had kappa been used instead of percentage agreement, between one-fourth and three-fourth of the reported observations could be judged as unreliable against a lenient criterion and between one-half and three-fourths could be judged as unreliable against a more stringent criterion. It is suggested that the continued use of the percentage agreement index has seriously undermined the reliabilities of past observations and can no longer be justified in future studies.

Hierarchical Linear Models of Factors Associated with Public Participation among Residents Living near the US Army's Chemical Weapons Stockpile Sites

The purpose of this study was to investigate public involvement among residents living near the US Armys eight Chemical Weapons Stockpile sites. A cross-sectional study was conducted across 10 states. Primary data were obtained through a random digit dialling population survey. The study sample consisted of 8315 residents living within emergency response zones surrounding the US Armys Chemical Weapons Stockpile sites. Three research questions were investigated: (1) to what extent respondents participated in specific civic activities, and to what extent respondents intended to participate in site-related decisions; (2) to what extent psychological, social, economic and programmatic factors influenced a respondents civic participation or personal intent to participate; and (3) to what extent a respondents civic participation or personal intent to participate were individually or contextually determined. Residents in this sample exhibited moderately high levels of civic participation. Overall, residents across the sites did not differ greatly with respect to either civic participation or the intent to participate. Civic participation was primarily associated with educational attainment and family size. The intent to participate was primarily associated with the following individual variables: perceived technological attributes; perceived risk; perceived impact; perceptions towards public outreach; educational attainment; and gender. In terms of community variables, the intent to participate was also indirectly associated with the number of activist groups.

The use of momentary time sampling to assess both frequency and duration of behavior

When the time-sampling interval size can be set so it is smaller than both the shortest behavior instance and the shortest interresponse time, momentary time sampling will yield an accurate count of the frequency of behavior occurrence. It will also yield an unbiased estimate of behavior duration.

Variables influencing one-zero and instantaneous time sampling outcomes

Previous studies have shown that instantaneous time sampling results are more highly correlated with true duration than are one-zero sampling results. One-zero results, however, reflected a composite of both duration and frequency better than did instantaneous results. It is shown in this analysis that these outcomes are the logical result of the definitions of the two sampling modes. When interval lengths meet specific requirements, correct frequency counts can be obtained with either mode and systematic duration error with one-zero sampling can be eliminated by subtracting frequency from the number of intervals scored ‘one.’

Assessing the impact of the local environment on birth outcomes: a case for HLM.

Hierarchical linear Models (HLM) is a useful way to analyze the relationships between community level environmental data, individual risk factors, and birth outcomes. With HLM we can determine the effects of potentially remediable environmental conditions (e.g., air pollution) after controlling for individual characteristics such as health factors and socioeconomic factors. Methodological limitations of ecological studies of birth outcomes and a detailed analysis of the varying models that predict birth weight will be discussed. Ambient concentrations of criterion air pollutants (e.g., lead and sulfur dioxide) demonstrated a sizeable negative effect on birth weight; while the economic characteristics of the mothers residential census tract (ex. poverty level) also negatively influenced birth weight.

A Methodological Comparison of Survey Techniques in Obtaining Self-Reports of Condom-Related Behaviors

To obtain more accurate accounts of sexual attitudes and practices, researchers must explore innovative ways to overcome the reluctance of individuals to disclose sensitive and perhaps incriminating information about themselves The differences among selected modes of inquiry and survey techniques used to gather self-reports about sensitive contraceptive behaviors among young adults were examined in this study. Comparisons were made between the randomized response versus the direct-inquiry survey techniques and personal interview versus self-administered modes of inquiry relative to the reporting of sensitive condom-related sexual practices of 352 students at a large northeastern university. Findings indicated that the “controlledchoice” randomized-response technique was less effective in obtaining self-reports about condom-related practices than were direct-inquiry techniques. Recommendations for investigations are proposed.

Significance Testing; Necessary but Insufficient.

Sampling fluctuation is an inherent characteristic of data. We cannot pretend that it does not exist, nor can we abandon significance testing without a viable replacement. It remains a concern as long as we wish to draw research conclusions within a positivistic paradigm. It is there regardless of whether we conduct a large-group or a single-subject study. It is also there whether we conduct a simple norm-referenced assessment, a complex judgment-based assessment, or any other assessment. The widespread misuse of significance testing suggests that sampling fluctuation has been an overbearing guest who has become the inappropriate center of attention at the party. Instead of wishing this guest away, we should put sampling fluctuation in its proper place; namely, as but one of the numerous epistemological concerns in any positivistic research.

Reducing Task-related Variance in Performance Assessment Using Concept Maps

Task-related variance causes scores from performance assessments not to be generalizable and thus inappropriate for high stakes use. It is possible that task-related variance is due, in part, to students’ inability to transfer their knowledge consistently from one assessment task to another. Therefore, concept-mapping, a cognitive tool, might be useful to aid this transfer. This study examines the effects of concept maps on the task-related variance components of Political Science performance assessments. On three quizzes, some students used concept maps while writing two essays, while other students did not. The task variance components remained unchanged across groups, but the person main effect components increased and the task-by-person interaction components decreased for those using concept maps. Also, the scores from the concept mapping groups had higher generalizability coefficients than for those who did not use a concept map.