W. Todd Rogers

Assessing the Reliability of Patient, Nurse, and Family Caregiver Symptom Ratings in Hospitalized Advanced Cancer Patients

PURPOSE The purpose of this study was to examine the reliability of symptom assessments in advanced cancer patients under various conditions, including multiple raters (patients, nurses, and family caregivers), occasions, and symptoms. PATIENTS AND METHODS The study sample consisted of 32 advanced cancer patients admitted to a tertiary palliative care unit. Symptom assessments were completed for each patient on two separate occasions, approximately 24 hours apart. On each occasion, the patient, the primary care nurse, and a primary family caregiver independently completed an assessment using the Edmonton Symptom Assessment System (ESAS). The ESAS is a nine-item visual analogue scale for assessing symptoms in palliative patients. The reliability of the assessments (r) was examined using generalizability theory. RESULTS Three important findings emerged from this analysis. First, the analysis of individual symptom ratings provided a more meaningful representation of the symptom experience than total symptom distress ratings. Secondly, patients, nurses, and caregivers varied in their ratings across different patients, as well as in their ratings of shortness of breath, which may have been a result of individual rater variability. Finally, reliability estimates (r), based on a single rater and one occasion, were less than.70 for all symptoms, except appetite. These estimates improved substantially (r >/=.70) for all symptoms except anxiety and shortness of breath, using three raters on a single occasion or two raters across two occasions. CONCLUSION The findings from this study reinforce the need for the development of an integrated symptom assessment approach that combines patient and proxy assessments. Further research is needed to explore individual differences among raters.

An Empirical Comparison of Three-and Four-Choice Items and Tests: Susceptibility to Testwiseness and Internal Consistency Reliability:

Theoretical and test simulation work reveals that under the knowledge-or-randomguessing assumption, three-option item tests are at least as good as four-option item tests in terms of item discrimination and internal consistency. Of concern, however, is the finding that multiple-choice items may be susceptible to testwiseness, thereby contradicting the random-guessing assumption. Both item-level and test-level characteristics were examined for items included in a high stakes school-leaving mathematics examination. As expected, the influence of testwiseness is lessened when three-option items are used instead of four-option items. Differences and nondifferences between the psychometric characteristics of the three-option and four-option test forms tend to agree with the findings of earlier studies: Tests consisting of three-option items are at least equivalent to tests composed of four options in terms of internal consistency score reliability, difficulty is inversely related to the number of options, and the findings for item discrimination are not conclusive.

Principles for Fair Student Assessment Practices for Education in Canada

Abstract Developed in response to examples of unacceptable assessment practices, the Principles for Fair Student Assessment Practices for Education in Canada contains a set of principles and guidelines generally accepted by professional organisations as indicative of fair assessment practice within the Canadian educational context. Assessments depend on professional judgment: the ’Principles’ identify the issues to consider in exercising this professional judgment and in striving for the fair and equitable assessment of all students.

A Confirmatory Factor Analysis of the Test Anxeety Inventory Using Canadian High School Students

The Test Anxiety Inventory (TAI) is one of the most widely used surveys to assess overall test anxiety. In addition, scores are available for two components of anxiety: emotionality and worry. However, there is controversy about the nature of these two components. The authors of the TAI suggest that these two factors are orthogonal, whereas others suggest that they are oblique. Working with a sample of 335 male and 389 female school-leaving students from Canada, confirmatory factor analyses revealed that a two-factor oblique model provided the best fit to the data. The two-factor oblique model was also found to be invariant across sex. However, given that the means differed between males and females, the use of separate norms should be continued.

Test-Wiseness: Its Nature and Application

If a test taker possesses test-wiseness and relevant partial knowledge, and if a test contains susceptible items, then the combination of these factors can result in improved or higher scores. In this article, test-wiseness is defined and described in terms of the elements that comprise test-wiseness. A model of test-wise test taking behavior is presented that shows the need for relevant partial knowledge in the application of test-wiseness. A review of the correlates, including race or ethnicity, of test-wiseness is then provided, followed by a review of the effects of training programs designed to minimize the differences in test-wiseness among examinees. The paper concludes with the recommendation, made in the interest of fairness to all examinees, that multifaceted, multimedia training programs directed toward the acquisition of test-wiseness be included as a regular part of the school program at the junior high school level.

Assessment purposes and procedures in ESL/EFL classrooms

University instructors’ classroom assessments play a central role in and inevitably influence their teaching and their students’ learning. This paper reports on a comparative interview study conducted in a range of three ESL/EFL university contexts in Canada, Hong Kong and China. Six major aspects of ESL/EFL classroom assessment practices were explored: instructors’ assessment planning for the courses they taught, the relative weight given to course work and tests in their instruction, the type of assessment methods (selection vs. supply methods) that they used, the purposes each assessment was used for, the source of each method used, and when they used each method. University instructors were also asked to indicate what they saw as the advantages and disadvantages of the methods they used, and whether they took into account prior student knowledge when making decisions about what assessment methods to use. The findings contribute to a better understanding of ESL/EFL university instructors’ classroom assessment practices at the tertiary level in a range of three ESL/EFL university teaching contexts.

Nature and Determinants of the Course of Chronic Low Back Pain Over a 12-Month Period: A Cluster Analysis

Background It has been suggested that low back pain (LBP) is a condition with an unpredictable pattern of exacerbation, remission, and recurrence. However, there is an incomplete understanding of the course of LBP and the determinants of the course. Objective The purposes of this study were: (1) to identify clusters of LBP patients with similar fluctuating pain patterns over time and (2) to investigate whether demographic and clinical characteristics can distinguish these clusters. Design This study was a secondary analysis of data extracted from a randomized controlled trial. Methods Pain scores were collected from 155 participants with chronic nonspecific LBP. Pain intensity was measured monthly over a 1-year period by mobile phone short message service. Cluster analysis was used to identify participants with similar fluctuating patterns of pain based on the pain measures collected over a year, and t tests were used to evaluate if the clusters differed in terms of baseline characteristics. Results The cluster analysis revealed the presence of 3 main clusters. Pain was of fluctuating nature within 2 of the clusters. Out of the 155 participants, 21 (13.5%) had fluctuating pain. Baseline disability (measured with the Roland-Morris Disability Questionnaire) and treatment groups (from the initial randomized controlled trial) were significantly different in the clusters of patients with fluctuating pain when compared with the cluster of patients without fluctuating pain. Limitations A limitation of this study was the fact that participants were undergoing treatment that may have been responsible for the rather positive prognosis observed. Conclusions A small number of patients with fluctuating patterns of pain over time were identified. This number could increase if individuals with episodic pain are included in this fluctuating group.

CONTEXTUAL AND SCHOOL FACTORS ASSOCIATED WITH ACHIEVEMENT ON A HIGH‐STAKES EXAMINATION

This study identified student and school ‐ level factors associated with student achievement on the Ontario Secondary School Literacy Test (OSSLT), an examination that includes a student questionnaire examining home literacy practices. Linked student and school contextual data enabled the use of hierarchical linear modeling to complete the analyses and examine both student and school level effects. Fourteen student and three school level variables were found to be associated with students’ reading achievement and twelve student and two school level variables were associated with writing. Significant variations between schools were also found for students with individualized education plans, English as a second language, or previous eligibility on the OSSLT. Key words: hierarchical linear modeling, literacy testing, achievement, education factors Dans cette etude, les auteurs ont identifie les facteurs eleves et niveaux scolaires associes au rendement scolaire dans le Test de competences linguistiques de l’Ontario, un examen qui comprend un questionnaire pour les eleves au sujet des habitudes en matiere de litteratie a la maison. Des donnees reliees aux eleves et au contexte scolaire ont permis l’utilisation d’une modelisation lineaire hierarchique en vue de completer les analyses et d’etudier les effets quant aux eleves et aux niveaux scolaires. Les auteurs ont etabli que quatorze variables ayant trait aux eleves et trois ayant trait aux niveaux scolaires etaient associees au rendement des eleves en matiere de lecture et que douze variables ayant trait aux eleves et deux ayant trait aux niveaux scolaires etaient associees a l’ecriture. Des ecarts importants entre les ecoles ont egalement ete notes chez les eleves ayant des programmes pedagogiques sur mesure, inscrits en anglais langue seconde ou deja admissibles au Test de competences linguistiques des ecoles secondaires de l’Ontario. Mots cles : modelisation lineaire hierarchique, test de competences linguistiques, rendement scolaire, facteurs pedagogiques

Teachers' Perceptions of Large-Scale Assessment Programs within Low-Stakes Accountability Frameworks.

The intent of this study was to examine the views of teachers regarding the appropriateness of the purposes and uses of the provincial assessments in Alberta and Ontario and the seriousness of the concerns raised about these assessments. These provinces represent educational jurisdictions that use large-scale assessments within a low-stakes accountability framework in which the results are intended to be used to support school-based improvement efforts. Despite being implemented at different times (1982–Alberta; 1996–Ontario), teachers in both provinces appear to hold relatively similar views about the testing programs in their own provinces and the issues associated with these programs. Teachers’ concerns regarding the use of these assessments for accountability purposes and the potential misuses of the results appear to be a dominant influence on teachers’ generally low ratings of other purposes and uses of the assessments in both provinces, including those related to improving instruction and learning.

Investigation of IRT-Based Equating Methods in the Presence of Outlier Common Items

Common items with inconsistent b-parameter estimates may have a serious impact on item response theory (IRT)—based equating results. To find a better way to deal with the outlier common items with inconsistent b-parameters, the current study investigated the comparability of 10 variations of four IRT-based equating methods (i.e., concurrent calibration, separate calibration with test characteristic curve [TCC] and mean/sigma [M/S] transformations, and calibration with fixed common item parameters [FCIP]) when outliers were either ignored or considered. Simulated data were generated for the common-item nonequivalent groups matrix design to reflect the manipulated factors: group ability differences and nonequivalent groups, number/score points of outliers, and types of outliers. When no outliers were present, the TCC and M/S transformations performed the best. When there were outliers, overall, the methods that considered them (except the M/S transformation with outliers weighted) resulted in a vast improvement compared to the methods that ignored them.