Mike Horton

Screening for the risk of job loss in multiple sclerosis (MS): development of an MS-specific Work Instability Scale (MS-WIS).

Background: Multiple Sclerosis (MS) mainly presents amongst those of working age. Depending upon the type of MS, many people embark upon a long period of managing their day-to-day work-related needs in the face of intermittent and sometimes persistent disabling symptoms. The objective of this study was to explore the concept of work instability (WI) following the onset of MS and develop a Work Instability Scale (WIS) specific to this population. Method: WI amongst those with MS in work was explored through qualitative interviews which were then used to generate items for a WIS. Rasch analysis was used to refine the scaling properties of the MS-WIS, which was then validated against expert vocational assessment by occupational health physiotherapists and ergonomists. Results: The resulting measure is a 22-item, self-administered scale which can be scored in three bands indicating low, medium and high risk of WI (job retention) problems. The scale meets modern psychometric requirements for measurement, indicated by adequate fit to the Rasch model with absence of local dependency and differential item functioning (DIF) by age, gender and hours worked. Conclusions: The scale presents an opportunity in routine clinical practice to take positive action to reduce sickness absence and prevent job loss.

Critical Values for Yen’s Q3: Identification of Local Dependence in the Rasch model using Residual Correlations

The assumption of local independence is central to all item response theory (IRT) models. Violations can lead to inflated estimates of reliability and problems with construct validity. For the most widely used fit statistic Q3, there are currently no well-documented suggestions of the critical values which should be used to indicate local dependence (LD), and for this reason, a variety of arbitrary rules of thumb are used. In this study, an empirical data example and Monte Carlo simulation were used to investigate the different factors that can influence the null distribution of residual correlations, with the objective of proposing guidelines that researchers and practitioners can follow when making decisions about LD during scale development and validation. A parametric bootstrapping procedure should be implemented in each separate situation to obtain the critical value of LD applicable to the data set, and provide example critical values for a number of data structure situations. The results show that for the Q3 fit statistic, no single critical value is appropriate for all situations, as the percentiles in the empirical null distribution are influenced by the number of items, the sample size, and the number of response categories. Furthermore, the results show that LD should be considered relative to the average observed residual correlation, rather than to a uniform value, as this results in more stable percentiles for the null distribution of an adjusted fit statistic.

Validation of the educational needs assessment tool as a generic instrument for rheumatic diseases in seven European countries

Objectives To validate the educational needs assessment tool (ENAT) as a generic tool for assessing the educational needs of patients with rheumatic diseases in European Countries. Methods A convenience sample of patients from seven European countries was included comprising the following diagnostic groups: ankylosing spondylitis, psoriatic arthritis, systemic sclerosis, systemic lupus erythematosus, osteoarthritis (OA) and fibromyalgia syndrome. Translated versions of the ENAT were completed through surveys in each country. Rasch analysis was used to assess the construct validity of the adapted ENATs including differential item functioning by culture (cross-cultural DIF). Initially, the data from each country and diagnostic group were fitted to the Rasch model separately, and then the pooled data from each diagnostic group. Results The sample comprised 3015 patients; the majority, 1996 (66.2%), were women. Patient characteristics (stratified by diagnostic group) were comparable across countries except the educational background, which was variable. In most occasions, the 39-item ENAT deviated significantly from the Rasch model expectations (item–trait interaction χ2 p<0.05). After correction for local dependency (grouping the items into seven domains and analysing them as ‘testlets’), fit to the model was satisfied (item–trait interaction χ2 p>0.18) in all pooled disease group datasets except OA (χ2=99.91; p=0.002). The internal consistency in each group was high (Person Separation Index above 0.90). There was no significant DIF by person characteristics. Cross-cultural DIF was found in some items, which required adjustments. Subsequently, interval-level scales were calibrated to enable transformation of ENAT scores when required. Conclusions The adapted ENAT is a valid tool with high internal consistency providing accurate estimation of the educational needs of people with rheumatic diseases. Cross-cultural comparison of educational needs is now possible.

Assessing the risk of self-harm in an adult offender population: an incidence cohort study.

BACKGROUND Self-harm is common among prisoners, particularly female prisoners. In 2007, concerned about the rising incidence, the prison service introduced a care-planning system called Assessment, Care in Custody, and Teamwork (ACCT). To date, it does not incorporate a standardised diagnostic test to estimate the risk of future self-harm. OBJECTIVE To identify potential screening instruments, or items from those instruments, to predict the risk of self-harm among prisoners. PARTICIPANTS Prisoners who had been assigned to an ACCT during the recruitment period. DESIGN A multistage prospective cohort study. Following a pilot study, instruments were administered to prisoners by interview at baseline, and followed up for 6 months (or until point of release if this was sooner) to ascertain self-harm status. Instruments were assessed for unidimensionality, scalability (Mokken) and quantitative structure (Rasch). Area under the curve (AUC) analysis was used to examine the ability of instruments and/or their items to predict future self-harm. Cox proportional hazards regression models were used to examine the multivariate predictive ability of the scales and various sociodemographic and sentencing factors. SETTING Three prisons (including one womens prison) in northern England. MAIN OUTCOME MEASURES A set of standardised questionnaires, including the Prison Screening Questionnaire (PriSnQuest), Revised Borderline Symptom List-23 (frequency-based responses) (BSL-23-F), Self-Harm Inventory (SHI), Patient Health Questionnaire (PHQ-9) and the Clinical Outcomes in Routine Evaluation - Outcome Measure (CORE-OM), together with sociodemographic and sentencing data. RESULTS In total, 450 prisoners consented to participate in the study, of whom 26% were female. The mean age of participants was 31.2 years. Over half of male prisoners recruited were on remand, compared with just over one-fifth (22.6%) of female prisoners. The average tariff of those sentenced was 41 months, of which 14.7 months, on average, had been served. Just over one-third of ACCTs had been initiated because of a known self-harm event, and over one-quarter (27.8%) of participants self-harmed during the follow-up period. Thus, almost half (46.7%) of those entered into the study were reported to have self-harmed, either from their index ACCT, or subsequently, or both. Cutting was the most frequent behaviour (51%). All screening instruments showed some evidence of unidimensionality, and four out of five showed scaling criteria consistent with ordinal scaling, so verifying the validity of the cut points. However, many showed gender bias and failure to fit the Rasch measurement model. While a resolution was made in most cases, both ordinal raw scores and latent interval scale estimates failed to show predictive value when applied within AUC analysis (0.491-0.566) or adjusted Cox proportional hazards models. However, good predictive values were shown for gender-specific sets of items, thus providing easily applied screening indexes. CONCLUSIONS While four out of five potential screening instruments were found to have acceptable psychometric properties within this setting, their predictive validity of all instruments was poor under AUC analysis. Gender-specific item sets were put together to form two screening indexes with formative indicators which gave reasonable AUC values, particularly so for females. The indexes provide identification of low-medium-high risk of self-harm, and so may help to inform potential care pathways and decisions to sign prisoners off from the ACCT. Future work should concentrate on refining a set of predictive screening items among different offender populations and investigating the time point at which this set of items should be administered. Future work may also look at the different magnitudes of risk as indicators for care pathways. FUNDING The National Institute for Health Research Health Technology Assessment programme.

Patient Reported Outcomes: misinference from ordinal scales?

Patient Reported Outcomes (PROs) are widely used in medical outcome studies, and usually take the form of administered of self-completed questionnaires. The data that these questionnaires produce is of the type known as ordinal scaling, where magnitudes of the attribute may be ascertained. At the same time, most outcome studies rely on the calculation of means, standard deviations, change scores, and concepts such as Minimally Important Difference (MID) or effect sizes. Yet, ordinal scales do not support the mathematical operations needed to calculate these type of statistic [3]. Indeed when several items are measured on an ordinal scale it is far from certain that the sum of scores has even ordinal properties [1]. Despite these constraints, these limitations are largely ignored, and thus statistics such as means and MID are widely reported for PROs. This runs the risk of drawing an incorrect inference from data based upon PROs [5]. This risk can be illustrated by considering the concepts of the ‘plateau’ and the calculation of the MID. Both are investigated by contrast of the ordinal raw score against the cardinal metric derived from fit of data to the Rasch measurement model [4]. It can be shown that as the raw score from a scale moves towards the margins, then a smaller and smaller raw score change is associated with as tandard metric unit of change. Thus patients may seem to be ‘slowing down’ in their improvement, or even ‘plateauing’ ,y et they are still moving the same metric distance. Likewise, when considering a magnitude of improvement such as an MID, the raw score distance associated with the MID can be shown to vary across the scale, depending upon the starting point. Thus for one patient the same MID may involve a change in the metric distance four times greater than that of another patient. PROs provide ordinal estimates of the magnitude of a patient on the trait being measured. Appropriate nonparametric statistics should be used. Else, where possible, the data should be converted to the cardinal metric through use of the Rasch model, which is consistent with the requirements of the theory of Additive Conjoint Measurement [2][6].

Urinary, bowel and sexual health in older men from Northern Ireland

To provide data on the prevalence of urinary, bowel and sexual dysfunction in Northern Ireland (NI), to act as a baseline for studies of prostate cancer outcomes and to aid service provision within the general population.

Assessing the risk of self-harm in an adult prison population

Abstract presented at the Australasian Professional Society on Alcohol and other Drugs Conference 2013, 24-27 November 2013, Brisbane, Australiapresented at the Australasian Professional Society on Alcohol and other Drugs Conference 2013, 24-27 November 2013, Brisbane, AustraliaAbstract presented at the Australasian Professional Society on Alcohol and other Drugs Conference 2013, 24-27 November 2013, Brisbane, AustraliaAbstract presented at the Australasian Professional Society on Alcohol and other Drugs Conference 2013, 24-27 November 2013, Brisbane, AustraliaAbstract presented at the Australasian Professional Society on Alcohol and other Drugs Conference 2013, 24-27 November 2013, Brisbane, Australia

Applying Rasch analysis to the SF-36 physical function scale: effect of dependent items

Medical outcome studies often use Patient Reported Outcomes (PROs), and these often take the form of administered or self-completed questionnaires. Often, the responses from these scales are simply added up as a total, and this total score is utilised for outcome purposes. However, for this approach to be valid, there are a number of underlying assumptions that are being made. One of these assumptions is that of Local Independence, which comprises two aspects; the items making up the scale should all be unidimensional (trait dependency) and, the response to an item should not directly influence the response to another item within the set (response dependency) [1]. These assumptions apply to the Rasch model (and other IRT models). Thus the process of Rasch analysis provides a means to test these assumptions, along with other key properties such as invariance. Where data satisfy these assumptions, and fit to the model, an interval scale transformation becomes available. A secondary analysis was performed on data from the Physical Function scale (PF-10) of the SF-36 [2] to demonstrate the application of Rasch Analysis and to investigate the influence of response dependency within the dataset. Initial results showed evidence of a lack of unidimensionality (t-tests = 8.39%; lower bound CI = 5.9%), along with apparent response dependencies between the three walking items of the PF-10 (walking 100 yards, walking half a mile & walking more than a mile), the two climbing stairs items of the PF-10 (climbing one flight of stairs & climbing several flights of stairs) and a dependency between the ‘moderate activities’ item and the ‘lifting or carrying groceries’ item. When adjustments had been made for the response dependency within the item set, it then appeared to be unidimensional (t-tests = 2.1%). Also to be noted is the artificial inflation of the reliability index due to the dependency within the item set. After dependency adjustments had been made, the reliability index (PSI) dropped from 0.851 (initially) to 0.789 (Cronbach’s alpha dropped from 0.91 to 0.80). In conclusion, response dependency can inflate reliability, and lead to spurious (dependency) factors. It has also been shown to influence both item and person parameters in IRT analysis. Rasch analysis provides a framework to assess, as well as adjust for, response dependency within an item set.

Development of an item pool for a needs-based measure of quality of life of carers of a family member with dementia

Background and ObjectivesThis paper describes the development of an item pool for a needs-based self-report outcome measure of the impact of caring for a relative, friend or neighbour with dementia on carer quality of life. The aims are to give a detailed account of the steps involved and describe the resulting item pool.MethodsSeven steps were followed: generation of an initial item set drawing on 42 needs-led interviews with carers; a content and face validity check; assessment of psychometric potential; testing of response formats; pre-testing through cognitive interviews with 22 carers; administration rehearsal with two carers; and final review.ResultsAn initial set of 99 items was refined to a pool of 70 to be answered using a binary response format. Items were excluded due to overlap with others, ceiling effects, ambiguity, dependency on function of the person with dementia or two-part phrasing. Items retained covered a breadth of areas of impact of caring and were understandable and acceptable to respondents.ConclusionsThe resulting dementia carer-specific item pool reflects the accounts of a diverse sample of those who provide care for a person with dementia, allowing them to define the nature of the impact on their lives and resulting in a valid, acceptable set of items.

The psychometric properties of the Evaluation of Daily Activity Questionnaire in seven musculoskeletal conditions

Abstract Purpose: The purpose of this study is to psychometrically test the Evaluation of Daily Activity Questionnaire in seven musculoskeletal conditions. Materials and methods: One thousand and two hundred people with ankylosing spondylitis; osteoarthritis; systemic lupus erythematosus; systemic sclerosis; chronic pain; chronic upper limb disorders; or Primary Sjögren’s syndrome completed the Evaluation of Daily Activity Questionnaire, Health Assessment Questionnaire and Short-Form Health Survey v2. We examined internal construct validity using Rasch analysis, internal consistency, concurrent validity with the Health Assessment Questionnaire and Short-Form Health Survey v2. Participants repeated the Evaluation of Daily Activity Questionnaire to assess test–retest reliability. Results: The 12 domains satisfied Rasch model expectations for fit, local dependency, unidimensionality and invariance by age and gender, in each musculoskeletal condition. Internal consistency was consistent with individual use (Cronbach’s α > 0.90); concurrent validity was strong (Health Assessment Questionnaire: rs = 0.60–0.92; Short-Form Health Survey v2 Physical Function: rs = −0.61 to −0.91) and test–retest reliability excellent (Intra-Class Correlation Coefficient(2,1) = 0.77–0.96). Conclusion: The Evaluation of Daily Activity Questionnaire satisfied Rasch model requirements for construct validity and has good reliability and validity in each MSC. The Evaluation of Daily Activity Questionnaire can be used as a measure of everyday activity in practice and research with people with musculoskeletal conditions. Implications for rehabilitation The Evaluation of Daily Activity Questionnaire evaluates users’ ability to perform common daily activities (in 12 domains) that were identified as problematic by people with seven musculoskeletal conditions (i.e., osteoarthritis, systemic lupus, ankylosing spondylitis, chronic pain, chronic upper limb conditions, systemic sclerosis, Sjogren’s syndrome). Most patients considered the Evaluation of Daily Activity Questionnaire was the right length and would be helpful for discussing everyday problems with an occupational therapist. The 12 domains have good reliability and validity and can be combined into two components: Self-Care and Mobility. The Evaluation of Daily Activity Questionnaire is suitable for use both in clinical practice and research and a User Manual is available for therapists and researchers.