Elizabeth E. Marfeo

Conceptual Foundation for Measures of Physical Function and Behavioral Health Function for Social Security Work Disability Evaluation

Physical and mental impairments represent the 2 largest health condition categories for which workers receive Social Security disability benefits. Comprehensive assessment of physical and mental impairments should include aspects beyond medical conditions such as a persons underlying capabilities as well as activity demands relevant to the context of work. The objective of this article is to describe the initial conceptual stages of developing new measurement instruments of behavioral health and physical functioning relevant for Social Security work disability evaluation purposes. To outline a clear conceptualization of the constructs to be measured, 2 content models were developed using structured and informal qualitative approaches. We performed a structured literature review focusing on work disability and incorporating aspects of the International Classification of Functioning, Disability and Health as a unifying taxonomy for framework development. Expert interviews provided advice and consultation to enhance face validity of the resulting content models. The content model for work-related behavioral health function identifies 5 major domains: (1) behavior control, (2) basic interactions, (3) temperament and personality, (4) adaptability, and (5) workplace behaviors. The content model describing physical functioning includes 3 domains: (1) changing and maintaining body position, (2) whole-body mobility, and (3) carrying, moving, and handling objects. These content models informed subsequent measurement properties including item development and measurement scale construction, and provided conceptual coherence guiding future empirical inquiry. The proposed measurement approaches show promise to comprehensively and systematically assess physical and behavioral health functioning relevant to work.

Development of a computer-adaptive physical function instrument for Social Security Administration disability determination.

OBJECTIVES To develop and test an instrument to assess physical function for Social Security Administration (SSA) disability programs, the SSA-Physical Function (SSA-PF) instrument. Item response theory (IRT) analyses were used to (1) create a calibrated item bank for each of the factors identified in prior factor analyses, (2) assess the fit of the items within each scale, (3) develop separate computer-adaptive testing (CAT) instruments for each scale, and (4) conduct initial psychometric testing. DESIGN Cross-sectional data collection; IRT analyses; CAT simulation. SETTING Telephone and Internet survey. PARTICIPANTS Two samples: SSA claimants (n=1017) and adults from the U.S. general population (n=999). INTERVENTIONS None. MAIN OUTCOME MEASURES Model fit statistics, correlation, and reliability coefficients. RESULTS IRT analyses resulted in 5 unidimensional SSA-PF scales: Changing & Maintaining Body Position, Whole Body Mobility, Upper Body Function, Upper Extremity Fine Motor, and Wheelchair Mobility for a total of 102 items. High CAT accuracy was demonstrated by strong correlations between simulated CAT scores and those from the full item banks. On comparing the simulated CATs with the full item banks, very little loss of reliability or precision was noted, except at the lower and upper ranges of each scale. No difference in response patterns by age or sex was noted. The distributions of claimant scores were shifted to the lower end of each scale compared with those of a sample of U.S. adults. CONCLUSIONS The SSA-PF instrument contributes important new methodology for measuring the physical function of adults applying to the SSA disability programs. Initial evaluation revealed that the SSA-PF instrument achieved considerable breadth of coverage in each content domain and demonstrated noteworthy psychometric properties.

Development of an Instrument to Measure Behavioral Health Function for Work Disability: Item Pool Construction and Factor Analysis

OBJECTIVES To develop a broad set of claimant-reported items to assess behavioral health functioning relevant to the Social Security disability determination processes, and to evaluate the underlying structure of behavioral health functioning for use in development of a new functional assessment instrument. DESIGN Cross-sectional. SETTING Community. PARTICIPANTS Item pools of behavioral health functioning were developed, refined, and field tested in a sample of persons applying for Social Security disability benefits (N=1015) who reported difficulties working because of mental or both mental and physical conditions. INTERVENTIONS None. MAIN OUTCOME MEASURE Social Security Administration Behavioral Health (SSA-BH) measurement instrument. RESULTS Confirmatory factor analysis (CFA) specified that a 4-factor model (self-efficacy, mood and emotions, behavioral control, social interactions) had the optimal fit with the data and was also consistent with our hypothesized conceptual framework for characterizing behavioral health functioning. When the items within each of the 4 scales were tested in CFA, the fit statistics indicated adequate support for characterizing behavioral health as a unidimensional construct along these 4 distinct scales of function. CONCLUSIONS This work represents a significant advance both conceptually and psychometrically in assessment methodologies for work-related behavioral health. The measurement of behavioral health functioning relevant to the context of work requires the assessment of multiple dimensions of behavioral health functioning. Specifically, we identified a 4-factor model solution that represented key domains of work-related behavioral health functioning. These results guided the development and scale formation of a new SSA-BH instrument.

Development of a Self-Report Physical Function Instrument for Disability Assessment: Item Pool Construction and Factor Analysis

OBJECTIVES To build a comprehensive item pool representing work-relevant physical functioning and to test the factor structure of the item pool. These developmental steps represent initial outcomes of a broader project to develop instruments for the assessment of function within the context of Social Security Administration (SSA) disability programs. DESIGN Comprehensive literature review; gap analysis; item generation with expert panel input; stakeholder interviews; cognitive interviews; cross-sectional survey administration; and exploratory and confirmatory factor analyses to assess item pool structure. SETTING In-person and semistructured interviews and Internet and telephone surveys. PARTICIPANTS Sample of SSA claimants (n=1017) and a normative sample of adults from the U.S. general population (n=999). INTERVENTIONS Not applicable. MAIN OUTCOME MEASURE Model fit statistics. RESULTS The final item pool consisted of 139 items. Within the claimant sample, 58.7% were white; 31.8% were black; 46.6% were women; and the mean age was 49.7 years. Initial factor analyses revealed a 4-factor solution, which included more items and allowed separate characterization of: (1) changing and maintaining body position, (2) whole body mobility, (3) upper body function, and (4) upper extremity fine motor. The final 4-factor model included 91 items. Confirmatory factor analyses for the 4-factor models for the claimant and the normative samples demonstrated very good fit. Fit statistics for claimant and normative samples, respectively, were: Comparative Fit Index=.93 and .98; Tucker-Lewis Index=.92 and .98; and root mean square error approximation=.05 and .04. CONCLUSIONS The factor structure of the physical function item pool closely resembled the hypothesized content model. The 4 scales relevant to work activities offer promise for providing reliable information about claimant physical functioning relevant to work disability.

Scale refinement and initial evaluation of a behavioral health function measurement tool for work disability evaluation.

OBJECTIVE To use item response theory (IRT) data simulations to construct and perform initial psychometric testing of a newly developed instrument, the Social Security Administration Behavioral Health Function (SSA-BH) instrument, that aims to assess behavioral health functioning relevant to the context of work. DESIGN Cross-sectional survey followed by IRT calibration data simulations. SETTING Community. PARTICIPANTS Sample of individuals applying for Social Security Administration disability benefits: claimants (n=1015) and a normative comparative sample of U.S. adults (n=1000). INTERVENTIONS None. MAIN OUTCOME MEASURE SSA-BH measurement instrument. RESULTS IRT analyses supported the unidimensionality of 4 SSA-BH scales: mood and emotions (35 items), self-efficacy (23 items), social interactions (6 items), and behavioral control (15 items). All SSA-BH scales demonstrated strong psychometric properties including reliability, accuracy, and breadth of coverage. High correlations of the simulated 5- or 10-item computer adaptive tests with the full item bank indicated robust ability of the computer adaptive testing approach to comprehensively characterize behavioral health function along 4 distinct dimensions. CONCLUSIONS Initial testing and evaluation of the SSA-BH instrument demonstrated good accuracy, reliability, and content coverage along all 4 scales. Behavioral function profiles of Social Security Administration claimants were generated and compared with age- and sex-matched norms along 4 scales: mood and emotions, behavioral control, social interactions, and self-efficacy. Using the computer adaptive test-based approach offers the ability to collect standardized, comprehensive functional information about claimants in an efficient way, which may prove useful in the context of the Social Security Administrations work disability programs.

Combining agreement and frequency rating scales to optimize psychometrics in measuring behavioral health functioning

OBJECTIVE The goal of this article was to investigate optimal functioning of using frequency vs. agreement rating scales in two subdomains of the newly developed Work Disability Functional Assessment Battery: the Mood & Emotions and Behavioral Control scales. STUDY DESIGN AND SETTING A psychometric study comparing rating scale performance embedded in a cross-sectional survey used for developing a new instrument to measure behavioral health functioning among adults applying for disability benefits in the United States was performed. RESULTS Within the sample of 1,017 respondents, the range of response category endorsement was similar for both frequency and agreement item types for both scales. There were fewer missing values in the frequency items than the agreement items. Both frequency and agreement items showed acceptable reliability. The frequency items demonstrated optimal effectiveness around the mean ± 1-2 standard deviation score range; the agreement items performed better at the extreme score ranges. CONCLUSION Findings suggest an optimal response format requires a mix of both agreement-based and frequency-based items. Frequency items perform better in the normal range of responses, capturing specific behaviors, reactions, or situations that may elicit a specific response. Agreement items do better for those whose scores are more extreme and capture subjective content related to general attitudes, behaviors, or feelings of work-related behavioral health functioning.

Work-related measures of physical and behavioral health function: Test-retest reliability

BACKGROUND The Work Disability Functional Assessment Battery (WD-FAB), developed for potential use by the US Social Security Administration to assess work-related function, currently consists of five multi-item scales assessing physical function and four multi-item scales assessing behavioral health function; the WD-FAB scales are administered as Computerized Adaptive Tests (CATs). OBJECTIVE The goal of this study was to evaluate the test-retest reliability of the WD-FAB Physical Function and Behavioral Health CATs. METHODS We administered the WD-FAB scales twice, 7-10 days apart, to a sample of 376 working age adults and 316 adults with work-disability. Intraclass correlation coefficients were calculated to measure the consistency of the scores between the two administrations. Standard error of measurement (SEM) and minimal detectable change (MDC90) were also calculated to measure the scales precision and sensitivity. RESULTS For the Physical Function CAT scales, the ICCs ranged from 0.76 to 0.89 in the working age adult sample, and 0.77-0.86 in the sample of adults with work-disability. ICCs for the Behavioral Health CAT scales ranged from 0.66 to 0.70 in the working age adult sample, and 0.77-0.80 in the adults with work-disability. The SEM ranged from 3.25 to 4.55 for the Physical Function scales and 5.27-6.97 for the Behavioral Health function scales. For all scales in both samples, the MDC90 ranged from 7.58 to 16.27. CONCLUSION Both the Physical Function and Behavioral Health CATs of the WD-FAB demonstrated good test-retest reliability in adults with work-disability and general adult samples, a critical requirement for assessing work related functioning in disability applicants and in other contexts.

Do claimants over-report behavioral health dysfunction when filing for work disability benefits?

BACKGROND Questions exist related to the best way to use medical evidence relative to self-report as part of the SSA disability determination process. OBJECTIVE To examine concordance between provider and claimant responses along the four dimensions of work related behavioral health functioning: Social Interactions, Mood and Emotions, Behavioral Control, and Self-Efficacy. METHODS Using secondary data from a larger study, which collected data on individuals reporting difficulties with work (claimants) due to mental conditions, 39 items were completed by claimants and their healthcare provider. Inter-rater agreement was assessed using three techniques: Cohens kappa, percent absolute agreement, and folded mountain plots. RESULTS A sample of 65 dyads was obtained. Inter-rater agreement was low for most items (k=0.0-0.20) with a minority of items having fair agreement (k=0.21-0.40) Percent agreement was fair: Mood and Emotions (46%), Self-Efficacy (44%), Behavioral Control (39%) and Social Interactions (38%). Overall, providers reported lower functioning compared to claimants for the Behavioral Control and Self-Efficacy scales; the reverse trend held for the Mood and Emotions scale. CONCLUSIONS Results indicate discordance between provider and claimant report of behavioral health functioning. Understanding reasons for and approaches to reconciling the inconsistencies between claimant and provider perspectives is a complex task. These findings have implications for how best to assess mental and behavioral-health related work disability in the absence of an established gold standard measure.

Improving measures of work-related physical functioning

Purpose To expand content of the physical function domain of the Work Disability Functional Assessment Battery (WD-FAB), developed for the US Social Security Administration’s (SSA) disability determination process.MethodsNewly developed questions were administered to 3532 recent SSA applicants for work disability benefits and 2025 US adults. Factor analyses and item response theory (IRT) methods were used to calibrate and link the new items to the existing WD-FAB, and computer-adaptive test simulations were conducted.ResultsFactor and IRT analyses supported integration of 44 new items into three existing WD-FAB scales and the addition of a new 11-item scale (Community Mobility). The final physical function domain consisting of: Basic Mobility (56 items), Upper Body Function (34 items), Fine Motor Function (45 items), and Community Mobility (11 items) demonstrated acceptable psychometric properties.ConclusionsThe WD-FAB offers an important tool for enhancement of work disability determination. The FAB could provide relevant information about work-related functioning for initial assessment of claimants; identifying denied applicants who may benefit from interventions to improve work and health outcomes; enhancing periodic review of work disability beneficiaries; and assessing outcomes for policies, programs and services targeting people with work disability.

INTERPRETING PHYSICAL AND BEHAVIORAL HEALTH SCORES FROM NEW WORK DISABILITY INSTRUMENTS

OBJECTIVE To develop a system to guide interpretation of scores generated from 2 new instruments measuring work-related physical and behavioral health functioning (Work Disability - Physical Function (WD-PF) and WD - Behavioral Function (WD-BH)). DESIGN Cross-sectional, secondary data from 3 independent samples to develop and validate the functional levels for physical and behavioral health functioning. SUBJECTS Physical group: 999 general adult subjects, 1,017 disability applicants and 497 work-disabled subjects. Behavioral health group: 1,000 general adult subjects, 1,015 disability applicants and 476 work-disabled subjects. METHODS Three-phase analytic approach including item mapping, a modified-Delphi technique, and known-groups validation analysis were used to develop and validate cut-points for functional levels within each of the WD-PF and WD-BH instruments scales. RESULTS Four and 5 functional levels were developed for each of the scales in the WD-PF and WD-BH instruments. Distribution of the comparative samples was in the expected direction: the general adult samples consistently demonstrated scores at higher functional levels compared with the claimant and work-disabled samples. CONCLUSION Using an item-response theory-based methodology paired with a qualitative process appears to be a feasible and valid approach for translating the WD-BH and WD-PF scores into meaningful levels useful for interpreting a persons work-related physical and behavioral health functioning.