Response to Rubanovich et al.

Abstract

Rubanovich et al. raise several concerns regarding the development of the Clinician-reported Genetic testing Utility InDEx (C-GUIDE). We appreciate the opportunity to deepen the discussion of measurement science as it applies to genomic medicine. The authors assert that we “did not align our approach with psychometric best practices” and that we “did not describe or refer to any theoretically sound methodology for scale development.” Although not stated explicitly in the paper, we thoroughly debated the assumptions and approaches associated with both the traditional psychometric and the more relevant and clinically oriented clinimetric approach to measurement. While the wellestablished toolkit provided by the psychometric approach is appealing, the purpose and target population for which C-GUIDE is intended precisely align with the clinimetric approach to measurement development. The aim of a psychometric approach is to create a scale in which multiple items measure a construct of interest that exists, but is not directly measurable. This is called a latent variable, the structure of which is determined by the statistical relationship among the items that reflect it. A large number of items are generated and statistics are used to establish a set of highly correlated items, representing a reflective model. In contrast, a clinimetric approach relies on notions of clinical relevance and clinical sensibility; clinimetric tools are not constrained to reflective items or statistical relationships among them. Rather, they include causal or formative items. In a formative model, the concept being measured does not exist; rather, it is constructed through a process of identifying items that form it. Clinical utility is a clinimetric construct that has been defined (i.e., formed) by experts in the field, beginning with the ACCE model which assesses the Analytic validity, Clinical validity, Clinical utility, and Ethical, legal and social implications of genetic tests. Outcomes related to diagnosis, management, health benefits, and family testing, for example, constitute this construct. We, as a community of practice, have formed this construct; the C-GUIDE aims to provide a clinically sensible way of quantifying it. In essence, psychometric and clinimetric approaches represent different theoretical perspectives, aims, and measurement development pathways. Often, different measurement tools emerge. For example, Juniper developed an asthma-related quality of life tool and contrasted psychometric methods using factor analysis with a clinimetric approach using patients’ opinions of importance. Psychometric analysis resulted in a 36-item instrument compared with 32 items by the “importance” method. Only 20 items were common to both. The “importance” method was chosen for the final instrument as it showed better correspondence to clinical sensibility. The psychometric method would have excluded three items of greatest importance to the stakeholders and other important items relating to functional impairment. We concur that a nomological network is appropriate for establishing the structure of a psychometric measure; the same requirement, however, does not apply to clinimetrics. The nomological net assumes the construct “exists” and the structure, as derived by the network, establishes the laws for what “it” is. A formative construct does not truly exist; we formulate it to meet a test of clinical sensibility. Further, Rubanovich et al. suggest that we should have generated many more items in the early stages of our work than we reported. In fact, the 25 preliminary items were culled from a longer list. Each item was worded several different ways and iteratively modified, where applicable, to suit the context of secondary and pharmacogenomics variants. As such, our preliminary list in all of its permutations far exceeded 25. Since the goal of a formative model, however, is to identify distinct aspects that form the construct, an extensive list of interchangeable items is not required in the way it is for a reflective measure. The authors also suggest that we “changed the model of our construct in response to feedback on specific items.” The “structure” of C-GUIDE was not derived de novo, but was informed by the Fryback and Thornbury model, relevant literature, and qualitative interviews. While the items constitute the construct, they do not form formally structured domains. The consolidation of the family and psychosocial impact “domains” did not relate to poorly written items; rather it related to respondents’ sentiment that familial implications could influence and be influenced by psychosocial impact. As such, these domains were not distinct. In the end, while C-GUIDE attends to diagnostic, management, and familial/psychosocial aspects of clinical utility, these domains do not represent the formal internal structure of a construct in a psychometric sense. They represent literature-derived, clinically sensible components of clinical utility, finer-grained aspects of which are represented by the index items. The decision to implement a total sum rather than domain scores was precisely because some domains—as Rubanovich et al. point out—would be disproportionately weighted over others, with no evidence to justify this. With respect to establishing item weights, factor analysis is not an appropriate tool because correlation among items is not expected. As such, item weights are typically decided by respondents and researchers, and rely on clinical sensibility. We presented our expert panel with three different weighting strategies. The