A. James O’Malley

Receiver-Operating Characteristic Analysis for Evaluating Diagnostic Tests and Predictive Models

Receiver-operating characteristic (ROC) analysis was originally developed during World War II to analyze classification accuracy in differentiating signal from noise in radar detection.1 Recently, the methodology has been adapted to several clinical areas heavily dependent on screening and diagnostic tests,2–4 in particular, laboratory testing,5 epidemiology,6 radiology,7–9 and bioinformatics.10 ROC analysis is a useful tool for evaluating the performance of diagnostic tests and more generally for evaluating the accuracy of a statistical model (eg, logistic regression, linear discriminant analysis) that classifies subjects into 1 of 2 categories, diseased or nondiseased. Its function as a simple graphical tool for displaying the accuracy of a medical diagnostic test is one of the most well-known applications of ROC curve analysis. In Circulation from January 1, 1995, through December 5, 2005, 309 articles were published with the key phrase “receiver operating characteristic.” In cardiology, diagnostic testing plays a fundamental role in clinical practice (eg, serum markers of myocardial necrosis, cardiac imaging tests). Predictive modeling to estimate expected outcomes such as mortality or adverse cardiac events based on patient risk characteristics also is common in cardiovascular research. ROC analysis is a useful tool in both of these situations. In this article, we begin by reviewing the measures of accuracy—sensitivity, specificity, and area under the curve (AUC)—that use the ROC curve. We also illustrate how these measures can be applied using the evaluation of a hypothetical new diagnostic test as an example. A diagnostic classification test typically yields binary, ordinal, or continuous outcomes. The simplest type, binary outcomes, arises from a screening test indicating whether the patient is nondiseased (Dx=0) or diseased (Dx=1). The screening test indicates whether the patient is likely to be diseased or not. When >2 categories are used, the test data can be on an ordinal rating …

Proximity to Food Establishments and Body Mass Index in the Framingham Heart Study Offspring Cohort Over 30 Years

Existing evidence linking residential proximity to food establishments with body mass index (BMI; weight (kg)/height (m)(2)) has been inconclusive. In this study, the authors assessed the relation between BMI and proximity to food establishments over a 30-year period among 3,113 subjects in the Framingham Heart Study Offspring Cohort living in 4 Massachusetts towns during 1971-2001. The authors used novel data that included repeated measures of BMI and accounted for residential mobility and the appearance and disappearance of food establishments. They calculated proximity to food establishments as the driving distance between each subjects residence and nearby food establishments, divided into 6 categories. The authors used cross-classified linear mixed models to account for time-varying attributes of individuals and residential neighborhoods. Each 1-km increase in distance to the closest fast-food restaurant was associated with a 0.11-unit decrease in BMI (95% credible interval: -0.20, -0.04). In sex-stratified analyses, this association was present only for women. Other aspects of the food environment were either inconsistently associated or not at all associated with BMI. Contrary to much prior research, the authors did not find a consistent relation between access to fast-food restaurants and individual BMI, necessitating a reevaluation of policy discussions on the anticipated impact of the food environment on weight gain.

Saphenous Vein Graft Stenting and Major Adverse Cardiac Events A Predictive Model Derived From a Pooled Analysis of 3958 Patients

Background— Treatment of saphenous vein graft (SVG) stenosis with percutaneous coronary intervention has a 15% to 20% incidence of major adverse cardiac events (MACE) within 30 days. Although MACE rates are reduced significantly by the use of embolic protection devices (EPDs), neither the level of baseline risk nor the benefit provided by EPDs has been well characterized. Methods and Results— Data from 5 randomized controlled trials and 1 registry evaluating EPDs in SVG percutaneous coronary intervention (n=3958 patients) were pooled for analysis. MACE was defined as a composite of death, myocardial infarction, and target vessel revascularization. Baseline variables and 2 summary angiographic variables (an SVG degeneration score and an estimate of lesion plaque volume) were included in a multivariable logistic regression model to predict 30-day MACE, with adjustment for the type of device used and inter-study variation. The angiographic variables were potent predictors of MACE (increasing SVG degeneration score, P<0.0001; larger estimated plaque volume, P<0.0001), with significant contributions from the presence of thrombus (P<0.01), increasing patient age (P<0.01), glycoprotein IIb/IIIa inhibitor use (P=0.02), and current tobacco abuse (P=0.03). The treatment benefit of EPDs was preserved across all categories of risk as categorized by SVG degeneration or plaque volume. Conclusions— The strongest predictors of 30-day MACE in SVG percutaneous coronary intervention are angiographic estimates of plaque volume and SVG degeneration. Identification of these predictors of 30-day MACE allows reliable prediction of patient outcomes and confirms consistent treatment benefit with the use of EPDs across the range of patients tested in randomized trials.

Relationship of Late Loss in Lumen Diameter to Coronary Restenosis in Sirolimus-Eluting Stents

Background—Observed rates of restenosis after drug-eluting stenting are low (<10%). Identification of a reliable and powerful angiographic end point will be useful in future trials. Methods and Results—Late loss (postprocedural minimum lumen diameter minus 8-month minimum lumen diameter) was measured in the angiographic cohorts of the SIRIUS (n=703) and E-SIRIUS (n=308) trials. Two techniques, the standard normal approximation and an optimized power transformation, were used to predict binary angiographic restenosis rates and compare them with observed restenosis rates. The mean in-stent late loss observed in the SIRIUS trial was 0.17±0.45 mm (sirolimus) versus 1.00±0.70 mm (control). If a normal distribution was assumed, late loss accurately estimated in-stent binary angiographic restenosis for the control arm (predicted 35.4% versus observed 35.4%) but underestimated it in the sirolimus arm (predicted 0.6% versus observed 3.2%). Power transformation improved the reliability of the estimate in the sirolimus arm (predicted 3.2% [CI 1.0% to 6.7%]) with similar improvements in the E-SIRIUS trial (predicted 4.0% [CI 1.2% to 7.0%] versus observed 3.9%). In the sirolimus-eluting stent arm, in-stent late loss correlated better with target-lesion revascularization than in-segment late loss (c-statistic=0.915 versus 0.665). Conclusions—Because distributions of late loss with a low mean are right-skewed, the use of a transformation improves the accuracy of predicting low binary restenosis rates. Late loss is monotonically correlated with the probability of restenosis and yields a more efficient estimate of the restenosis process in the era of lower binary restenosis rates.

Long-Term Outcomes of Abdominal Aortic Aneurysm in the Medicare Population

BACKGROUND Randomized trials and observational studies have shown that perioperative morbidity and mortality are lower with endovascular repair of abdominal aortic aneurysm than with open repair, but the survival benefit is not sustained. In addition, concerns have been raised about the long-term risk of aneurysm rupture or the need for reintervention after endovascular repair. METHODS We assessed perioperative and long-term survival, reinterventions, and complications after endovascular repair as compared with open repair of abdominal aortic aneurysm in propensity-score-matched cohorts of Medicare beneficiaries who underwent repair during the period from 2001 through 2008 and were followed through 2009. RESULTS We identified 39,966 matched pairs of patients who had undergone either open repair or endovascular repair. The overall perioperative mortality was 1.6% with endovascular repair versus 5.2% with open repair (P<0.001). From 2001 through 2008, perioperative mortality decreased by 0.8 percentage points among patients who underwent endovascular repair (P=0.001) and by 0.6 percentage points among patients who underwent open repair (P=0.01). The rate of conversion from endovascular to open repair decreased from 2.2% in 2001 to 0.3% in 2008 (P<0.001). The rate of survival was significantly higher after endovascular repair than after open repair through the first 3 years of follow-up, after which time the rates of survival were similar. Through 8 years of follow-up, interventions related to the management of the aneurysm or its complications were more common after endovascular repair, whereas interventions for complications related to laparotomy were more common after open repair. Aneurysm rupture occurred in 5.4% of patients after endovascular repair versus 1.4% of patients after open repair through 8 years of follow-up (P<0.001). The rate of total reinterventions at 2 years after endovascular repair decreased over time (from 10.4% among patients who underwent procedures in 2001 to 9.1% among patients who underwent procedures in 2007). CONCLUSIONS Endovascular repair, as compared with open repair, of abdominal aortic aneurysm was associated with a substantial early survival advantage that gradually decreased over time. The rate of late rupture was significantly higher after endovascular repair than after open repair. The outcomes of endovascular repair have been improving over time. (Funded by the National Institutes of Health.).

Variation in Patient-Sharing Networks of Physicians Across the United States

CONTEXT Physicians are embedded in informal networks that result from their sharing of patients, information, and behaviors. OBJECTIVES To identify professional networks among physicians, examine how such networks vary across geographic regions, and determine factors associated with physician connections. DESIGN, SETTING, AND PARTICIPANTS Using methods adopted from social network analysis, Medicare administrative data from 2006 were used to study 4,586,044 Medicare beneficiaries seen by 68,288 physicians practicing in 51 hospital referral regions (HRRs). Distinct networks depicting connections between physicians (defined based on shared patients) were constructed for each of the 51 HRRs. MAIN OUTCOMES MEASURES Variation in network characteristics across HRRs and factors associated with physicians being connected. RESULTS The number of physicians per HRR ranged from 135 in Minot, North Dakota, to 8197 in Boston, Massachusetts. There was substantial variation in network characteristics across HRRs. For example, the mean (SD) adjusted degree (number of other physicians each physician was connected to per 100 Medicare beneficiaries) across all HRRs was 27.3 (range, 11.7-54.4); also, primary care physician relative centrality (how central primary care physicians were in the network relative to other physicians) ranged from 0.19 to 1.06, suggesting that primary care physicians were more than 5 times more central in some markets than in others. Physicians with ties to each other were far more likely to be based at the same hospital (69.2% of unconnected physician pairs vs 96.0% of connected physician pairs; adjusted rate ratio, 0.12 [95% CI, 0.12-0.12]; P < .001), and were in closer geographic proximity (mean office distance of 21.1 km for those with connections vs 38.7 km for those without connections, P < .001). Connected physicians also had more similar patient panels in terms of the race or illness burden than unconnected physicians. For instance, connected physician pairs had an average difference of 8.8 points in the percentage of black patients in their 2 patient panels compared with a difference of 14.0 percentage points for unconnected physician pairs (P < .001). CONCLUSIONS Network characteristics vary across geographic areas. Physicians tend to share patients with other physicians with similar physician-level and patient-panel characteristics.

The analysis of social networks

Many questions about the social organization of medicine and health services involve interdependencies among social actors that may be depicted by networks of relationships. Social network studies have been pursued for some time in social science disciplines, where numerous descriptive methods for analyzing them have been proposed. More recently, interest in the analysis of social network data has grown among statisticians, who have developed more elaborate models and methods for fitting them to network data. This article reviews fundamentals of, and recent innovations in, social network analysis using a physician influence network as an example. After introducing forms of network data, basic network statistics, and common descriptive measures, it describes two distinct types of statistical models for network data: individual-outcome models in which networks enter the construction of explanatory variables, and relational models in which the network itself is a multivariate dependent variable. Complexities in estimating both types of models arise due to the complex correlation structures among outcome measures.

Physician Patient-Sharing Networks and the Cost and Intensity of Care in US Hospitals

Background:There is substantial variation in the cost and intensity of care delivered by US hospitals. We assessed how the structure of patient-sharing networks of physicians affiliated with hospitals might contribute to this variation. Methods:We constructed hospital-based professional networks based on patient-sharing ties among 61,461 physicians affiliated with 528 hospitals in 51 hospital referral regions in the US using Medicare data on clinical encounters during 2006. We estimated linear regression models to assess the relationship between measures of hospital network structure and hospital measures of spending and care intensity in the last 2 years of life. Results:The typical physician in an average-sized urban hospital was connected to 187 other doctors for every 100 Medicare patients shared with other doctors. For the average-sized urban hospital an increase of 1 standard deviation (SD) in the median number of connections per physician was associated with a 17.8% increase in total spending, in addition to 17.4% more hospital days, and 23.8% more physician visits (all P<0.001). In addition, higher “centrality” of primary care providers within these hospital networks was associated with 14.7% fewer medical specialist visits (P<0.001) and lower spending on imaging and tests (−9.2% and −12.9% for 1 SD increase in centrality, P<0.001). Conclusions:Hospital-based physician network structure has a significant relationship with an institution’s care patterns for their patients. Hospitals with doctors who have higher numbers of connections have higher costs and more intensive care, and hospitals with primary care-centered networks have lower costs and care intensity.

Use Of Telemedicine Can Reduce Hospitalizations Of Nursing Home Residents And Generate Savings For Medicare

Hospitalizations of nursing home residents are frequent and result in complications, morbidity, and Medicare expenditures of more than a billion dollars annually. The lack of a physician presence at many nursing homes during off hours might contribute to inappropriate hospitalizations. Findings from our controlled study of eleven nursing homes provide the first indications that switching from on-call to telemedicine physician coverage during off hours could reduce hospitalizations and therefore generate cost savings to Medicare in excess of the facilitys investment in the service. But those savings were evident only at the study nursing homes that used the telemedicine service to a greater extent, compared to the other study facilities. Telemedicine service providers and nursing home leaders might need to take additional steps to encourage buy-in to the use of telemedicine at facilities with such services. At the same time, closer alignment of the stakeholders that bear the costs of telemedicine and those that might realize savings because of its use could offer further incentives for the adoption of telemedicine.

Egocentric Social Network Structure, Health, and Pro- Social Behaviors in a National Panel Study of Americans

Using a population-based, panel survey, we study how egocentric social networks change over time, and the relationship between egocentric network properties and health and pro-social behaviors. We find that the number of prosocial activities is strongly positively associated with having more friends, or an increase in degree, with approximately 0.04 more prosocial behaviors expected for every friend added. Moreover, having more friends is associated with an improvement in health, while being healthy and prosocial is associated with closer relationships. Specifically, a unit increase in health is associated with an expected 0.45 percentage-point increase in average closeness, while adding a prosocial activity is associated with a 0.46 percentage-point increase in the closeness of one’s relationships. Furthermore, a tradeoff between degree and closeness of social contacts was observed. As the number of close social contacts increases by one, the estimated average closeness of each individual contact decreases by approximately three percentage-points. The increased awareness of the importance of spillover effects in health and health care makes the ascertainment of egocentric social networks a valuable complement to investigations of the relationship between socioeconomic factors and health.