Susan Ostmo

Expert Diagnosis of Plus Disease in Retinopathy of Prematurity From Computer-Based Image Analysis

IMPORTANCE Published definitions of plus disease in retinopathy of prematurity (ROP) reference arterial tortuosity and venous dilation within the posterior pole based on a standard published photograph. One possible explanation for limited interexpert reliability for a diagnosis of plus disease is that experts deviate from the published definitions. OBJECTIVE To identify vascular features used by experts for diagnosis of plus disease through quantitative image analysis. DESIGN, SETTING, AND PARTICIPANTS A computer-based image analysis system (Imaging and Informatics in ROP [i-ROP]) was developed using a set of 77 digital fundus images, and the system was designed to classify images compared with a reference standard diagnosis (RSD). System performance was analyzed as a function of the field of view (circular crops with a radius of 1-6 disc diameters) and vessel subtype (arteries only, veins only, or all vessels). Routine ROP screening was conducted from June 29, 2011, to October 14, 2014, in neonatal intensive care units at 8 academic institutions, with a subset of 73 images independently classified by 11 ROP experts for validation. The RSD was compared with the majority diagnosis of experts. MAIN OUTCOMES AND MEASURES The primary outcome measure was the percentage of accuracy of the i-ROP system classification of plus disease, with the RSD as a function of the field of view and vessel type. Secondary outcome measures included the accuracy of the 11 experts compared with the RSD. RESULTS Accuracy of plus disease diagnosis by the i-ROP computer-based system was highest (95%; 95% CI, 94%-95%) when it incorporated vascular tortuosity from both arteries and veins and with the widest field of view (6-disc diameter radius). Accuracy was 90% or less when using only arterial tortuosity and 85% or less using a 2- to 3-disc diameter view similar to the standard published photograph. Diagnostic accuracy of the i-ROP system (95%) was comparable to that of 11 expert physicians (mean 87%, range 79%-99%). CONCLUSIONS AND RELEVANCE Experts in ROP appear to consider findings from beyond the posterior retina when diagnosing plus disease and consider tortuosity of both arteries and veins, in contrast with published definitions. It is feasible for a computer-based image analysis system to perform comparably with ROP experts, using manually segmented images.

Diagnostic Discrepancies in Retinopathy of Prematurity Classification

PURPOSE To identify the most common areas for discrepancy in retinopathy of prematurity (ROP) classification between experts. DESIGN Prospective cohort study. PARTICIPANTS A total of 281 infants were identified as part of a multicenter, prospective, ROP cohort study from 7 participating centers. Each site had participating ophthalmologists who provided the clinical classification after routine examination using binocular indirect ophthalmoscopy (BIO) and obtained wide-angle retinal images, which were independently classified by 2 study experts. METHODS Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were obtained from study subjects, and 2 experts evaluated each image using a secure web-based module. Image-based classifications for zone, stage, plus disease, and overall disease category (no ROP, mild ROP, type II or pre-plus, and type I) were compared between the 2 experts and with the clinical classification obtained by BIO. MAIN OUTCOME MEASURES Inter-expert image-based agreement and image-based versus ophthalmoscopic diagnostic agreement using absolute agreement and weighted kappa statistic. RESULTS A total of 1553 study eye examinations from 281 infants were included in the study. Experts disagreed on the stage classification in 620 of 1553 comparisons (40%), plus disease classification (including pre-plus) in 287 of 1553 comparisons (18%), zone in 117 of 1553 comparisons (8%), and overall ROP category in 618 of 1553 comparisons (40%). However, agreement for presence versus absence of type 1 disease was >95%. There were no differences between image-based and clinical classification except for zone III disease. CONCLUSIONS The most common area of discrepancy in ROP classification is stage, although inter-expert agreement for clinically significant disease, such as presence versus absence of type 1 and type 2 disease, is high. There were no differences between image-based grading and clinical examination in the ability to detect clinically significant disease. This study provides additional evidence that image-based classification of ROP reliably detects clinically significant levels of ROP with high accuracy compared with the clinical examination.

Influence of Fluorescein Angiography on the Diagnosis and Management of Retinopathy of Prematurity

PURPOSE To examine the influence of fluorescein angiography (FA) on the diagnosis and management of retinopathy of prematurity (ROP). DESIGN Prospective cohort study. PARTICIPANTS Nine recognized ROP experts (3 pediatric ophthalmologists and 6 retina specialists) interpreted 32 sets (16 color fundus photographs and 16 color fundus photographs paired with the corresponding FA images) of wide-angle retinal images from infants with ROP. METHODS All experts independently reviewed the 32 image sets on a secure website and provided a diagnosis and management plan for the case presented, first based on color fundus photographs alone, and then based on color fundus photographs and corresponding FA images. MAIN OUTCOME MEASURES Sensitivity and specificity of the ROP diagnosis (zone, stage, plus disease, and category, i.e., no ROP, mild ROP, type 2 ROP, and ROP requiring treatment) were calculated using a consensus reference standard diagnosis, determined from the diagnosis of the color fundus photographs by 3 experienced readers in combination with the clinical diagnosis based on ophthalmoscopic examination. The κ statistic was used to analyze the average intergrader agreement among experts for the diagnosis of zone, stage, plus disease, and category. RESULTS Addition of FA to color fundus photography resulted in a significant improvement in sensitivity for diagnosis of stage 3 or worse disease (39.8% vs. 74.1%; P = 0.008), type 2 or worse ROP (69.4% vs. 86.8%; P = 0.013), and pre-plus or worse disease (50.5 vs. 62.6%; P = 0.031). There was a nonsignificant trend toward improved sensitivity for diagnosis of ROP requiring treatment (22.2% vs. 40.3%; P = 0.063). Using the κ statistic, addition of FA to color fundus photography significantly improved intergrader agreement for diagnosis of ROP requiring treatment. Addition of FA to color fundus photography did not affect intergrader agreement significantly for the diagnosis of stage, zone, or plus disease. CONCLUSIONS Compared with color fundus photography alone, FA may improve the sensitivity of diagnosis of ROP by experts, particularly for stage 3 disease. In addition, intergrader agreement for diagnosis of ROP requiring treatment may improve with FA interpretation.

Color Fundus Photography Versus Fluorescein Angiography in Identification of the Macular Center and Zone in Retinopathy of Prematurity

PURPOSE To examine the usefulness of fluorescein angiography (FA) in identifying the macular center and diagnosis of zone in patients with retinopathy of prematurity (ROP). DESIGN Validity and reliability analysis of diagnostic tools. METHODS Thirty-two sets (16 color fundus photographs and 16 color fundus photographs paired with the corresponding FA images) of wide-angle retinal images obtained from 16 eyes of 8 infants with ROP were compiled on a secure web site. Nine ROP experts (3 pediatric ophthalmologists and 6 vitreoretinal surgeons) participated in the study. For each image set, experts identified the macular center and provided a diagnosis of zone. MAIN OUTCOME MEASURES (1) Sensitivity and specificity of zone diagnosis and (2) computer-facilitated diagnosis of zone, based on precise measurement of the macular center, optic disc center, and peripheral ROP. RESULTS Computer-facilitated diagnosis of zone agreed with the experts diagnosis of zone in 28 (62%) of 45 cases using color fundus photographs and in 31 (69%) of 45 cases using FA images. Mean (95% confidence interval) sensitivity for detection of zone I by experts compared with a consensus reference standard diagnosis when interpreting the color fundus images alone versus interpreting the color fundus photographs and FA images was 47% (range, 35.3% to 59.3%) and 61.1% (range, 48.9% to 72.4%), respectively (t(9) ≥ (2.063); P = .073). CONCLUSIONS There is a marginally significant difference in zone diagnosis when using color fundus photographs compared with using color fundus photographs and the corresponding FA images. There is inconsistency between traditional zone diagnosis (based on ophthalmoscopic examination and image review) compared with a computer-facilitated diagnosis of zone.

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis.

PURPOSE To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. DESIGN We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. PARTICIPANTS Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. METHODS Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. MAIN OUTCOME MEASURES Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. RESULTS There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). CONCLUSIONS Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.

Practice Patterns in Retinopathy of Prematurity Treatment for Disease Milder Than Recommended by Guidelines

PURPOSE To characterize the frequency of and clinical indications for which experts treat retinopathy of prematurity (ROP) milder than type 1 disease, the recommended threshold for treatment from established consensus guidelines. DESIGN Descriptive analysis. METHODS setting: Multicenter. STUDY POPULATION A database of 1444 eyes generated prospectively from all babies screened for ROP at 1 of 6 major ROP centers whose parents provided informed consent. INTERVENTION Retrospective review of the database and charts to identify all patients treated for ROP milder than type 1. MAIN OUTCOME MEASURE Indication(s) for treatment. RESULTS A total of 137 eyes of 70 infants were treated for ROP. Of these 137 eyes, 13 (9.5%) were treated despite a clinical diagnosis milder than type 1 ROP. Indications for treatment included active ROP with the fellow eye being treated for type 1 ROP (2 eyes, 15.4%); concerning structural changes (9 eyes, 69.2%), including tangential traction with temporal vessel straightening concerning for macular dragging (8 eyes, 61.5%) and thick stage 3 membranes with anteroposterior traction concerning for progression to stage 4 ROP (3 eyes, 23.1%); persistent ROP at an advanced postmenstrual age (4 eyes, 30.8%); and/or vitreous hemorrhage (3 eyes, 23.1%). CONCLUSIONS Experts in this study occasionally recommended treatment in eyes with disease less than type 1 ROP. This study has important clinical implications and highlights the role of individual clinical judgment in situations not covered by evidence-based treatment guidelines.

Automated Diagnosis of Plus Disease in Retinopathy of Prematurity Using Deep Convolutional Neural Networks

Importance Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable. Objective To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs. Design, Setting, and Participants A deep convolutional neural network was trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre–plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017. Exposures A deep learning algorithm trained on retinal photographs. Main Outcomes and Measures Receiver operating characteristic analysis was performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted &kgr; coefficients were calculated for ternary classification (ie, normal, pre–plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts. Results Of the 5511 included retinal photographs, 4535 (82.3%) were graded as normal, 805 (14.6%) as pre–plus disease, and 172 (3.1%) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre–plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre–plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93% with 94% specificity. For detection of pre–plus disease or worse, the sensitivity and specificity were 100% and 94%, respectively. On the same test set, the algorithm achieved a quadratic-weighted &kgr; coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts. Conclusions and Relevance This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.

Diagnostic Accuracy of Ophthalmoscopy vs Telemedicine in Examinations for Retinopathy of Prematurity

Importance Examinations for retinopathy of prematurity (ROP) are typically performed using binocular indirect ophthalmoscopy. Telemedicine studies have traditionally assessed the accuracy of telemedicine compared with ophthalmoscopy as a criterion standard. However, it is not known whether ophthalmoscopy is truly more accurate than telemedicine. Objective To directly compare the accuracy and sensitivity of ophthalmoscopy vs telemedicine in diagnosing ROP using a consensus reference standard. Design, Setting, and Participants This multicenter prospective study conducted between July 1, 2011, and November 30, 2014, at 7 neonatal intensive care units and academic ophthalmology departments in the United States and Mexico included 281 premature infants who met the screening criteria for ROP. Exposures Each examination consisted of 1 eye undergoing binocular indirect ophthalmoscopy by an experienced clinician followed by remote image review of wide-angle fundus photographs by 3 independent telemedicine graders. Main Outcomes and Measures Results of both examination methods were combined into a consensus reference standard diagnosis. The agreement of both ophthalmoscopy and telemedicine was compared with this standard, using percentage agreement and weighted &kgr; statistics. Results Among the 281 infants in the study (127 girls and 154 boys; mean [SD] gestational age, 27.1 [2.4] weeks), a total of 1553 eye examinations were classified using both ophthalmoscopy and telemedicine. Ophthalmoscopy and telemedicine each had similar sensitivity for zone I disease (78% [95% CI, 71%-84%] vs 78% [95% CI, 73%-83%]; P > .99 [n = 165]), plus disease (74% [95% CI, 61%-87%] vs 79% [95% CI, 72%-86%]; P = .41 [n = 50]), and type 2 ROP (stage 3, zone I, or plus disease: 86% [95% CI, 80%-92%] vs 79% [95% CI, 75%-83%]; P = .10 [n = 251]), but ophthalmoscopy was slightly more sensitive in identifying stage 3 disease (85% [95% CI, 79%-91%] vs 73% [95% CI, 67%-78%]; P = .004 [n = 136]). Conclusions and Relevance No difference was found in overall accuracy between ophthalmoscopy and telemedicine for the detection of clinically significant ROP, although, on average, ophthalmoscopy had slightly higher accuracy for the diagnosis of zone III and stage 3 ROP. With the caveat that there was variable accuracy between examiners using both modalities, these results support the use of telemedicine for the diagnosis of clinically significant ROP.

Fully automated disease severity assessment and treatment monitoring in retinopathy of prematurity using deep learning

Retinopathy of prematurity (ROP) is a disease that affects premature infants, where abnormal growth of the retinal blood vessels can lead to blindness unless treated accordingly. Infants considered at risk of severe ROP are monitored for symptoms of plus disease, characterized by arterial tortuosity and venous dilation at the posterior pole, with a standard photographic definition. Disagreement among ROP experts in diagnosing plus disease has driven the development of computer-based methods that classify images based on hand-crafted features extracted from the vasculature. However, most of these approaches are semi-automated, which are time-consuming and subject to variability. In contrast, deep learning is a fully automated approach that has shown great promise in a wide variety of domains, including medical genetics, informatics and imaging. Convolutional neural networks (CNNs) are deep networks which learn rich representations of disease features that are highly robust to variations in acquisition and image quality. In this study, we utilized a U-Net architecture to perform vessel segmentation and then a GoogLeNet to perform disease classification. The classifier was trained on 3,000 retinal images and validated on an independent test set of patients with different observed progressions and treatments. We show that our fully automated algorithm can be used to monitor the progression of plus disease over multiple patient visits with results that are consistent with the experts’ consensus diagnosis. Future work will aim to further validate the method on larger cohorts of patients to assess its applicability within the clinic as a treatment monitoring tool.

Changes in relative position of choroidal versus retinal vessels in preterm infants

Purpose The purpose of this study was to characterize a novel finding that relative positions of choroidal and retinal vessels change over time in preterm infants and to identify factors associated with this finding using quantitative analysis. Methods Fundus images were obtained prospectively through a retinopathy of prematurity (ROP) cohort study. Images were excluded if choroidal vessels could not be identified. Changes in relative position of characteristic choroidal landmarks with respect to retinal vessels between two time points 5 to 7 weeks apart were measured. Univariate and multivariate regression analyses were performed to identify associated factors with the amount of change. Results The discovery and replication cohorts included 45 and 58 patients, respectively. Ninety-two of them (89%) were non-Hispanic Caucasians. Changes in relative position of choroidal versus retinal vessels were detected in all eyes of the discovery and replication cohorts (mean amount = 0.42 ± 0.12 and 0.35 ± 0.12 mm, respectively). On combined multiple regression analysis of the two cohorts, type 1 ROP, higher postmenstral age at the first time point, and shorter distance from optic disc to choroidal landmark were significantly associated with less change in relative position. Conclusions Choroidal vessels grow anteriorly with respect to retinal vessels at posterior pole in preterm infants, suggesting relatively faster peripheral growth of choroidal versus retinal vessels. Eyes with severe ROP showed less difference in growth, which might represent alterations in choroidal development due to advanced ROP. These findings may contribute to better understanding about the physiology of choroidal development and involvement in ROP.