Karyn Jonas

Expert Diagnosis of Plus Disease in Retinopathy of Prematurity From Computer-Based Image Analysis

IMPORTANCE Published definitions of plus disease in retinopathy of prematurity (ROP) reference arterial tortuosity and venous dilation within the posterior pole based on a standard published photograph. One possible explanation for limited interexpert reliability for a diagnosis of plus disease is that experts deviate from the published definitions. OBJECTIVE To identify vascular features used by experts for diagnosis of plus disease through quantitative image analysis. DESIGN, SETTING, AND PARTICIPANTS A computer-based image analysis system (Imaging and Informatics in ROP [i-ROP]) was developed using a set of 77 digital fundus images, and the system was designed to classify images compared with a reference standard diagnosis (RSD). System performance was analyzed as a function of the field of view (circular crops with a radius of 1-6 disc diameters) and vessel subtype (arteries only, veins only, or all vessels). Routine ROP screening was conducted from June 29, 2011, to October 14, 2014, in neonatal intensive care units at 8 academic institutions, with a subset of 73 images independently classified by 11 ROP experts for validation. The RSD was compared with the majority diagnosis of experts. MAIN OUTCOMES AND MEASURES The primary outcome measure was the percentage of accuracy of the i-ROP system classification of plus disease, with the RSD as a function of the field of view and vessel type. Secondary outcome measures included the accuracy of the 11 experts compared with the RSD. RESULTS Accuracy of plus disease diagnosis by the i-ROP computer-based system was highest (95%; 95% CI, 94%-95%) when it incorporated vascular tortuosity from both arteries and veins and with the widest field of view (6-disc diameter radius). Accuracy was 90% or less when using only arterial tortuosity and 85% or less using a 2- to 3-disc diameter view similar to the standard published photograph. Diagnostic accuracy of the i-ROP system (95%) was comparable to that of 11 expert physicians (mean 87%, range 79%-99%). CONCLUSIONS AND RELEVANCE Experts in ROP appear to consider findings from beyond the posterior retina when diagnosing plus disease and consider tortuosity of both arteries and veins, in contrast with published definitions. It is feasible for a computer-based image analysis system to perform comparably with ROP experts, using manually segmented images.

Diagnostic Discrepancies in Retinopathy of Prematurity Classification

PURPOSE To identify the most common areas for discrepancy in retinopathy of prematurity (ROP) classification between experts. DESIGN Prospective cohort study. PARTICIPANTS A total of 281 infants were identified as part of a multicenter, prospective, ROP cohort study from 7 participating centers. Each site had participating ophthalmologists who provided the clinical classification after routine examination using binocular indirect ophthalmoscopy (BIO) and obtained wide-angle retinal images, which were independently classified by 2 study experts. METHODS Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were obtained from study subjects, and 2 experts evaluated each image using a secure web-based module. Image-based classifications for zone, stage, plus disease, and overall disease category (no ROP, mild ROP, type II or pre-plus, and type I) were compared between the 2 experts and with the clinical classification obtained by BIO. MAIN OUTCOME MEASURES Inter-expert image-based agreement and image-based versus ophthalmoscopic diagnostic agreement using absolute agreement and weighted kappa statistic. RESULTS A total of 1553 study eye examinations from 281 infants were included in the study. Experts disagreed on the stage classification in 620 of 1553 comparisons (40%), plus disease classification (including pre-plus) in 287 of 1553 comparisons (18%), zone in 117 of 1553 comparisons (8%), and overall ROP category in 618 of 1553 comparisons (40%). However, agreement for presence versus absence of type 1 disease was >95%. There were no differences between image-based and clinical classification except for zone III disease. CONCLUSIONS The most common area of discrepancy in ROP classification is stage, although inter-expert agreement for clinically significant disease, such as presence versus absence of type 1 and type 2 disease, is high. There were no differences between image-based grading and clinical examination in the ability to detect clinically significant disease. This study provides additional evidence that image-based classification of ROP reliably detects clinically significant levels of ROP with high accuracy compared with the clinical examination.

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis.

PURPOSE To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. DESIGN We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. PARTICIPANTS Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. METHODS Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. MAIN OUTCOME MEASURES Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. RESULTS There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). CONCLUSIONS Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.

Diagnostic Accuracy of Ophthalmoscopy vs Telemedicine in Examinations for Retinopathy of Prematurity

Importance Examinations for retinopathy of prematurity (ROP) are typically performed using binocular indirect ophthalmoscopy. Telemedicine studies have traditionally assessed the accuracy of telemedicine compared with ophthalmoscopy as a criterion standard. However, it is not known whether ophthalmoscopy is truly more accurate than telemedicine. Objective To directly compare the accuracy and sensitivity of ophthalmoscopy vs telemedicine in diagnosing ROP using a consensus reference standard. Design, Setting, and Participants This multicenter prospective study conducted between July 1, 2011, and November 30, 2014, at 7 neonatal intensive care units and academic ophthalmology departments in the United States and Mexico included 281 premature infants who met the screening criteria for ROP. Exposures Each examination consisted of 1 eye undergoing binocular indirect ophthalmoscopy by an experienced clinician followed by remote image review of wide-angle fundus photographs by 3 independent telemedicine graders. Main Outcomes and Measures Results of both examination methods were combined into a consensus reference standard diagnosis. The agreement of both ophthalmoscopy and telemedicine was compared with this standard, using percentage agreement and weighted &kgr; statistics. Results Among the 281 infants in the study (127 girls and 154 boys; mean [SD] gestational age, 27.1 [2.4] weeks), a total of 1553 eye examinations were classified using both ophthalmoscopy and telemedicine. Ophthalmoscopy and telemedicine each had similar sensitivity for zone I disease (78% [95% CI, 71%-84%] vs 78% [95% CI, 73%-83%]; P > .99 [n = 165]), plus disease (74% [95% CI, 61%-87%] vs 79% [95% CI, 72%-86%]; P = .41 [n = 50]), and type 2 ROP (stage 3, zone I, or plus disease: 86% [95% CI, 80%-92%] vs 79% [95% CI, 75%-83%]; P = .10 [n = 251]), but ophthalmoscopy was slightly more sensitive in identifying stage 3 disease (85% [95% CI, 79%-91%] vs 73% [95% CI, 67%-78%]; P = .004 [n = 136]). Conclusions and Relevance No difference was found in overall accuracy between ophthalmoscopy and telemedicine for the detection of clinically significant ROP, although, on average, ophthalmoscopy had slightly higher accuracy for the diagnosis of zone III and stage 3 ROP. With the caveat that there was variable accuracy between examiners using both modalities, these results support the use of telemedicine for the diagnosis of clinically significant ROP.

Changes in relative position of choroidal versus retinal vessels in preterm infants

Purpose The purpose of this study was to characterize a novel finding that relative positions of choroidal and retinal vessels change over time in preterm infants and to identify factors associated with this finding using quantitative analysis. Methods Fundus images were obtained prospectively through a retinopathy of prematurity (ROP) cohort study. Images were excluded if choroidal vessels could not be identified. Changes in relative position of characteristic choroidal landmarks with respect to retinal vessels between two time points 5 to 7 weeks apart were measured. Univariate and multivariate regression analyses were performed to identify associated factors with the amount of change. Results The discovery and replication cohorts included 45 and 58 patients, respectively. Ninety-two of them (89%) were non-Hispanic Caucasians. Changes in relative position of choroidal versus retinal vessels were detected in all eyes of the discovery and replication cohorts (mean amount = 0.42 ± 0.12 and 0.35 ± 0.12 mm, respectively). On combined multiple regression analysis of the two cohorts, type 1 ROP, higher postmenstral age at the first time point, and shorter distance from optic disc to choroidal landmark were significantly associated with less change in relative position. Conclusions Choroidal vessels grow anteriorly with respect to retinal vessels at posterior pole in preterm infants, suggesting relatively faster peripheral growth of choroidal versus retinal vessels. Eyes with severe ROP showed less difference in growth, which might represent alterations in choroidal development due to advanced ROP. These findings may contribute to better understanding about the physiology of choroidal development and involvement in ROP.

Accuracy and Reliability of Eye-Based vs Quadrant-Based Diagnosis of Plus Disease in Retinopathy of Prematurity

Importance Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. Objective To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. Design, Setting, and Participants In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. Interventions Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. Main Outcomes and Measures Intragrader and intergrader reliability (absolute agreement and &kgr; statistic) and accuracy compared with the reference standard diagnosis. Results Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen &kgr; range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss &kgr;, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. Conclusions and Relevance Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

Inconsistencies in the diagnosis of aggressive posterior retinopathy of prematurity

Purpose: To determine the accuracy and reliability of diagnosing aggressive posterior retinopathy of prematurity (AP-ROP). Methods: A total of 1220 eye examinations from 230 infants were prospectively obtained at 8 major ROP centers. An ophthalmologist at each center provided a clinical diagnosis using indirect ophthalmoscopy. Wide-angle retinal images were then obtained, which were independently read by 2 ROP experts using a web-based system for an image-based diagnosis. Sensitivity and specificity of image-based AP-ROP diagnosis by the ROP experts were calculated using the clinical diagnosis as the reference standard. Agreement of AP-ROP diagnosis through image-based diagnosis and clinical diagnosis was calculated using the unweighted κ statistic. Results: One hundred four (9%) of the 1220 examinations had a clinical diagnosis of AP-ROP. Sensitivity and specificity for the presence of AP-ROP were 35% and 96% for expert 1 and 17% and 99% for expert 2. Using the κ statistic, expert image-based versus clinical diagnostic agreement for the diagnosis of AP-ROP was 0.34 (fair) for expert 1 and 0.24 (fair) for expert 2. Agreement for the diagnosis of AP-ROP between the image-based diagnoses of expert 1 and expert 2 was 0.49 (moderate). Conclusion: There are inconsistencies between the clinical diagnosis of AP-ROP (as determined by indirect ophthalmoscopy) and the image-based diagnosis of AP-ROP. This may have important implications for ROP management and the current international ROP classification system.

Toward a severity index for ROP: An unsupervised approach

Retinopathy of prematurity (ROP) is a disease affecting low birth-weight infants and is the major cause of childhood blindness. Although accurate diagnosis is important, there is a high variability among expert decisions mostly due to subjective thresholds. Existing work focused on automated diagnosis of ROP. In this study, we construct a continuous severity index as an alternative to discrete classification. We follow an unsupervised approach by performing nonlinear dimensionality reduction. Instead of extracting several statistics of image features, each image is represented by the probability distribution of its features. The distance between distributions are then used in manifold learning methods as the distance between samples. The experiments are carried out on a dataset of 104 wide-angle retinal images. The results are promising and they reflect the challenges of the discrete classification.

Wide-Field Imaging of the Pediatric Retina

Traditionally, children posed a unique challenge to imaging in the clinical setting because significant compliance from the patient was required, especially if the pathology was peripheral and required direction of the patient’s gaze into the necessary positions. As such, wide-field imaging of children was typically performed under anesthesia using handheld photography tools such as the RetCam (Clarity Medical Systems, Pleasanton, CA), which provides approximately 130° of visualization. The advent of ultra-wide-field (UWF) imaging technology, for example, the Optos (Optos, Marlborough, MA), enabled capturing a wider retinal area (approximately 200°) with potentially less requisite cooperation from the young patient. In recent retrospective case series, ultra-wide-field outpatient imaging has shown to exhibit utility in the diagnosis of a variety of retinal conditions, allowing wide-field imaging and fluorescein angiography (FA) in the clinic, without the use of anesthesia [1, 2]. Retrospective case series of pediatric patients undergoing ultra-wide-field angiography (UWFA) using Optos in the clinic found that imaging was useful for the evaluation of conditions including uveitis (pars planitis), Coats’ disease, retinopathy of prematurity, schisis, Stargardt’s disease, Best disease, history of intraocular foreign body, toxoplasmosis, melanoma, and familial exudative vitreoretinopathy in children as young as 5 years old [1, 2]. It was noted in one series of 16 patients under age 12 that imaging of earlier phases (i.e. the choroidal phase) was limited, likely due to the time required to console the patient after intravenous injection and to reposition the head [1]. UWFA may therefore be limited in cases where visualization of early phases is critical. Conditions in which later phase imaging is sufficient may therefore be particularly amenable to ultra-wide-field angiography in the office, allowing the patient to avoid a trip to the operating room for examination under anesthesia. This chapter presents wide-field and ultra-wide-field fundus photographs and angiograms from a spectrum of pediatric retinal conditions.