Nicholas P. Gruszauskas

Breast US Computer-aided Diagnosis System: Robustness across Urban Populations in South Korea and the United States

PURPOSE To evaluate the robustness of a breast ultrasonographic (US) computer-aided diagnosis (CAD) system in terms of its performance across different patient populations. MATERIALS AND METHODS Three US databases were analyzed for this study: one South Korean and two United States databases. All three databases were utilized in an institutional review board-approved and HIPAA-compliant manner. Round-robin analysis and independent testing were performed to evaluate the performance of a computerized breast cancer classification scheme across the databases. Receiver operating characteristic (ROC) analysis was used to evaluate performance differences. RESULTS The round-robin analyses of each database demonstrated similar results, with areas under the ROC curve ranging from 0.88 (95% confidence interval [CI]: 0.820, 0.918) to 0.91 (95% CI: 0.86, 0.95). The independent testing of each database, however, indicated that although the performances were similar, the range in areas under the ROC curve (from 0.79 [95% CI: 0.730, 0.842] to 0.87 [95% CI: 0.794, 0.923]) was wider than that with the round-robin tests. However, the only instances in which statistically significant differences in performance were demonstrated occurred when the Korean database was used in a testing capacity in independent testing. CONCLUSION The few observed statistically significant differences in performance indicated that while the US features used by the system were useful across the databases, their relative importance differed. In practice, this means that a CAD system may need to be adjusted when applied to a different population.

Performance of breast ultrasound computer-aided diagnosis: dependence on image selection.

RATIONALE AND OBJECTIVES The automated classification of sonographic breast lesions is generally accomplished by extracting and quantifying various features from the lesions. The selection of images to be analyzed, however, is usually left to the radiologist. Here we present an analysis of the effect that image selection can have on the performance of a breast ultrasound computer-aided diagnosis system. MATERIALS AND METHODS A database of 344 different sonographic lesions was analyzed for this study (219 cysts/benign processes, 125 malignant lesions). The database was collected in an institutional review board-approved, Health Insurance Portability and Accountability Act-compliant manner. Three different image selection protocols were used in the automated classification of each lesion: all images, first image only, and randomly selected images. After image selection, two different protocols were used to classify the lesions: (a) the average feature values were input to the classifier or (b) the classifier outputs were averaged together. Both protocols generated an estimated probability of malignancy. Round-robin analysis was performed using a Bayesian neural network-based classifier. Receiver-operating characteristic analysis was used to evaluate the performance of each protocol. Significance testing of the performance differences was performed via 95% confidence intervals and noninferiority tests. RESULTS The differences in the area under the receiver-operating characteristic curves were never more than 0.02 for the primary protocols. Noninferiority was demonstrated between these protocols with respect to standard input techniques (all images selected and feature averaging). CONCLUSION We have proved that our automated lesion classification scheme is robust and can perform well when subjected to variations in user input.

Research Imaging in an Academic Medical Center

RATIONALE AND OBJECTIVES Managing and supervising the complex imaging examinations performed for clinical research in an academic medical center can be a daunting task. Coordinating with both radiology and research staff to ensure that the necessary imaging is performed, analyzed, and delivered in accordance with the research protocol is nontrivial. The purpose of this communication is to report on the establishment of a new Human Imaging Research Office (HIRO) at our institution that provides a dedicated infrastructure to assist with these issues and improve collaborations between radiology and research staff. MATERIALS AND METHODS The HIRO was created with three primary responsibilities: 1) coordinate the acquisition of images for clinical research per the study protocol, 2) facilitate reliable and consistent assessment of disease response for clinical research, and 3) manage and distribute clinical research images in a compliant manner. RESULTS The HIRO currently provides assistance for 191 clinical research studies from 14 sections and departments within our medical center and performs quality assessment of image-based measurements for six clinical research studies. The HIRO has fulfilled 1806 requests for medical images, delivering 81,712 imaging examinations (more than 44.1 million images) and related reports to investigators for research purposes. CONCLUSIONS The ultimate goal of the HIRO is to increase the level of satisfaction and interaction among investigators, research subjects, radiologists, and other imaging professionals. Clinical research studies that use the HIRO benefit from a more efficient and accurate imaging process. The HIRO model could be adopted by other academic medical centers to support their clinical research activities; the details of implementation may differ among institutions, but the need to support imaging in clinical research through a dedicated, centralized initiative should apply to most academic medical centers.

SU-E-I-45: The Human Imaging Research Office (HIRO): Advancing the Role of Imaging in Clinical Research

Purpose: The clinical imaging infrastructure has been designed for the consistent execution of standard‐of‐care imaging examinations. Many imaging protocols for clinical research, however, have unique requirements that may burden the established infrastructure. Our purpose was to develop a solution to the many imaging demands and challenges presented by clinical research.Methods: We have created the Human Imaging Research Office (HIRO) at our institution. The HIRO is responsible for coordinating the acquisition, collection, analysis, and maintenance of images and related data associated with clinical research studies involving human subjects. The purpose of the HIRO is to coordinate, harmonize, and streamline the imaging aspects of clinical research, so that these non‐standard examinations can be performed as required by the study protocol. In effect, the HIRO is becoming the institutional resource for a broad spectrum of research imaging needs. The activities and services of the HIRO include site validation and image quality verification, development of standardized research‐specific requisition and reporting forms, scheduling, implementation of research billing codes, study‐specific image‐basedmeasurements, collection of customized imagedatabases, and distribution of anonymized images to researchers and study sponsors.Results: The HIRO has helped facilitate the acquisition of images for clinical research for 117 active, pending, or recently closed clinical research studies. These clinical research studies are being conducted by 60 unique principal investigators across 14 different sections and departments within the Medical Center. The HIRO has filled over 650 requests for de‐identified images, accounting for over 42.3 million images Conclusions: Imaging for clinical research often has requirements that differ substantially from those of clinical imaging. The complex infrastructure for standard‐of‐care imaging typically is stressed by the unique technical and administrative needs of a research protocol. The HIRO is able to respond effectively to the dynamic imaging needs of clinical research. SGA, HM, and MLG receive royalties and licensing fees through the University of Chicago related to computer‐aided diagnosis. HM is a consultant for Riverain Medical.

WE‐C‐332‐05: Preliminary Robustness Study of a Breast Ultrasound Computer‐Aided Diagnosis System Across Different Patient Populations

Purpose: To determine the robustness of a breast ultrasound computer‐aided diagnosis system (CADx) when it is used across different patient populations. Method and Materials: A sonographic database consisting of 433 lesions (127 malignant, 306 benign) from patients in the United States was used in the training of our breast ultrasoundCADx system. A second sonographic database consisting of 456 lesions (145 malignant, 311 benign) was obtained from a separate patient population in Asia and used to test the trained classifier. Four sonographic features were extracted from each lesion (shape, margin sharpness, posterior acoustic behavior, and texture). These features were used to train a Bayesian neural network classifier. Both round‐robin and independent testing were used to evaluate the classifier with the two databases. Performance was assessed by calculating the area under the ROC curve (AUC) for each test. Results: The AUC of the independent test was 0.80 while the AUCs for the round‐robin tests were 0.87 and 0.88 for the Asian and American databases, respectively. The difference between the independent test AUC and the round‐robin AUC for the Asian database was statistically significant (p‐value=0.02). Conclusion: This work indicates that the breast ultrasoundCADx system is moderately robust across different patient populations. The statistically significant difference between the AUCs of the Asian database along with the similarity of the round‐robin AUCs indicate that while the sonographic features used by the system are useful in both databases, their relative importance differs. This motivates the future exploration of optimal feature sets to improve the overall performance of the CADx system. Conflict of Interest: Research supported in part by NIH. Some authors receive royalties, research funding, and/or are stock holders in Hologic.

SU-GG-I-149: Preliminary Study of the Accuracy of An Automated Segmentation Algorithm for Sonographic Breast Lesions Across Different Patient Populations

Purpose: To determine the accuracy and variability of an automated sonographic breast lesion segmentation algorithm when it is used across different patient populations. Method and Materials: Two sonographic databases containing images of breast lesions were collected: one consisting of 456 lesions (145 malignant, 311 benign) from patients in Asia(database A), and one consisting of 433 lesions (127 malignant, 306 benign) from patients in the United States (database B). The same model of ultrasoundscanner was used to generate the images in both databases. Our average radial derivative‐based segmentation method was used to segment all the lesions in each database by using the center of each lesion as the starting seed point. The amount of overlap between the automated segmentation and an outline drawn by a radiologist was calculated for each lesion using O = area(S∩R)/area(S∪R) where S is the automated segmentation and R is the radiologist‐drawn outline. An overlap value of less than 0.4 is considered to be poor. Results: In database A, 85% of the lesions had an overlap value of 0.4 or greater. In database B, 80% of the lesions had an overlap value of 0.4 or greater. The automated segmentation performed better on the Asian database (p‐value=0.003). Conclusion: While the automated segmentation algorithm performs well on both databases, the disparity in performance raises interesting questions. The difference in performance could be the result of variation due to different radiologist‐drawn truths, differences in clinical imaging protocol, or even differences between the breast anatomy of the American and Asian populations. Further investigation is necessary to determine both the cause and magnitude of the variability. Conflict of Interest: Research supported in part by NIH. Some authors receive royalties, research funding, and/or are stock holders in Hologic.

An image database management system for conducting CAD research

The development of image databases for CAD research is not a trivial task. The collection and management of images and their related metadata from multiple sources is a time-consuming but necessary process. By standardizing and centralizing the methods in which these data are maintained, one can generate subsets of a larger database that match the specific criteria needed for a particular research project in a quick and efficient manner. A research-oriented management system of this type is highly desirable in a multi-modality CAD research environment. An online, webbased database system for the storage and management of research-specific medical image metadata was designed for use with four modalities of breast imaging: screen-film mammography, full-field digital mammography, breast ultrasound and breast MRI. The system was designed to consolidate data from multiple clinical sources and provide the user with the ability to anonymize the data. Input concerning the type of data to be stored as well as desired searchable parameters was solicited from researchers in each modality. The backbone of the database was created using MySQL. A robust and easy-to-use interface for entering, removing, modifying and searching information in the database was created using HTML and PHP. This standardized system can be accessed using any modern web-browsing software and is fundamental for our various research projects on computer-aided detection, diagnosis, cancer risk assessment, multimodality lesion assessment, and prognosis. Our CAD database system stores large amounts of research-related metadata and successfully generates subsets of cases that match the users desired search criteria.

TH‐C‐I‐609‐10: Computerized Classification of Non‐Biopsied Lesions Seen On Breast Ultrasound

Purpose: To investigate the performance of a computerized lesion classification scheme in a realistic testing protocol resembling clinical practice. Computerized classification of breast lesions has generally been tested on lesions of biopsy-proven pathology. In that fashion, the known pathology serves as the truth in the performance evaluation of the computer. In practice, however, many patients are never sent to biopsy because their lesions are deemed to be most likely benign. In order for a computerized classification scheme to be useful and its results believable to the radiologists, it needs to be able to classify those lesions correctly. Method and Materials: We investigated the performance of our computerized lesion segmentation and classification scheme. There were 42 images of 11 cancerous lesions, 114 images of 30 biopsy-proven benign lesions (including both cystic and solid lesions), and 243 images of 57 lesions that to date have not been sent to biopsy (including suspected cysts, benign solid lesions, and other benign breast disease). The computer was trained on the biopsy-proven set and we performed stepwise feature selection to obtain a 4-feature subset that best distinguished malignant from biopsy-proven benign lesions. The computer scheme was tested on the non-biopsy cases, and the ability to distinguish these from the cancers in the training set was assessed. Results: The area under the ROC curve (Az value) was 0.96 for the training of the scheme on biopsy-proven pathologies in the distinction between cancers and benign lesions. The Az value was 0.93 for testing in the distinction between cancers and non-biopsy-proven benign lesions. Conclusion: Our computerized classification scheme shows promising performance in a testing protocol that is more representative of its intended use in clinical practice than the typical testing on lesions with biopsy-proven pathology only. COI: grants: USPHS and U.S. Army. Shareholder: R2 Technology (last author)

Character recognition and image manipulation for the clinical translation of CAD for breast ultrasound

To be clinically viable, computer-aided diagnosis (CAD) systems must be as automated and user-friendly as possible. CAD systems for breast ultrasound are still preliminary and are not adapted for use in a standard clinical environment. For example, computer detection and classification schemes need the pixel size of each image to operate correctly, and while the DICOM standard allows pixel size to be encoded in the image file, some equipment manufacturers neglect to utilize the encoding. As a result, the pixel size is calculated from user input. In order to increase clinical efficiency and reduce the likelihood of error due to incorrect image specifications, automating this input process is a highly desirable asset. We developed and applied a character recognition algorithm to the annotation region of each ultrasound image in our database. A set of numerical masks, which corresponded to the characters used in the annotation information, enabled the filtering of each image. Numerical masks yielding the maximum output from the comparison operation between image data and mask were output to obtain the annotation information. Each image was then automatically cropped to remove the annotation banner and leave only the image data. The cropped image matrix dimensions and character recognition output were used to determine the corresponding pixel size. The algorithm was tested on 1110 images with various pixel sizes. In every case, the value output by the algorithm corresponded exactly to the true value. Our recognition algorithm now allows for the clinical translation of our fully-automated breast ultrasound CAD system.