Christine M. Schubert Kabban

Practice patterns for peer review in radiation oncology

PURPOSE Physician peer review seeks to improve the quality of care through the evaluation of physician performance, specifically medical decision making and technical expertise. To establish current peer review practice patterns, evaluate interest in recommendations for peer review, and establish a framework for future recommendations, the American Society for Radiation Oncology (ASTRO) surveyed its physician members. METHODS AND MATERIALS A radiation oncology-specific peer review survey instrument was developed, formally tested, and found to meet established levels of reliability and validity. The final instrument was delivered using a web-based survey platform including reminders. All ASTRO physician-members and members-in-training worldwide were invited by email to participate. RESULTS A total of 5674 physicians were contacted starting in January 2013. A total of 572 physicians participated (10%) yielding a ±4% margin of error. Those responding were split evenly between academic providers and private practice and others. The median time since training=16 years, median number of new patients per year=215, and median practice size=6 physicians; 83% of respondents were involved in peer review and 75% were comfortable with their program. Of those involved, 65% report doing some review before radiation begins. Of patients treated by these physicians, 56% are reviewed before treatment. Peer review elements reviewed include overall treatment strategy (86%), dose and fractionation (89%), contouring (59%), and isodose or dose-volume histogram (75%). Ninety percent of physicians have changed radiation plans because of peer review. These providers make changes in 7%-10% of cases. Seventy-four percent of physicians agree that ASTRO should make formal peer review recommendations, with 7% in opposition. CONCLUSIONS This survey suggests that peer review in radiation oncology is common and leads to changes in management in a meaningful fraction of cases. There is much variation in the manner of conducting, and reported utility of, peer review. The majority of ASTRO physician members support formal recommendations and guidance on peer review.

The probability of detection for structural health monitoring systems: Repeated measures data

The United States Air Force currently relies on schedule-based inspections using nondestructive evaluation methods for ensuring airframe integrity. The sensitivity of a nondestructive evaluation method is quantified statistically using a probability of detection process. The purpose of the probability of detection process is to generate a a 90 | 95 metric for a given nondestructive evaluation technique and corresponding defect (e.g. crack). This process could be conducted under various inspection conditions and defect sizes. The set of factors varied in the process is controlled to allow each nondestructive evaluation inspection to be treated as statistically independent. Current United States Air Force structural inspections are performed at time intervals that adhere to the independence assumption. However, the United States Air Force plans to service airframes based on their actual condition instead of the current schedule-based approach. Accordingly, there is emphasis on developing advanced health management technologies, such as structural health monitoring systems, which provide an automated and real-time assessment of a structure’s ability to serve its intended purpose. Therefore, structural health monitoring is considered to be equivalent to an in situ nondestructive evaluation structural inspection device. With a structural health monitoring system, the time interval between inspections will be much smaller than the time intervals between nondestructive evaluation inspections. Since structural health monitoring measurements are from the same sensors, in the same location, the independent measurement assumption used to analyze nondestructive evaluation methods is invalid. In this article, we present a statistical method consistent with current probability of detection process, yet designed to appropriately analyze dependent data. We demonstrate this method first with simulated data and then with experimental data from three test specimens of a representative aircraft structural component. This method leverages the advantages of a structural health monitoring system through its frequent measurements while maintaining its usefulness through appropriately computed probability of detection values. Furthermore, we present a numerical method for estimating the number of test specimens needed to achieve a desired a 90 | 95 value.

Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation

A deep LSTM architecture is proposed to improve cross-day EEG feature stationarity.Mean, variance, skewness, and kurtosis input features are statistically evaluated.Models account for temporal dependencies in brain activity data, improving results.Achieves average classification accuracy of 93.0% using a deep LSTM architecture.A 59% reduction in error compared to best previously published results for dataset. Using deeply recurrent neural networks to account for temporal dependence in electroencephalograph (EEG)-based workload estimation is shown to considerably improve day-to-day feature stationarity resulting in significantly higher accuracy (p < .0001) than classifiers which do not consider the temporal dependence encoded within the EEG time-series signal. This improvement is demonstrated by training several deep Recurrent Neural Network (RNN) models including Long Short-Term Memory (LSTM) architectures, a feedforward Artificial Neural Network (ANN), and Support Vector Machine (SVM) models on data from six participants who each perform several Multi-Attribute Task Battery (MATB) sessions on five separate days spread out over a month-long period. Each participant-specific classifier is trained on the first four days of data and tested using the fifths. Average classification accuracy of 93.0% is achieved using a deep LSTM architecture. These results represent a 59% decrease in error compared to the best previously published results for this dataset. This study additionally evaluates the significance of new features: all combinations of mean, variance, skewness, and kurtosis of EEG frequency-domain power distributions. Mean and variance are statistically significant features, while skewness and kurtosis are not. The overall performance of this approach is high enough to warrant evaluation for inclusion in operational systems.

Industrial Age non-destructive evaluation to Information Age structural health monitoring

This article was originally presented as a keynote address at the Ninth International Workshop on Structural Health Monitoring with the intent of provoking discussions relating to the transformation of aircraft maintenance practices by exploiting opportunities and benefits offered by Information Age technology and techniques. The US Air Force currently manages its aircraft using a schedule-based maintenance philosophy. This schedule-based approach works well for ensuring aircraft integrity; however, it is very costly, labor-intensive, and reduces aircraft availability. Structural health monitoring systems have the potential to analyze near-real-time and historical weapon systems data to provide a predictive maintenance capability. However, much aerospace structural health monitoring research has focused on in situ structural inspection techniques instead of structural monitoring. Structural inspections typically entail examining key locations of an airframe for material degradation or flaws. These examinations usually occur at predefined time intervals. As such, each inspection is considered an independent evaluation. Conversely, structural monitoring involves continuous condition surveillance of an airframe over an extended period of time. Structural monitoring uses past conditions and expected future conditions for producing a comprehensive understanding of the current health state. A new architecture, Cognitive Architecture for State Exploitation, is introduced as a monitoring technique that combines diagnostic or state (i.e. health) assessments, prognostic assessments, and mission objectives into a common framework to enable goal-based decision making. Results from a laboratory experiment are utilized to demonstrate the application of Cognitive Architecture for State Exploitation and to illustrate the potential to improve effectiveness and efficiency metrics compared to those of the current US Air Force maintenance procedures.

Label fusion of classification systems via their ROC functions

A classification system with N possible output labels (or decisions) will have N(N-1) possible errors. The Receiver Operating Characteristic (ROC) manifold was created to quantify all of these errors. When multiple classication systems are fused, the assumption of independence is usually made in order to mathematically combine the individual ROC manifolds for each system into one ROC manifold. This paper will investigate the label fusion (also called decision fusion) of multiple classication systems that have the same number of output labels. Boolean rules do not exist for multiple symbols, thus, we will derive possible Boolean-like rules as well as other rules that will yield label fusion rules. The formula for the resultant ROC manifold of the fused classication systems which incorporates the individual classication systems will be derived. Specically, given a label rule and two classication systems, the ROC manifold for the fused system is produced. We generate formulas for other non-Boolean-like OR and non-Boolean-like AND rules and give the resultant ROC manifold for the fused system. Examples will be given that demonstrate how each formula is used.

Development of an updated tensile neck injury criterion

BACKGROUND Ejection neck safety remains a concern in military aviation with the growing use of helmet mounted displays (HMDs) worn for entire mission durations. The original USAF tensile neck injury criterion proposed by Carter et al. (4) is updated and an injury protection limit for tensile loading is presented to evaluate escape system and HMD safety. METHODS An existent tensile neck injury criterion was updated through the addition of newer post mortem human subject (PMHS) tensile loading and injury data and the application of Survival Analysis to account for censoring in this data. The updated risk function was constructed with a combined human subject (N = 208) and PMHS (N = 22) data set. RESULTS An updated AIS 3+ tensile neck injury criterion is proposed based upon human and PMHS data. This limit is significantly more conservative than the criterion proposed by Carter in 2000, yielding a 5% risk of AIS 3+ injury at a force of 1136 N as compared to a corresponding force of 1559 N. DISCUSSION The inclusion of recent PMHS data into the original tensile neck injury criterion results in an injury protection limit that is significantly more conservative, as recent PMHS data is substantially less censored than the PMHS data included in the earlier criterion. The updated tensile risk function developed in this work is consistent with the tensile risk function published by the Federal Aviation Administration used as the basis for their neck injury criterion for side facing aircraft seats.

A Macro-Stochastic Model for Improving the Accuracy of Department of Defense Life Cycle Cost Estimates

The authors present a prognostic cost model that is shown to provide significantly more accurate estimates of life cycle costs for Department of Defense programs. Unlike current cost estimation approaches, this model does not rely on the assumption of a fixed program baseline. Instead, the model presented here adopts a stochastic approach to program uncertainty, seeking to identify and incorporate top-level (i.e., “macro”) drivers of estimating error to produce a cost estimate that is likely to be more accurate in the real world of shifting program baselines. The predicted improvement in estimating accuracy provided by this macro-stochastic cost model translates to hundreds of billions of dollars across the Department of Defense portfolio. Furthermore, improved cost estimate accuracy could reduce actual life cycle costs and/or allow defense acquisition officials the ability to make better decisions on the basis of more accurate assessments of value and affordability.

Improving ATR system performance through sequences of classification tasks

Complex ATR tasks are often decomposed into the identification of sub-targets, that is, objects are sorted and identified as one particular target type and then those targets are further identified. For instance, a field of view may be partitioned into natural and man-made objects. After which, the man-made objects are screened to identify a particular object of interest. These tasks combine classifiers which operate in isolation of each other, yet in fact, perform as a classification sequence. This work examines this scenario, building the ATR task as a sequence of target identifications. Two sequences will be highlighted: Believe the Negative (BN) and Believe the Extremes (BE). In a BN sequence, the second classification system only operates if a target is identified from the first classification system. In a BE sequence, the second classification system only operates if there is no identification from the first classification system. Performance of these classification sequences will be compared to classification systems operating separately. Further, sequence augmentation will be examined to demonstrate how the ATR task may still be completed when information is missing on the primary target. This missing information may represent atmospheric blurring, alternate field of view, or other disturbances. An example of the performance of the sequences under simulated, theoretical levels of missing information is examined, and formulas are presented to describe the optimal performance of these systems when augmented and un-augmented. In conclusion, this work demonstrates utility in how these sequences fuse target information in order to complete an ATR task.

Cross-Participant EEG-Based Assessment of Cognitive Workload Using Multi-Path Convolutional Recurrent Neural Networks

Applying deep learning methods to electroencephalograph (EEG) data for cognitive state assessment has yielded improvements over previous modeling methods. However, research focused on cross-participant cognitive workload modeling using these techniques is underrepresented. We study the problem of cross-participant state estimation in a non-stimulus-locked task environment, where a trained model is used to make workload estimates on a new participant who is not represented in the training set. Using experimental data from the Multi-Attribute Task Battery (MATB) environment, a variety of deep neural network models are evaluated in the trade-space of computational efficiency, model accuracy, variance and temporal specificity yielding three important contributions: (1) The performance of ensembles of individually-trained models is statistically indistinguishable from group-trained methods at most sequence lengths. These ensembles can be trained for a fraction of the computational cost compared to group-trained methods and enable simpler model updates. (2) While increasing temporal sequence length improves mean accuracy, it is not sufficient to overcome distributional dissimilarities between individuals’ EEG data, as it results in statistically significant increases in cross-participant variance. (3) Compared to all other networks evaluated, a novel convolutional-recurrent model using multi-path subnetworks and bi-directional, residual recurrent layers resulted in statistically significant increases in predictive accuracy and decreases in cross-participant variance.

Fusion within a classification system family

A detection system outputs two distinct labels, thus, there are two errors it can make. The Receiver Operating Characteristic (ROC) function quantifies both of these errors as parameters vary within the system. Combining two detection systems typically yields better performance when a combining rule is chosen appropriately. When detection systems are combined the assumption of independence is usually made in order to simplify the math- ematics, so that we need only combine the individual ROC curve from each system into one ROC curve. This paper investigates label fusion of two detection systems drawn from a single Detection System Family (DSF). Given that one knows the ROC function for the DSF, we seek a formula with the resultant ROC function of the fused detection systems as a function (specifically, a transformation) of the ROC function. In this paper, we derive this transformation for the disjunction and conjunction label rules. Examples are given that demonstrates this transformation. Furthermore, another transformation is given to account for the dependencies between the two systems within the family. Examples will be given that demonstrates these ideas and the corresponding transformation acting on the ROC curve.