Martha R. Horton

A Method for Evaluating Computer Programs for Electrocardiographic Interpretation II. Application to Version D of the PHS Program and the Mayo Clinic Program of 1968

A previously described method for evaluating computer programs for electrocardiographic (ECG) interpretation was applied to Version D of the Public Health Service (PHS) program and to the Mayo Clinic program of 1968. Staff cardiologists found agreement with the results of the PHS program in 45.5% of 1150 unselected tracings. Clinically significant disagreements based strictly on application of different criteria occurred in 29%, while disagreements based on program errors were found in 25.5%. The corresponding results for the Mayo Clinic program are: agreement in 47%, disagreements due to criteria differences in 30.9%, and disagreements due to program errors in 22.1%.Both programs had serious deficiencies, particularly in the diagnostic categories of myocardial infarction and cardiac arrhythmias. PHS program errors resulted primarily from mismeasurements and deficient program logic, while Mayo Clinic program errors more frequently resulted from pattern recognition failures. Neither program appears suitable for routine clinical use at the present time.

A Method for Evaluating Computer Programs for Electrocardiographic Interpretation I. Application to the Experimental IBM Program of 1971

A method for evaluating computer programs for electrocardiographic interpretation is described. This method allows a clinician to judge the usefulness of a program for his specific setting and needs. The method requires a significant proportion and variety of abnormal tracings, the application of specific fixed criteria, and the separation of disagreements between the computer program and the clinician into those resulting from criteria differences and those resulting from programming errors, viz., pattern recognition failures, mismeasurements, and/or deficient program logic. When applied to the experimental IBM program 1971, staff cardiologists found essential agreement with the programs results in 76% of 1150 unselected tracings. Clinically significant disagreements based strictly on the application of different criteria occurred in 20% of the tracings, whereas disagreements based on program errors were found in only 4%. Although this program requires some system of human overview and quality checking, its potential for clinical implementation is worthy of consideration.

A Method for Evaluating Computer Programs for Electrocardiographic Interpretation III. Reproducibility Testing and the Sources of Program Errors

A simple method for testing reproducibility in ECG computer program performance results from using two digital representations of the same analog ECG tracing. Each digital representation is separated from the other by one millisecond in time. When the digital representations are processed by the Mayo Clinic program (1968), the diagnostic statements are identically reproduced in only 60% of 33 tracings. When the method is applied to version D of the PHS program and to the newly released IBM program of 1973, identical reproducibility is 43.3% and 76.0%, respectively, of 217 tracings. After analog filtering these figures are improved to 49.8% and 79.7%, respectively. These results show that reproducibility is most affected by a programs algorithms for pattern recognition, measurement, consistency checking, and noise handling. Reproducibility is less affected by attenuation of high frequency noise at the analog level. The relationship of reproducibility to program error rate in previous studies is discussed. Hence poor performance on this test obviates the need for a more time-consuming clinical evaluation. The need for human overview and quality checking is re-emphasized.

A review of algorithms for molecular sequence comparison

Computers have recently become an essential component of research in molecular biology. Most computer analyses of nucleic acid and protein sequences depend on comparisons between sequences. These comparisons, depending on their purpose, may differ not only in the kinds of comparisons that are done, but also in the way the results of the comparison are used by molecular biologists or by other computer programs. This paper reviews algorithms currently in use to solve comparison problems in molecular biology. Each algorithm is explained in detail and discussed in terms of the molecular biology problems it is most suited to solve.

The importance of reproducibility testing of computer programs for electrocardiographic interpretation: application to the automatic vectorcardiographic analysis program (AVA 3.4).

Abstract The automatic vectorcardiogram analysis (AVA 3.4) program developed by Pipberger and associates has a unique manner of using multivariate statistics to make morphological diagnoses of the QRS and P waveforms. This scheme produces statements of probabilities for up to seven different QRS diagnoses and three different P wave diagnoses. It would require a complicated protocol with a very large population to establish the validity of this approach. However we have described a simple method for evaluating that part of an ECG program which is independent of its criteria or clinical accuracy. Poor performance on this test obviates the need for more time-consuming clinical evaluation. Results of applying this test to the AVA 3.4 program are reported.

Nonparametric comparison of entire ROC curves for computerized ECG left ventricular hypertrophy algorithms using data from the framingham heart study

A computer program may be capable of several different statements for left ventricular hypertrophy (eg, possible LVH, probable LVH, consistent with LVH), but such statements resulting from discretized levels of sensitivity/specificity would represent only isolated points on a receiver-operating characteristic (ROC) curve, which is a plot of all levels of sensitivity versus specificity. Even if two algorithms use the same discrete scales, their performances may not readily be compared. The authors present a comparison methodology for ROC curves using ROC area as a nonparametric measure of the ability of the algorithm to separate the two populations; the ROC area ranges from 0.5 (no ability) to 1.0 (perfect separation) and is unbiased if the normal versus abnormal populations have no common values for the measurement. The methodology compares the performance of ECG algorithms on the same population of cases by testing for significant differences of ROC areas and incorporating correlation of the algorithms in a nonparametric way. To illustrate this methodology, they use ECG and echocardiographic data from the Framingham Heart Study.

Computing on a shoestring: Initial data entry for service organizations

This paper addresses the feasibility of computerized record-keeping for low-budget volunteer organizations, and presents results of an experiment designed to determine a fast, reliable, and comfortable data entry technique for enabling non-computer-user to enter large amounts of manually recorded data into a computer file.

Testing Of Updated Program For ECG Analysis

The reproducibility of Version 2 of the IBM program was tested in 217 unselected electrocardiograms according to a previously described method and compared to Version 1 of the IBM program. On the average it was found that Version 2 attempted a more elaborate scheme of feature extraction and commentary, resulting in fewer cases being identical and more cases being similar. Heavy filtering improved the reproducibilities of Version 1, though sometimes with a loss of diagnostic statements. However, the same filtering did not appear to affect the results of Version 2 significantly.

Evaluating Computer ECG Programs The authors reply

For myocardial infarction (MI) the situation is equally unsatisfactory. Every criterion for MI diagnosis listed on p 85 has a certain sensitivity and a certain specificity. But reliable data on these are lacking. All that can be concluded from the evaluation is that readers and computers disagreed more often for MI than for LVH. But the question remains: who is right? In the absence of an independent standard of diagnosis, given by nonECG criteria, we have merely a game in which blind men are pitted against a blind computer. Most distressing is the way in which the trappings of objectivity and quantification, and the aura of NIH sponsorship, despite the initial disclaimer, will mislead the unwary clinician into accepting these results. To list the criteria does not validate them, and to call a conditional agreement rate sensitivity does not make it sensitivity. In the absence of documentation of the diagnoses the evaluation tells us nothing about the quality of either human reader or computer interpretation. If such documentation cannot be supplied at the NIH, it should stay out of clinical evaluations of the ECG. HUBERT V. PIPBERGER, M. D. JEROME CORNFIELD The George Washington University Washington, D.C. References