Douglas Mossman

Three-way ROCs

Receiver operating characteristic (ROC) analysis traditionally has dealt with dichoto mous diagnostic tasks (e.g., determining whether a disorder is present or absent). Often, however, medical problems involve distinguishing among more than two diag nostic alternatives. This article extends ROC concepts to diagnostic enterprises with three possible outcomes. For a trichotomous decision task, one can plot a ROC surface on three-dimensional coordinates; the volume under the ROC surface (VUS) equals the probability that test values will allow a decision maker to correctly sort a trio of items containing a randomly-selected member from each of three populations. Thus, the VUS summarizes global diagnostic accuracy for trichotomous tests, just as the area under a ROC curve does for a two-alternative diagnostic task. Information gain at points on the surface can be calculated just as is done for two-dimensional ROC curves, and investigators can thus compare three-way ROCs by comparing maximum information gain on each ROC surface. Key words: receiver operating characteristic, ROC, three-way ROC, three-alternative decision task, resampling, bootstrap, infor mation. (Med Decis Making 1999;19:78-89)

Comparing and Optimizing Diagnostic Tests An Information-theoretical Approach

An ideal method for assessing performance of non-binary diagnostic tests would specify each tests optimal operating point and would tell a diagnostician which of many tests was the best one to use in a particular clinical situation. This article shows how information theory and receiver operating characteristic (ROC) analysis can be combined to evaluate and compare diagnostic tests at their optimum cutoffs once disease prevalence and test properties are specified. Though it is not appropriate for all clinical situations, the method can be used for most diagnostic tests whenever information is desired for its own sake or when reducing uncertainty is the goal of testing. The method also is appropriate in those situations where benefits and costs cannot be specified precisely enough to permit test optimization based on a balancing of anticipated goods and evils. Key words: non-binary diagnostic tests; information theory; receiver operating characteristic curves. (Med Decis Making 1992;12:179- 188)

Further comments on portraying the accuracy of violence predictions

Hart, Webster, and Menzies (1993) recently offered several recommendations for describing the accuracy of violence predictions using 2×2 contingency tables. This Comment describes some problems with their recommendations and suggests that researchers use receiver operating characteristic (ROC) analysis to quantify prediction accuracy.

Evaluation and optimization of diagnostic tests using receiver operating characteristic analysis and information theory

We describe a mathematical technique and an associated computer program for comparing, evaluating and optimizing diagnostic tests. The technique combines receiver operating characteristic (ROC) analysis with information theory and cost-benefit analysis to accomplish this. The program is menu driven and highly interactive; it generates 13 possible user-determined ASCII disk files which can be easily converted to graphs. These graphs allow the user to make detailed comparisons among various diagnostic tests for all values of disorder prevalence, and also provide guidelines for cut-off selection in order to optimize tests. These techniques are applied to three published studies of the enzyme screening assay for diagnosis of infection with the HIV virus. We show how graphs produced by this program can be used to compare and optimize these diagnostic tests. The program is written for an IBM-compatible microcomputer running on a DOS operating system.

Intervals for posttest probabilities: a comparison of 5 methods.

Background . Several medical articles discuss methods of constructing confidence intervals for single proportions and the likelihood ratio, but scant attention has been given to the systematic study of intervals for the posterior odds, or the positive predictive value, of a test. Methods . The authors describe 5 methods of constructing confidence intervals for posttest probabilities when estimates of sensitivity, specificity, and the pretest probability of a disorder are derived from empirical data. They then evaluate each method to determine how well the intervals’ coverage properties correspond to their nominal value. Results . When the estimates of pretest probabilities, sensitivity, and specificity are derived from more than 80 subjects and are not close to 0 or 1, all methods generate intervals with appropriate coverage properties. When these conditions are not met, however, the best-performing method is an objective Bayesian approach implemented by a simple simulation using a spreadsheet. Conclusion . Physicians and investigators can generate accurate confidence intervals for posttest probabilities in small-sample situations using the objective Bayesian approach.

An Open Trial of Olanzapine in Patients With Treatment-refractory Psychoses

Olanzapines structural similarities to clozapine and the results of premarketing clinical trials suggested potential usefulness in treating patients with treatment-refractory psychoses. Sixteen inpatients from the state hospital with severe, refractory schizophrenic or schizoaffective psychoses received olanzapine in a prospective, 12-week, open-label trial. The olanzapine dose was 10 mg/day for at least the first 6 weeks and never exceeded 20 mg/day. Mood stabilizers and other antipsychotic agents were discontinued before olanzapine was started. Patients frequently became more agitated within the first several weeks of initiating treatment, requiring the increased use of benzodiazepines and often leading to the discontinuation of olanzapine. Two patients improved significantly. Overall, significant clinical improvement was noted only for motor side effects. This study concluded that olanzapine was not effective in this heterogeneous group with chronic, severe, treatment-resistant psychosis when used in this manner. Further research is needed to explain the tendency toward agitation upon transition to olanzapine, which is reminiscent of reported risperidone complications. Clinicians should be alert for this complication and should minimize concomitant medication changes that might add to the risk of emergent agitation.

Daubert, Cognitive Malingering, and Test Accuracy

Daubert v. Merrell Dow Pharmaceuticals (1993) held that trial judges should permit expert scientific testimony only when “the reasoning or methodology underlying the testimony is scientifically valid, and ... properly can be applied to the facts in issue.” Vallabhajosula and van Gorp (“V&vG,” 2001) have suggested that when the Daubert standard is applied to tests for malingered cognitive deficits, courts should deem admissible only results that meet this mathematical standard: assuming a pretest probability of .3, a “positive” score on the malingering test should yield a posttest probability of at least .8. This paper shows that V&vGs criterion may lead to misunderstandings about the kind of information malingering measures provide. After reviewing cases that have discussed both the Daubert decision and malingered cognitive deficits, this paper uses data from the Test of Memory Malingering (T. N. Tombaugh, 1996) to provide a general characterization of the mathematical properties of malingering measures. The paper then describes how pretest knowledge about malingering is combined with knowledge about a tests performance to generate a posttest probability of malingering. The results can help mental health experts respond to Daubert-inspired challenges to conclusions based on malingering measures.

Quantifying the Accuracy of Forensic Examiners in the Absence of a “Gold Standard”

This study asked whether latent class modeling methods and multiple ratings of the same cases might permit quantification of the accuracy of forensic assessments. Five evaluators examined 156 redacted court reports concerning criminal defendants who had undergone hospitalization for evaluation or restoration of their adjudicative competence. Evaluators rated each defendant’s Dusky-defined competence to stand trial on a five-point scale as well as each defendant’s understanding of, appreciation of, and reasoning about criminal proceedings. Having multiple ratings per defendant made it possible to estimate accuracy parameters using maximum likelihood and Bayesian approaches, despite the absence of any “gold standard” for the defendants’ true competence status. Evaluators appeared to be very accurate, though this finding should be viewed with caution.

Presenting Evidence of Malingering to Courts: Insights from Decision Theory

The past decade has witnessed a significant increase in research on measures that help detect malingered performance on psychological tests conducted in forensic contexts. After reviewing the principles and procedures that have been used to design and evaluate these measures, we propose a new approach to the presentation of their results. We show how clinicians can use currently-available data and simple mathematical principles first discussed over two centuries ago (Bayes, 1763) to estimate the likelihood that an evaluee is feigning cognitive or emotional problems. Reanalyses of available data may permit professionals to present empirically-grounded probabilistic evidence about malingering in a host of commonly-encountered forensic situations.

Is expert psychiatric testimony fundamentally immoral

*Associate Clinical Professor and Director, Division of Forensic Psychiatry, Wright State University School of Medicine, P.O. Box 927, Dayton, OH 45401, U.S.A. The author thanks Michael L. Perlin and Edward M. Hundert for their advice and critiques of an earlier draft of this article, and Eugenie Moody for her research and editing assistance. ‘To simplify discussion, I have focused this article on a set of ethical problems faced bypsychiatrists, who are but one subset of clinicians who serve as courtroom experts. But the general ethical issues I describe also are relevant to the courtroom activities of nonpsychiatric physicians and other mental health professionals, especially psychologists. For example, the issues I raise concerning psychiatrists’ testimony apply in loto to psychologists’ testimony, even though their professional ethical principles have different historical roots. See, e.g., infra notes 5, 6, 13, and 23. ‘See, e.g., Jay Katz, “The Fallacy of the Impartial ExperYRevisited, 20 BULL. AM. ACAD. PSYCHIATRY LAW 141, 143-44 (1992) (concerns about experts-for-hire are exaggerated, but “one finds a greater number of ‘hired guns’ stalking the courlroom who, aided and abetted by members of the legal profession, prostitute their psychiatric knowledge”); Paul S. Appelbaum, Forensic Psychiatry: The Need for Self-Regulation, 20 BULL. AM. ACAD. PSYCHIATRY LAW 153, 154 (1992) (discussing psychiatrist who is notorious “as an opinion for hire” and “shop” talk among forensic psychiatrists about questionable testimony and “the purportedly self-serving motives that lie behind it”). Physician concern about the behavior of forensic clinicians is not a recent development: See Frank J. Ayd, Readings & Reflections, PSYCHIATRIC TIMES 41 (August, 1992) (quoting commentary in 18 JAMA 304 [1892]) (“The boast of some secular papers, that it is possible to buy medical opinions of every character and description, has some shadow of truth”). The belief that some physicians are just “whores for hire” has a longlegalhistory.See,e.g., 1 NIGEL WALKER,~RIMEAND INSANITY IN E~~~~~~(1968)at82,who quotes crown counsel’s cross-examination of Dr. Leo in Rex. v. Lawrence (1801): “Have you not been here before as a witness and a Jew physician, to give an account of a prisoner as a madman, to get him off upon the ground of insanity?” Dr. Leo was a prominent member of London’s Sephardic Jewish community who had participated in the Hudfield trial. For additional examples, see Phillip J. Resnick, Perceptions of Psychiatric Testimony: A Historical Perspective on the Hysterical Invective, 14 BULL AM. ACAD. PSYCHIATRY LAW 203 (1986).